HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

Thank you for your valuable comments. Please see our responses below.

Q1: The claim and the effects on other domains.

Training Stability and Domain Generalization

Training stability involves reducing data variability by transforming chaotic pre-training data into consistent QA format that aligns closely with SFT data. This results in a steadily decrease in training loss, as depicted in Figure 5.

Regarding domain generalization, we realize our explanation was vague. We aimed to convey that one-stage training can better generalize domain-specific knowledge to downstream tasks. We will revise this statement to avoid confusion.

Effects on Other Domains

Our one-stage training protocol is applicable beyond the Chinese medical domain.

To address the reviewer’s question, we test it on the MedQA-English (Jin et al., 2020) and CaseHOLD (Zheng et al., 2021) datasets, representing the English medical and law domains, respectively. For MedQA-English, we used MedQA's English textbooks as the pre-training corpus and its training set for SFT. For CaseHOLD, we sample 4% corpora of FreeLaw Opinions (Biderman et al., 2022) as the pre-training corpus and used its training set for SFT. Data characteristics are as follows:

We trained on Baichuan2-7B-Base and compared one-stage training with traditional two-stage training:

The results indicate that one-stage training can enhance performance across domains.

Q2. Please address the issues in writing quality noted, such as the misuse of "Camel Case" and other linguistic inaccuracies.

We acknowledge and appreciate your feedback on the writing quality of our manuscript, including the misuse of "Camel Case" and several grammatical errors. We commit to a thorough revision to address these issues and to ensure that our manuscript meets the high standards of academic writing. Your feedback is appreciated and will certainly help improve the quality of the paper.

We sincerely appreciate your valuable comments. Below, we provide responses to your concerns.

Q1: The role of GPT-4 and fairness.

We appreciate your suggestions. We will expand the discussion as follows:

Upper Bound GPT-4 is not the upper bound of our models in the Chinese medical domain, as they learn from domain-specific corpora rather than relying solely on GPT-4. Appendix E shows that the SFT data (generated by GPT-4) is not essential for domain expertise.

Fairness of Comparison We strive to enhance the fairness of evaluations by utilizing expert assessments and the latest exams.

Q2: Clarify the distribution shifts.

Our training data is rich and varied, containing not only biomedical content but also clinical content, such as web corpora and textbooks. We value your suggestion and will clarify the distribution shifts from general to medical.

Q3: The suggestion about key appendices.

Thank you for your advice and feedback! In the next version, we will endeavor to move important appendices to the forefront to enhance readability.

Q4: Future directions about HuatuoGPT.

The evolution from Huatuo to Huatuo-II highlights our focus on two critical areas:

• Enhancing Trustworthiness: Minimizing hallucinatory outputs is crucial in the medical domain. We are exploring retrieval-augmented generation and reliability scoring for trustworthiness.
• Enhancing Domain Capability: We are advancing their abilities to analyze medical images, such as CT and MRI scans, and improve decision-making abilities in diagnostic support.

Q5: Other LLMs to generate SFT data.

In response to your concern, we conducted experiments with different LLMs to generate SFT data, using 5% of the data due to time constraints. The results are as follows:

Q6: Performance on smaller or larger models.

We will train a tiny HuatuoGPT-II to assess the smaller models. In fact, there is a 34B version of HuatuoGPT-II (based on Yi-34B), and its performance compared is as follows:

Thank you for reviewing our paper. Below, we provide responses to your concerns.

Q1: Concern about the latest exams.

Thank you for raising a valid point regarding the potential overlaps with prior materials. We will add the following clarification to our paper:

• Uniqueness of the Exam: According to the official exam website, the questions for the National Pharmacist Licensing Exam are strictly designed and newly formulated. This significantly reduces the risk of data leakage and ensures greater fairness.

• Comparison with Previous Exams It is quite worthwhile to examine the scores from previous exams! In Appendix B, we provide the results of the pharmacist exams from the past three years for comparison:

Note that the exam questions from 2017-2019 are incomplete, with details available in Appendix B.

Nonetheless, we observe that: (i) HuatuoGPT-II scores are relatively close to those of previous exams, and (ii) some models (such as ERNIE Bot and Baichuan2-13B-Chat) scored notably higher in previous years compared to the 2023 exam.

Q2: "Vertical Domains" & "Philosophy of Parsimony"

“Vertical domains” refer to specialized fields such as medicine, law, and finance, where training focuses on deep, domain-specific knowledge and disregards other unnecessary domains. This contrasts with horizontally trained models, which possess broader but shallower general knowledge.

We appreciate the reviewer's suggestion and will add a footnote explaining this "Philosophy of Parsimony".

Q3: Terminology and grammatical corrections

Special thanks to the reviewer for contributing to the improvement of our manuscript. We commit to correct these linguistic errors and thoroughly review our paper to enhance its quality.

We sincerely appreciate your valuable comments. Below, we address your concerns.

Q1: The Connection between Pre-Training Corpora and Pre-Training Instructions

Both learning approaches aim for autoregressive goals, yet the differences between them are both subtle and significant:

• Vanilla Pre-Training focuses on learning the probability of a sequence $p(y_i \mid y_0, \ldots, y_{i-1})$ , primarily targeting next-token prediction. This method enables the model to learn correlations between words, with the expectation that it can generalize to downstream scenarios to handle various types of questions.

• In contrast, HuatuoGPT-II aims to learn $p(y_i \mid X, y_0, \ldots, y_{i-1})$, integrating the input $X$. This method learns a direct mapping from $X$ to $Y$, establishing a more high-level relationship (e.g. causal) rather than merely correlations, which aligns more closely with knowledge understanding and the SFT objective.

Validation Experiment As shown in Appendix G, training in a QA format outperforms vanilla pre-training:

Q2: The use of baselines.

We extra conduct a comparison of eight medical baselines in Appendix N for a more comprehensive domain-specific analysis.

Q3: Usage of GPT-4 in data generation and evaluation.

GPT-4 for Evaluation In addition to using GPT-4 for evaluation, we also employed medical expert evaluations and the latest medical exams to ensure fairness.

GPT-4 in Data Generation GPT-4 is used to generate SFT data for instruction-following. Moreover, Appendix E demonstrates that SFT data is not essential for domain expertise but rather for pre-training instruction.

Q4: Types of questions generated.

We sampled 100 questions from the pre-training instructions and manually categorized their type as follows:

Q5: Intuition behind training with the pre-training instruction.

As outlined in response to Q1, our intuition is that pre-training instruction facilitates a more direct method to learn explicit causal relationships, as opposed to mere word correlations in the pre-training. We appreciate this valuable inquiry and will clarify this further in our revised paper.