Adapting Large Language Models via Reading Comprehension

Thank you for recognizing the interesting observations in our paper and acknowledging the effectiveness of our method. In response to your questions regarding the comparison with RECKONING [1], other models, and the decrease in prompting ability, we provide the following analysis.

Q1: Comparison with RECKONING [1]

Thank you for providing such a valuable reference. Below is a comparison between our work with RECKONING, and we have incorporated RECKONING into our paper.

RECKONING enhances reasoning ability by incorporating contextual knowledge into the model's parameters before exposing it to a question at inference time. If we view the model's learning from contextual knowledge as the "reading" process and the model's predicting on the question as the "comprehension" process, the overall RECKONING process can be viewed as a form of reading comprehension.

However, our reading comprehension differs from RECKONING in many aspects:

• The most obvious difference lies in the goal: RECKONING enhances reasoning ability, while we improve domain expertise.
• Another difference is the timing of model parameter updates: RECKONING performs these updates at inference time, whereas we perform updates during the pre-training stage.
• Moreover, if I understand it correctly, RECKONING learns contextual knowledge specifically for ad-hoc single or multiple related questions (the number of knowledge-related questions is set up to 18 in the experiment), while we learn domain-specific knowledge for various downstream tasks.

Q2: Uncertain whether consistent results would be observed on larger-scale models and models after RLHF.

Many thanks to your inspirational suggestion! Please see the General Response where we conduct experiments on a larger model---LLaMA-13B. Moreover, we find our reading comprehension is well-suited to the data format of the model after RLHF, contributing to significant gains!

Q3: Why pre-training on domain-specific knowledge leads to a decrease in LLM's prompting ability. Does the term 'input-output patterns' pertain to limited data patterns?

Thanks again for your insightful advice! Your explanation of the term "input-output patterns" is right, it relates to limited data patterns in the domain-specific raw corpora.

However, we'd like to clarify that it is not the domain-specific knowledge that hurts the prompting ability for question answering; instead, it is the limitation of data patterns within the domain-specific raw corpora that impacts this ability. We provide an analysis on the data patterns of domain-specific corpora below, and this analysis has also been included in our paper.

Domain-Specific Corpora Have Fewer QAs than General Corpora. We conduct a case study on the question answering pattern. Specifically, we search over all three domain-specific corpora and a representative corpora in the general domain---RedPajama [2]---to estimate how many sentence pairs follow the question-answering pattern, utilizing this mining pattern:

{What|Who|When|Where|Why|Which|How}{SENT1}? {SENT2}

The following table lists the number of tokens belonging to the mined question-answering pairs per million tokens. Compared with the general corpora, domain-specific corpora have much lower rates of data following the question-answering pattern, which could lead to the observed decrease in question-answering ability.

References:

[1] Zeming Chen et al. RECKONING: reasoning through dynamic knowledge encoding. CoRR, abs/2305.06349, 2023.

[2] Together Computer. Redpajama: an open dataset for training large language models, 2023.

Thank you very much for your great appreciation of our work! Below, we provide clarifications on the supplementary materials and implementation details.

Q1: More Supplementary Materials

Thank you for the feedback on our supplementary material. In the original supplementary material, we included the code for transferring raw texts to reading comprehension texts, and the evaluation datasets. To address your concerns, we provide more code and data through this anonymous link.

Q2: Implementation Details

We appreciate your feedback. The implementation details, including the use of 32 V100-32GB GPUs, were presented in Appendix B. To enhance accessibility for readers, we have incorporated more explicit descriptions in the main body of our paper, making it easier for the readers to locate the implementation details.

Thanks for your positive attitude! We hope the following analysis can address your concerns about performance improvement, effectiveness for smaller and larger models, and integrity of transformation.

Q1: Whether the performance improvements are significant.

Demonstrating an average improvement on our evaluation benchmarks is challenging because the diversity of tasks within each domain requires consistent gains across multiple tasks.

As shown below, in the biomedicine domain, MedAlpaca-7B, trained on diverse domain-specific data, exhibits significant improvements on certain tasks (e.g., 7.6% on ChemProt and 2.2% on USMLE). However, the improvements are not consistent across other tasks, resulting in an average improvement of 0.9%, whereas we achieve 3.1%. This trend is also noticeable in LLaMA-13B, where MedAlpaca-13B outperforms the LLaMA-13B by 0.8% on average, while we achieve a 5.1% absolute gain.

(* AdaptLLM-13B is trained from LLaMA-13B using our method)

In the law domain, LexGPT outperforms its base model GPT-J on one task but does not exhibit the same trend on other tasks, resulting in less favorable average performance. In finance, our average improvement enables us to compete with BloombergGPT-50B.

Q2: Whether the approach is similarly effective for smaller and larger models.

Thanks for your helpful suggestion, please refer to the General Response above to see that our approach remains effective for smaller and larger models!

Q3: Have the authors manually verified whether their transformation is effective without compromising the integrity of the text?

Yes! We have manually checked the transformed texts before the training process. First, in the mining stage, our strategy is based on [1], where the authors have verified the reliability of the mining strategy. Besides, we have made some simple but effective modifications, such as requiring longer sentence lengths and avoiding the newline character \n, to ensure data quality. Second, in the transformation stage where the data is converted into reading comprehension, each transformation template has been manually checked.

To further address your concerns, we sample 200 reading comprehension texts from each domain and manually evaluate the readability and coherence. The results indicate high satisfaction:

Reference:

[1] Mozes van de Kar et al. Don’t prompt, search! mining-based zero-shot learning with language models. In EMNLP, Association for Computational Linguistics, 2022.

Q4: Does the approach benefit another type of models?

Thank you for the insightful suggestion. As shown in the General Response above, we add experiments on another type of model---Pythia [4]---and demonstrate positive effects of our method!

References:

[1] Mozes van de Kar et al. Don’t prompt, search! mining- based zero-shot learning with language models. In EMNLP, Association for Computational Linguistics, 2022.

[2] Leo Gao et all. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021.

[3] Hongyang Yang et all. Fingpt: Open-source financial large language models. CoRR, abs/2306.06031, 2023.

[4] Stella Biderman et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML, volume 202 of Proceedings of Machine Learning Research, PMLR, 2023.

Q3: Why the performance improvement on domain "law" is the least?

Thanks for your careful review, but actually we do not have a very definitive explanation for this phenomenon. By comparing pre-training data in the three domains, we infer that the low readability of law pre-training corpora might be the reason:

• Biomedicine---PubMed Abstract [2]: Each text is an abstract for a research paper, which means the text should be highly clear, concise and readable. A randomly sampled example is as follows:

Pattern-recognition receptors: signaling pathways and dysregulation in canine chronic enteropathies-brief review.
Pattern-recognition receptors (PRRs) are expressed by innate immune cells and recognize pathogen-associated molecular patterns (PAMPs) as well as endogenous damage-associated molecular pattern (DAMP) molecules. With a large potential for synergism or convergence between their signaling pathways, PRRs orchestrate a complex interplay of cellular mediators and transcription factors, and thus play a central role in homeostasis and host defense. Aberrant activation of PRR signaling, mutations of the receptors and/or their downstream signaling molecules, and/or DAMP/PAMP complex-mediated receptor signaling can potentially lead to chronic auto-inflammatory diseases or development of cancer. PRR signaling pathways appear to also present an interesting new avenue for the modulation of inflammatory responses and to serve as potential novel therapeutic targets. Evidence for a dysregulation of the PRR toll-like receptor (TLR)2, TLR4, TLR5, and TLR9, nucleotide-binding oligomerization domain-containing protein (NOD)2, and the receptor of advanced glycation end products (RAGE) exists in dogs with chronic enteropathies. We describe the TLR, NOD2, and RAGE signaling pathways and evaluate the current veterinary literature-in comparison to human medicine-to determine the role of TLRs, NOD2, and RAGE in canine chronic enteropathies.

• Finance---Stock News: We collect stock news using the codebase of FinGPT [3], which could ensure the data quality. A randomly sampled example is as follows:

Chevron, Johnson & Johnson share gains lead Dow's 100-point jump
Powered by strong returns for shares of Chevron and Johnson & Johnson, the Dow Jones Industrial Average is up Wednesday morning. Shares of Chevron CVX, -2.35% and Johnson & Johnson JNJ, -0.13% are contributing to the blue-chip gauge's intraday rally, as the Dow DJIA, -1.01% is trading 105 points higher (0.3%). Chevron's shares have climbed $3.84, or 2.4%, while those of Johnson & Johnson are up $2.85, or 1.8%, combining for an approximately 44-point boost for the Dow. Merck MRK, -0.25%, Apple Inc. AAPL,, and Intel INTC, -1.95% are also contributing significantly to the gain. A $1 move in any one of the 30 components of the benchmark equates to a 6.59-point swing. Editor's Note: This story was auto-generated by Automated Insights, an automation technology provider, using data from Dow Jones and FactSet. See our market data terms of use.

• Law—--FreeLaw Opinions [2]: Each text is a legal opinion from federal and state courts, documenting many oral snippets. In contrast, the corpora in the other domains mainly consist of written passages. We infer that the unique feature of incorporating spoken language may affect the overall readability of law corpora. A randomly sampled example is as follows (with certain portions omitted and presented as (...)). We can observe that the readability of law corpora appears to be lower than that of the other two domains.

92 P.3d 294 (2004)                                                                                                                                
2004 WY 71                                                                                                                                        
STATE of Wyoming, ex rel., WYOMING WORKERS' SAFETY AND COMPENSATION DIVISION, Appellant (Petitioner),                                             
v.                                                                                                                                                
Anthony N. SAVICKI, Appellee (Respondent).                                                                                                        
(...)
We have previously entertained the Division's dissatisfaction with our failure to consider subsequent wage increases based upon merit and said this:
(...)
[¶ 16] Conner rejected the notion that an employee may be penalized by wage increases earned for merit at a later time. We reiterate that the pivotal question is whether the employee's injury has caused a loss of earning capacity and, if so, whether a comparison of the pre-injury wage to the post-injury wage offered is compensable under the applicable statute. In this case, Savicki's post-injury wage is compensable, and the order awarding benefits is affirmed.

(continue in part 3)

Many thanks for your appreciation of our soundness and contribution! We hope the following analysis can address your questions about the verbalizer, baseline, law domain improvement, and another type of model.

Q1: How did you choose the verbalizer? How the verbalizer affects the performance?

Thanks for your in-depth suggestion, we conduct the following analysis.

Verbalizer Selection: Our choice of verbalizers for the Natural Language Inference (NLI) task type aligns with [1], where the effectiveness has been verified. Then, we observe that the entailment relationship in NLI can be extended to other task types like Commonsense Reasoning and Paraphrase Detection. In Table 3 in our paper, you may observe that the verbalizers for NLI are also applied to Commonsense Reasoning and Paraphrase Detection. Finally, we complement verbalizers for the remaining task types based on commonsense; for example, a verbalizer like "is defined as" can be commonly accepted to connect a concept with its definition.

Verbalizer Effect: Intuitively, our hypothesis is that a greater diversity of verbalizers leads to a broader range of comprehension tasks, ultimately enhancing performance. To test this hypothesis, we conduct an ablation by preserving only one verbalizer for each task type. Specifically, we preserve only the first verbalizer in each row in Table 3, and remake the reading comprehension data to observe the effect.

First, the most direct impact is that the number of mined task examples decreases. When using all the verbalizers (All Verbal), we mine an average of 2.1 examples per text, while using one verbalizer (Single Verbal) results in 1.4 examples per text.

Next, we conduct an experiment to train the general model using the new data, and the task results are as follows:

These results demonstrate that with fewer verbalizers, the downstream task performance declines, verifying that including more verbalizers can contribute to better performance.

Q2: The baselines are not comparable to the results reported by the authors. E.g., GPT-J 6B. It will be great to have the AdaptLLM result on top of the baselines wherever the models are publicly available.

We included GPT-J-6B as one baseline to facilitate a comparison with LexGPT-6B, which was trained from GPT-J-6B, and LexGPT-6B was the only publicly available large language model in the English-language law domain when we submitted our paper. To address your concern, we conduct experiments on GPT-J-6B and our method shows a positive average score compared to GPT-J-6B:

Considering that you may also be curious about our comparison with another baseline, MedAlpaca-13B, which is trained from LLaMA-13B, we conduct experiments on LLaMA-13B and observe favorable results.

We've updated Table 4 in our paper, so now that we can achieve the great goal to have the AdaptLLM result on top of the baselines wherever the models are publicly available!

(continue in part 2)