Efficient Continual Pre-training for Building Domain Specific Large Language Models

We would like to thank the reviewer for their insightful comments and taking the time to review our work.

We agree with the reviewer that our work only demonstrates the proposed methods on Pythia and not on any other LLMs, which is unfair to other LLMs like OPT. However, the main goal of our paper was not to compare between different LLMs but to demonstrate:

• Effectiveness of continual pre-training: The focus of the paper is the comparison between Pythia and FinPythia models, with FinPythia performing better than Pythia.
• Effectiveness of Data selection methods: We demonstrate that using data selection methods, we can perform than the vanilla pre-training.

Response to questions While we would love to do continual pre-training and ablation or LLaMA 7B and OPT/BLOOM, the costs for doing so are quite prohibitive. It took us nearly 20 days to train Pythia 7B model on one P4.24xlarge instance we have access to. We had to scale down to Pythia 1B for our data selection methods because of these costs, to compare between our proposed data selection methods. Note, GPT-J was trained by EleutherAI as well, the predecessor of Pythia. The main reason of selecting Pythia was its availability of different sized models: 14 m to 12B which allowed us this flexibility of comparing continual pre-training at different sizes as well as out results with perplexity selection, as perplexity from Pythia 70M model has a correlation of 0.97 with Pythia 1B/7B models, allowing us for easy data selection. This flexibility is not available in other model families, where 7B is typically the smallest model size.

We hope this clarifies the reviewers concerns about not doing continual pre-training on other LLMs.

We would like to request the reviewer to re-consider their revision. Absence of ablation study with other LLMs does not limit the strength of this work, as the main claim of the paper is the effectiveness of the data selection method results presented in Tables 3 and 4. We are NOT making making a claim that other LLMs when continually pre-trained would do better or worse than FinPythia. Table 2 results of comparison with other LLMs (BLOOM, OPT, GPT-J, BloombergGPT) is just presented for benchmarking; it does not add or take away from the central claims of this work.

Doing a continual pre-training of LLaMA-2, BLOOM, GPT-J does not have a bearing on our claims. Our central claim is how performing data selection increases the effectiveness of domain continual pre-training. So the main comparison is with all the various data selection methods by fixing a base model (Pythia).

We would also like to emphasize about the costs involved here to run each data selection (10 rows in Table 3) for each model of size 7B. All the techniques we have presented (similarity-based, entropy-based, and perplexity-based data selection) use smaller models to be of practical use to typical researchers with low compute resources, democratizing LLMs. We're unable to perform the experiments with 7B models with our limited resources. This raises the bar too high for any work in this space and disadvantages researchers in LLM community not having access to significant compute resources at their disposal. This is what this paper is helping to address by presenting a method that allows researchers to create domain LLMs without having to train from scratch which only industrial labs with million dollar budgets can do like BloombergGPT.

We would really appreciate if the reviewer can re-consider their decision, as this work opens up the space for researchers with limited compute budgets.

Thank you for your clarification. We agree that these experiments would strengthen the paper but we do think that it would be too strict to reject the paper because of these experiments. Most of the papers that have been accepted in the past in ICLR use a single LLM model for their experiments. For instance:

• Chain-of-thought reasoning [1] only shows the CoT on GPT-3 with in-context learning
• Compositional task representations [2] demonstrate the experiments on a T5 with in-context learning
• Compositional Semantic parsing with LLMs [3] uses InstructGPT for all the experiments on a single task of semantic parsing
• Ke et al [4] only used RoBERTa as the base model for continual pre-training

Notably, none of these experiments in these papers needed high computational resources unlike our case for pre-training as they use in-context learning or smaller models.

We think this increases the bar too high, limiting the exploration in this area by rejecting the work due to lack of these additional experiments. In an era, where large industrial research labs are not openly sharing details about pre-training, too high barriers would be demotivating for open source community with limited resources.

We appreciate your time and responses, and would request you again to re-consider, if possible. Thank you once again for your thoughtful review and responses.

[1] Zhang, Z., Zhang, A., Li, M. and Smola, A.. Automatic Chain of Thought Prompting in Large Language Models. ICLR 2023.

[2] Shao, N., Cai, Z., Liao, C., Zheng, Y. and Yang, Z.. Compositional task representations for large language models. ICLR 2023.

[3] Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O. and Zhou, D.. Compositional Semantic Parsing with Large Language Models. ICLR 2023.

[4] Ke, et al., Continual Pre-training of Language Models. ICLR 2023.

We really appreciate reviewer’s feedback on our paper and pointing out a significant weakness in the paper.

We agree with the reviewer that Domain-adaptive pre-training (DAPT) is not a novel concept and has been studied in the realm of traditional language models. These works [1, 2], however are limited to using smaller LMs like Roberta for continually pre-training on much smaller pre-training datasets like ACL papers.

• These methods are not feasible for LLMs because of cost and run-time considerations, as they select sample importance while pre-training. For example, DAS [1] costs 300% over the vanilla pre-training, while our method costs 10% of vanilla pre-training. Not to mention the difficulty in infrastructure needed: DAS/DGA needs two copies of the model in memory as they calculate the importance of a sample based on different dropout ratios for the same model to have comparable run times to conventional pre-training. Otherwise, the run time for pre-training is in the orders of vanilla pre-training’s run time by swapping different configurations of the model sequentially.
• While our work does not have the complexity, the challenges from creating one of the largest domain specific corpus and the costs of comparing different methods of data selection at this scale in our resources were not trivial. Our work takes a novel approach of filtering the samples before the pre-training versus filtering while pre-training, which is not feasible at the scale of LLM pre-training. Our method costs less than 3.33% of DAS/DGA.

We have added this discussion in Section 5 under continual pre-training of language models section. Thanks once again for pointing this out, and we hope this clarifies why our work while being simpler is significant for LLM field.

[1] Ke, et al. "Continual Pre-training of Language Models." ICLR. 2023.

[2] Ke, et al. "Adapting a Language Model While Preserving its General Knowledge." EMNLP. 2022.

We would first like to thank the reviewer for their insightful questions by reading the paper carefully and raising valid points.

We would first like to respond to the weakness mentioned by the reviewer:

• The NER task is a generation task. We did experiment on ConvFinQA task, which is a QA type task on Finance domain. However, all the methods scored near zero with it as this task needs a really long context window for generation. Most of the other tasks for Finance are similar to ones we experimented with.
• We agree with the reviewer that the paper is mostly an empirical paper, though we did discuss the intuition behind our data selection in Section 2.4 with existing theory, which is indeed demonstrated in the empirical conclusions; similarity based data selection is indeed the best. This kind of work is first in the LLM domain where theory is limited.
• We agree with the reviewer that knowing task data is not usually possible in real world. Hence, our proposed task-agnostic methods are much more useful and practical in the real world LLMs. We have added this discussion in Section 4.3 under Comparison of Data Selection Metrics. If there is a task or group of tasks available, similarity based method can be used.

Response to Questions:

• Thanks for pointing it out. We have added the discussion about Ke et al’s work in Section 5 under continual pre-training of language models section. In summary, Ke at al’s DGA [1] and DAS [2] works need multiple passes over the entire pre-training corpus, costing 3x of vanilla pre-training, which is infeasible for LLMs and the large amount of data used in our experiments. Our methods pre filter the samples even before pre-training starts, while DGA/DAS filter the sample’s importance during the pre-training step. Our methods only cost 10% of vanilla pre-training: 3.33% of cost of training with DGA/DAS, which is quite substantial given the cost of training LLMs.
• Thank you for pointing it out, we have reworded the abstract to make it more clear. Hope it is not confusing anymore to the readers.

[1] Ke, et al. "Continual Pre-training of Language Models." ICLR. 2023.

[2] Ke, et al. "Adapting a Language Model While Preserving its General Knowledge." EMNLP. 2022.

We would like to thank the reviewer for such a thorough review with a careful reading of our work. We really appreciate your insights and have tried to address your valid concerns in the new revision. We hope these have improved the quality of our work significantly.

Response to the weakness by reviewer

• We have rewritten and added details in sections (listed below) as rightly pointed out by the reviewer. We have simplified the Tables 2, 3, and 4 based on the suggestions from the reviewer.
• The results of the paper are only presented on 10% and 100% of the domains-specific corpus: We have added Figure 2, with ablation on % of pre-training data selected vs performance and discussion of these results (Section 4.3). In summary, we see a saturation of performance between 5%-10% of training data selection, which starts to degrade for both similarity based (ETS-DACP) and entropy based (ETA-DACP-ent) beyond 10%.
• We agree with the reviewer on the high std deviation in Table 3. We actually found that standard deviation of results in Table 2 was higher as 5-shots are selected from the entire test set randomly and selected examples can make a huge difference on quality of results. To keep these numbers comparable to ones in the literature, we kept the evaluation framework same in Table 2. In order to make a fair comparison for data selection methods, we fixed the pool of shot samples to 50, which is same across all the methods for our results in Table 3. Hence, the difference in results for Pythia/FinPythia in Table 2 and 3. We have added a note on this in Table 3. This provided lower standard deviation, which is not typically reported by other works in the literature. However, we are confident on these conclusions because of trends (Table 3) and loss functions (Section C in Appendix) matching our intuitions, exception being perplexity based results, which are explained next with our investigation.
• We have added a discussion of comparison between the different metrics in Section 4.3 under Comparison of Data Selection Metrics. Further, we have added Figure 3 in Appendix C that pictorially depicts task aware vs task agnostic data selection and tied it to an expanded discussion in Appendix Section C about loss curves for different metrics to show that loss curves have an expected behavior based on the way sampling is being done. In summary, to answer review’s question about why perplexity based selection leads to poor performance , we observed a lot of long form tables in the top perplexity samples beyond the top 1% perplexity samples, which indeed can be noisy for the model to learn on, as pointed out correctly by the reviewer. These samples drop the performance of perplexity based method (presented in ablation in Figure 2) until 5% of data selection, after which it starts to recover. In a higher quality dataset, perplexity could still be a good measure but owing to noisy samples scoring higher on perplexity/novelty, it is not a good metric for data selection. Your intuition is right that similarity and entropy based methods avoid these noisy examples as shown by the low correlation 0.21 and 0.14 with perplexity, respectively, in Figure 6 in Appendix.

Response to the questions We have rewritten and added sections which were not clear as rightly pointed out by the reviewer:

• FiQA SA Results: We believe that it is not an outlier. As we target a portfolio of tasks, the continual pre-training method may affect the tasks unevenly. Moreover, the mixing of the task dataset might play an uneven role in data selection as well.
• TACP: We have expanded on this section and elaborated on the difference between TACP and DACP/fine-tuning. Further, we have added Table 1, which gives a summary of data selection metrics and data sources used by all the methods used in this work + Fine tuning.
• Soft Sampling: We have revised section (2.4.3) by explaining soft sampling as well as adding an example. We have also added a discussion about effect of soft versus hard sampling on the loss curves of our proposed methods in the Appendix C, to give a further intuition of how soft sampling makes a difference to what model sees - effect of which is quite prominent on previously confined ETS-DACP whose loss resembles other methods with soft sampling as it now sees the wider corpus distribution.
• Simplified Table 2 (former Table 1), Table 3, and Table 4. We further added deltas in all the tables to make it easier for the readers to follow. Thanks for this suggestion, it really helps the reader without getting lost in individual columns.
• We have added description about the four open domain tasks in Section 3.1 with relevant papers.
• Former Table 2 with qualitative examples, has been moved to Appendix. Thanks for this suggestion, this freed up a lot of space to explain the unclear sections.