TiC-LM: A Multi-Year Benchmark for Continual Pretraining of Language Models

Clarification on Figure 4. We believe this happens because when using replay, the earlier months will be seen for more tokens throughout the continual run (as they are replayed the most amount of times). This is also why we see some trade-offs between ID and backwards performance when using replay or not in Table 2. In our updated manuscript, we have mentioned this more explicitly in Sec 6.1.

Impacts of replay. We’d like to clarify that one of our takeaways is that on some downstream evaluations, replay is actually crucial such as TiC-StackE-Math and TiC-CodeDocs-NumPy. For the evaluations that you mentioned, they either directly try to isolate the most recent information (TiC-Wiki-Diff) or cover domains/topics that are more likely to evolve quickly (StackOverflow, Pytorch). In general, we see the varying impacts of replay as an important finding from our study, one that results from the realities of pre-training on generic web-data (as opposed to the single-domain continual benchmarks from previous works) where the aim is to perform well across many domains but the relevant data for each may exist more prominently in certain time periods of Common Crawl.

Thank you for your time and thoughtful feedback! We hope to address your concerns and questions below.

Novelty of dataset/evaluations. Generally, we see the scale and scope of the TiC-LM as its most important contribution. To our knowledge, it is the first continual learning benchmark that captures two key aspects of LLM pre-training : (1) the time-evolving training data is generic web-data rather than limited domains such as Wikipedia/Tweets/News; (2) the evaluations span a variety of domains, reflecting the general goal of LLMs to be good across many possible contexts.

Based upon (1), we intentionally designed the training data to build upon the preprocessing of existing strong pre-training datasets (i.e., DCLM, RefinedWeb) so as to provide meaningful insights on realistic pre-training distributions, though alternative data-centric interventions would certainly be interesting as future work!

For evaluations, we see (2) as providing significant value on its own, especially as we showcase that different methods (e.g. replay) are more/less effective depending on domain. On top of this, we would like to highlight some technical contributions of our evaluations, which we described in the Appendix but will do better to highlight in future revisions:

• TiC-Wiki dramatically increases the evaluation timespan of the original TemporalWiki from 4 months to 10 years. Also, as discussed in more detail in App. B.1, we used own parsing to handle format changes and maintain temporal consistency across Wikipedia/Wikidata dumps. At a high-level, the approach used by TemporalWiki relies on hardcoded offsets to extract Knowledge Graph triples of Wikidata and requires web API requests to match Wikipedia pages with Wikidata IDs. Our approach, amongst other optimizations, instead systematically processes dumps of both Wikidata and Wikipedia to improve robustness and efficiency.
• TiC-StackExchange is a new evaluation we created which spans a wide time-range (>10 years) and diversity of topics/domains via the many different StackExchange sites. From a technical point of view, it involved careful reconstruction of historical post states which we share in our scripts.
• TiC-CodeDocs is the first evaluation based on API documentation evolution, requiring version-aware parsers to process 32 major releases of NumPy and PyTorch while handling complex format changes.

Overall, we plan to open-source our data generation scripts that will also allow for generating variations of evaluation sets (e.g. different time granularities) as well as the unified framework that we used to run all evaluations together.

Evaluation metrics. We primarily report perplexity results since our work focuses on base model evaluation without instruction tuning. Practically speaking, perplexity can be evaluated on any domain or data slice without any need for specialized formatting and also allows meaningful comparisons across domains (whereas QA-based evaluations would highly depend on the inherent difficulty of the QA creation method/format). In general, at our compute scales, the variance of perplexity metrics in our evaluations was also small enough to allow us to observe clearer differences between methods.

Regarding the contrast between TiC-CC-Wiki and TiC-Wiki-Diff, we believe this occurs because of the different ways they are constructed. The former is based on all tokens of any Wikipedia page in TiC-CC (exactly as they appear) whereas the latter differs in that it: (a) isolates specifically the most recently changed page segments; (b) focuses on factual information (by measuring perplexity only on proper nouns) (c) more comprehensively covers Wikipedia instead of whatever CC happened to crawl (which may over-represent some topics). Regarding factor (a), since the submission we also measured performance on TiC-Wiki-Unchanged (see Tab. 8 in App. D.2), which is based upon unchanged segments across Wikipedia dumps. On this evaluation, we observe that Replay + AR schedules return to offering the best backward transfer, though the gap is much smaller than on TiC-CC-Wiki. We speculate this could perhaps be due to factors (b) and (c). In our updated manuscript, we have added this discussion to Sec 6.2.

Hi Reviewer YDQj! As the discussion period is coming towards an end, we wanted to gently reach out to see if you had a chance to consider our reply. We'd very much appreciate knowing if it helped address your initial concerns, and if so, whether you might be willing to reconsider your score. We'd also of course be happy to answer any additional questions you might have!

Thank you for your time and thoughtful feedback! We hope to address your concerns and questions below.

Limited Insights. Generally, we see the unique scale and scope of the TiC-LM as one of its primary contributions. To our knowledge, it is the first CL benchmark that captures two key aspects of LLM pre-training : (1) the time-evolving training data is generic web-data rather than single domains such as Wikipedia/Tweets/News; (2) the evaluations also span a variety of domains, reflecting the goal of pre-training to be useful in many possible contexts. As such, we believe that even takeaways about methods which re-affirm existing works remain non-trivial. At the same time, we’d also like to highlight some takeaways that uniquely show up in our setting:

• Optimization methods:

  • We found it necessary to make the maxLR in each continual round at least 30x smaller than for the initial pre-training (see the expanded App. C). This is in contrast to recipes from Ibrahim et al.; 2024 and Gupta et al., 2023, who re-warm to up to a similar maxLR or Parmar et al., 2024 who start from the minLR of the pre-trained model and cool down further. We believe this is because their setups involve only 2 or 3 (roughly) equally sized training rounds and distribution shifts across data quality / language rather than many (likely more gradual) temporal shifts.
  • Since AR schedules roughly decay maxLR at a 1/t rate across rounds, they naturally result in a similar (though slightly better) performance profile to using a constant maxLR = minLR since our setup has >100 rounds.

• Replay methods: While other works have experimented with replay and found it effective for avoiding forgetting, our results additionally shed light on important challenges that arise when continually pre-training over long timeframes:

  • How should replay be implemented when there are many timesteps (>100) to balance? We find that $\alpha_t = 1/2$ offer much better plasticity than $\alpha_t = 1/t$, in contrast to TiC-CLIP which finds that different replay strategies perform similarly but only involves an order of magnitude fewer training rounds. Even for $\alpha_t = 1/2$, we find that while it can offer good balance between Backward and ID for a few years, it can result in increasingly reduced plasticity in the longer timescales captured by TiC-CC.
  • How does replay affect different domains when training on generically web-data? In contrast to prior works that train and evaluate on single domains (e.g., Jang et al; 2021), here we find that different domains change at different rates and that certain time periods of Common Crawl can be crucial for specific evaluations (e.g TiC-StackE-Math and TiC-NumPy v.s. TiC-Stackoverflow and TiC-Pytorch).

• Regularization methods: previous works that trained larger models often did not test LwF/EWC (perhaps due to the implementation challenges that we also faced, e.g. combining EWC with FSDP). In contrast to the more negative results from Garg et al.;, 2023 and Jin et al;, 2022, we found that EWC can be useful in our setup, resulting in the best performance on TiC-Wiki.

• A practical continual learning strategy: since the submission we have also added an additional analysis (see Sec 6.3) of the practical utility of our current methods. We find that when scaling up Replay ($\alpha_t=1/2$) + AR for 2x more training tokens, it can become competitive to re-training Oracle methods every two years despite still using 62% less total compute.

We have modified our manuscript (Sec 6.2, Sec 6.3, and App. C) to either add or make these points more explicit.

Unconvincing Experiments / Hyperparameter Sensitivity. Since the submission, we have expanded our discussion about hyperparameters in App. C. While we shared details about our hyperparameter search procedure (which we wanted to make realistic by using only the first 10 months of TiC-CC), we agree that showing some results for the different configurations tested would be helpful. Given the compute costs of each continual run, we unfortunately were not able to complete full runs for all the methods/configurations explored during tuning. That being said, we do now include several results on the tuning months as well as a comparison to full results for Cyclic Cosine. For EWC specifically, we copy the table below, where we observe $\lambda$ needs to be set quite high to have any noticeable effect but also enters a range where you trade-off backwards and ID performance. Our choice of $\lambda = 10^7$ strictly outperformed other choices except $10^6$ which had slightly better ID performance but considerably worse backward transfer. Going forwards, we are happy to further expand our study of hyperparameters, such as by training more tuning configurations to completion!

Poor Presentation. Thank you for pointing this out. In our revised manuscript, we have modified the Experiments section to make the main text less reliant on the captions and better highlight the key takeaways in relation to existing works. We also welcome any additional suggestions you might have!

References

[1] Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. TMLR 2024.

[2] Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023.

[3] Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263, 2024

[4] Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. EMNLP 2022.

[5] Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.

[6] Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. NACCL 2022.

Thank you for considering our response. We appreciate the chance for continued discussion and hope to address the points from your latest reply below:

The experimental section reads more like a descriptive experimental report...Additionally, I believe it is necessary to highlight the modifications in the manuscript using a different color to improve clarity.

During the rebuttal period and after further integrating your feedback, we've re-formatted Sections 6.1 and 6.2 to better highlight the novel takeaways and added more discussion about their relationships to existing work. In the latest upload, we have also marked the new/changed text in blue, (apologies if it was confusing earlier). We hope these changes help address your concerns about presentation and of course would welcome any additional suggestions!

Neither Table 2 nor Table 3 demonstrates that any particular strategy consistently outperforms others across different settings.

We see the different trade-offs between methods as an important result reflecting the inherent and realistic challenges of trying to continually pre-train LLMs from general web-data while also aiming to achieve strong performance across many domains. Because different domains evolve at different rates (and may be over/under-represented in different months of CC), the natural trade-offs between forgetting/plasticity that different methods (e.g., replay) induce will lead to corresponding trade-offs across evaluations. We believe due to its novel scope, our benchmark is the first to capture and highlight these realistic challenges (e.g., compared to older efforts that trained and evaluated on single domains.) Furthermore, we find that despite these difficulties, we are still able to identify a continual method that, while not the best individually on every evaluation, demonstrates significant practical utility overall. As mentioned in our first response, our new results in Sec 6.3 show that Replay ($\alpha_t = 1/2$) + AR results in a competitive alternative to periodically re-training Oracle models while being 2.6x more compute efficient.

Moreover, setting $\alpha_t = 1/2$ seems trivial. This hyperparameter is likely influenced by both the ratio of data from new tasks versus old tasks and the degree of overlap between new and old tasks. Consequently, I find it difficult to identify a practical strategy for continual pre-training on other datasets.

In our study, we chose to try $\alpha_t = 1/2$ and $\alpha_t = 1/t$ exactly because they were (a) simple/intuitive strategies that did not require careful tuning (which indeed might risk overfitting to our setup); (b) found to be effective choices in TiC-CLIP (there named Cumulative-Equal and Cumulative-All). As mentioned, our new results in Sec 6.3 show that despite its simplicity, $\alpha_t = 1/2$ can already result in 2.6x practical efficiency gains. Further, even within the 11 years covered by our dataset, it is quite likely that the data changes more/less quickly in some time periods. This suggests that $\alpha_t = 1/2$'s effectiveness may already show some robustness to natural variations in how fast data evolves.

At the same time, we do agree that varying $\alpha_t$ based upon your data is something that could yield even better results. In the remainder of the discussion period, we are happy to try running with other fixed settings of $\alpha_t$, while more complex/adaptive strategies for setting $\alpha_t$ would certainly be interesting directions for future work!

To follow up on our promise to try different $\alpha_t$ values, we have now completed a set of runs for $\alpha_t \in \\{0.1, 0.3, 0.5, 0.7, 0.9\\}$. By doing a controlled sweep over the amount of replay used, we find that:

• Different evaluations benefit from more/less replay (reinforcing some of our earlier takeaways about Replay)
  • On TiC-CC subsets, we see trade-offs between Backward and ID. More replay helps on Backward but does worse on ID.
  • On downstream evals, the optimal $\alpha_t$ varies. Lower $\alpha_t$ (more replay) helps more on the slower-changing domains (e.g., TiC-CodeDocs-NumPy) and higher $\alpha_t$ on faster moving ones (e.g., TiC-CodeDocs-PyTorch)
• Using $\alpha_t = 1/t$ is generally sub-optimal, as we now see it is typically dominated by $\alpha_t = 0.1$ on all three metrics. As mentioned earlier, this is a notable difference from the findings in TiC-CLIP, which pushes forward the $\alpha_t = 1/t$ variant of replay, which we believe scales more poorly here given the >10x larger amount of timesteps captured by TiC-LM.
• Given the trade-off across evals, amongst fixed choices of $\alpha_t$, using a middling value like $\alpha_t = 0.5$ (or slightly higher) seems to be a reasonable practical recommendation to avoid performing poorly on any one evaluation/domain. Of course, there's certainly room for future work to develop more sophisticated replay methods that are adaptive (e.g., determining $\alpha_t$. based upon monthly loss histories) and/or data-dependent (e.g., replaying only certain subsets of older data)!

Here, we share a portion of the new results and in future revisions, while we plan to include the complete set into our manuscript in future revisions!

Thank you for your time and thoughtful feedback! We hope to address your concerns below.

Novelty of findings. We appreciate you recognizing the uniqueness and potential of our benchmark! Regarding our findings, we believe that given the novel scale/scope of our setting (as you pointed out) even takeaways about methods which re-affirm existing works are non-trivial. At the same time, we’d also like to highlight some takeaways that uniquely show up in our setup:

• Optimization methods:

  • We found it necessary to make the maxLR in each continual round at least 30x smaller than for the initial pre-training (see expanded App. C). This is in contrast to recipes from Ibrahim et al.; 2024 and Gupta et al., 2023, who re-warm to up to a similar maxLR or Paramar et al., 2024 who start from the minLR of the pre-trained model and cool down further. We believe this is because their setups involve only 2 or 3 (roughly) equally sized training rounds and distribution shifts across data quality / language rather than many (likely more gradual) temporal shifts.
  • Since AR schedules roughly decay maxLR at a 1/t rate across rounds, they naturally result in a similar (though slightly better) performance profile to using a constant maxLR = minLR since our setup has >100 rounds.

• Replay methods: While other works have experimented with replay and found it effective for avoiding forgetting, our results additionally shed light on important challenges that arise when continually pre-training over long timeframes:

  • How should replay be implemented when there are many timesteps (>100) to balance? We find that $\alpha_t = 1/2$ offer much better plasticity than $\alpha_t = 1/t$, in contrast to TiC-CLIP which finds that different replay strategies perform similarly but only involves an order of magnitude fewer training rounds. Even for $\alpha_t = 1/2$, we find that while it can offer good balance between Backward and ID for a few years, it can result in increasingly reduced plasticity in the longer timescales captured by TiC-CC.
  • How does replay affect different domains when training on generically web-data? In contrast to prior works that train and evaluate on single domains (e.g., Jang et al; 2021), here we find that different domains change at different rates and that certain time periods of Common Crawl can be crucial for specific evaluations (e.g TiC-StackE-Math and TiC-NumPy v.s. TiC-Stackoverflow and TiC-Pytorch).

• Regularization methods: previous works that trained larger models often did not test LwF/EWC (perhaps due to the implementation challenges that we also faced, e.g. combining EWC with FSDP). In contrast to the more negative results from Garg et al.;, 2023 and Jin et al;, 2022, we found that EWC can be useful in our setup, resulting in the best performance on TiC-Wiki.

• A practical continual learning strategy: since the submission we have also added an additional analysis (see Sec 6.3) of the practical utility of our current methods. We find that scaling up Replay ($\alpha_t=1/2$) + AR for 2x more training tokens, it can become competitive to re-training Oracle methods every two years despite still using 62% less total compute.

We have updated our manuscript (Sec 6.1, 6.2, Sec 6.3, and App. C) to either add or discuss these points more explicitly!

Complexity of setup. We plan to share all code for reproducing TiC-CC from Common Crawl as well as the training code for all methods. We will also provide model checkpoints for the initialization, oracle, and various continual runs. For lower-resource research groups, our scripts can be used to generate smaller subsets for continual pre-training with fewer tokens/months/model parameters while our pre-trained checkpoints can also allow for evaluating methods when using only a few months of additional training. We’re happy to provide some additional experiments in these regimes in future revisions.

Sensitivity to hyperparameters. Since the submission, we have expanded our discussion about hyperparameters in App. C. While we previously shared higher-level details about our search procedure (which tunes on the first 10 continual months), we agree that showing results for different configurations would be helpful. Given the compute costs of each continual run, we were not able to complete full runs for all the methods/configurations we previously only ran for the tuning months. However, we were able to do so for the example you asked about re: LRs for Cyclic Cosine (copied below) and also included results on the tuning months for some other methods, observing for instance:

• Max LR in each cyclic schedule needs to be low enough but once it reaches that point, lowering it further can trade-off backwards and ID performance. However, even a very small or AR-decayed max LR does not offer better backwards transfer compared to using replay.
• EWC’s regularization weight needs to be set quite high but also enters a regime where you start to trade-off backwards and ID performance. Our choice of $\lambda = 10^7$ strictly outperformed most other choices except $10^6$ which had slightly better ID performance but considerably worse backward transfer.
• For replay, we have not yet tried anything beyond the two configurations we mentioned ($\alpha_t = 1/t$ and $\alpha_t = 1/2$), with the takeaways already described in the above respone.

Going forwards, we are happy to further expand our study of hyperparameters (e.g., training more tuning configurations to completion)!

References

[1] Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. TMLR 2024.

[2] Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023.

[3] Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263, 2024

[4] Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. EMNLP 2022.

[5] Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.

[6] Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. NACCL 2022.

Thank you for your time and thoughtful feedback! We hope to address your two main concerns below.

Novelty + Closing gap to Oracle. We agree the ultimate goal is to find practical continual learning methods. To that end, we see our more realistic benchmark, which is the first of appropriate scale/scope for LLM pre-training (i.e., using generic web-crawled training data and evaluations from multiple domains), as an important pre-requesite that is missing from the current literature.

Regarding our methods not matching the Oracle, we note that this Oracle is quite strong for two reasons: (1) while it is compute matched with full continual runs, it has seen considerably more tokens compared to most continual checkpoints in our eval matrices (up to 2x for earliest months); (2) it would not actually be obtainable until the very last month; if one wanted to consistently have up-to-date models re-trained like the Oracle, the costs would increase linearly with the number of desired checkpoints. Since the submission, we have run additional experiments to better contextualize the cost effectiveness of our current continual strategies relative to re-training Oracles:

• Token-matched oracles: We consider a baseline which is a series of 7 oracles based on different cutoff dates. Each is token matched with the existing 220B continual runs based upon cutoff date (e.g., a 2019-01 oracle is trained on all data up to 2019-01 for the same number of total tokens seen by a continual run’s 2019-01 checkpoint). We use the cutoffs {2013-05, 2015-01, 2017-01, 2019-01, 2021-01, 2023-01, 2024-07} which combined see 1.16T tokens. When measuring sub-optimality for continual checkpoints against this series, we now subtract the performance of the most recent Oracle (instead of always 2024-07).
• Scaled-up continual runs: We try scaling up continual runs by increasing the budget allocated to each month in the continual phase by 2x and 3x (while keeping the same initialization trained on 110B tokens). These runs see 330B and 440B total tokens.

We add these new results in Sec 6.3 of our updated manuscript (and copied below), where we report the average of the Backwards and ID elements of the matrices for different evaluations. Overall, we observe that despite requiring about 62% less compute, the longest 440B continual runs are fairly competitive with the series of the Oracles, surpassing it on all TiC-CC and TiC-Wiki evaluations and closing the gap on most others. This demonstrates that our continual methods already showcase practical relevance, though we hope further development using our benchmark can yield even larger efficiency gains! In future revisions, we're happy to expand this analysis (e.g., obtain results for scaling up other continual methods).

Leakage between training and evaluation sets in TiC-CC. We agree that this is a potential concern, which we also noted previously in App. A. To clarify, we held out test examples at a document-level, but certainly the contents of particular pages may overlap. To test whether potential leakages affect our results, we have now also run an additional set of TiC-CC evaluations on decontaminated versions of the evaluation sets, where we used the same Bloom Filter approach to deduplicate the evaluation sets (except we now pre-populate the Bloom Filter with the corresponding training set). We then plot the two loss values (relative to the Oracle) given by the original and decontaminated evaluation sets for all (method, checkpoint, evaluation month) triplets in Figure 6 in Appendix A . Overall, we observe that the results are highly correlated, suggesting that data leakages do not affect our conclusions.