How Do Large Language Models Acquire Factual Knowledge During Pretraining?

We sincerely thank reviewer 1qMb for the detailed review and valuable feedback.

We will respond to the weaknesses and questions in order:

The interpretation about batch size may be too assertive. It is not certain that the retention rate will really go to 0 as the number of tokens increases. The interpretation is based on extrapolation.

We acknowledge the need for extended training to minimize extrapolation in Figures 4 and 5. However, given the significant computational burden of pretraining experiments (L690-691), such extensions are currently infeasible in our academic setting. We aim to address this with additional resources in future work. Importantly, Figure 16 (Appendix E.5) demonstrates the power-law holds near the x-axis for 1B models, which show faster forgetting than 7B models, supporting our interpretation's reliability.

The authors should also plot the relationship between batch size and effectivity.

The effect of reducing the batch size on effectivity is demonstrated in Figure 21 in Appendix G (Note: the LR was fixed for the experiments with the reduced batch size, and L246-248 & L725-727 are corrected in our latest version). Comparing Figure 2 with Figure 21, our results suggest that effectivity is increased with reduced batch size.

it is possible that when the batch size is larger than a certain number, the model will not be able to learn long-tail knowledge in the batch.

We acknowledge the need for experiments with various batch sizes (noted in Limitations, L504-506) to generalize to extreme values. However, our experiments with batch sizes used in actual OLMo pretraining (4M tokens) provide significant implications for recent LLMs in practice.

The evaluation metric (measuring the likelihood of the target sequence) may not effectively reflect whether the model acquires the knowledge, as the target phrase may not be the only correct continuation (at least it is unclear in the main text).

This is a good point! While the target span may not be the only correct continuation, we believe this does not have a significant impact on our interpretations:

• Knowledge acquisition should increase log probability for all correct continuations, including our target span.
• Using fictitious and factual knowledge mitigates the effects of multiple possible answers.
• Our focus is on relative comparisons across training conditions, not absolute likelihood increments.

These factors mitigate concerns about multiple-answer scenarios affecting our conclusions.

Line 70: [13] does not study NLP tasks!

Thank you for pointing this out. We will correct it.

the first paragraph in Section 4.2 contains information about model scales and pretraining stages. It's better to be re-organized as two paragraphs.

The purpose of the experiment, the purpose of the dataset, the design of the dataset, the way they generate the dataset, the inspection of the examples, should be in separated paragraphs.

We appreciate your thoughtful suggestions to make our paper more readable. We will improve them in the revised version.

In Section 3, the authors should provide more details about how the dataset is created.

Although we could not provide all the details in the main text due to the page limit, the detailed dataset generation process is in Appendix B (referenced in L98-99):

• Generation and filtering process (L509-535)
• Full examples in Tables 3 and 4
• Exact prompts used in Appendix C

We'll add more details to the main text in the revised version.

This paper defines many symbols but only few of them are used frequently throughout the whole paper.

Our symbol definitions have dual purposes: quantifying our experimental results and providing a framework for future studies on LLM pretraining dynamics. While this may slightly compromise readability, it ensures interchangeability between studies with different setups. In the revised version, we will enhance the clarity and simplicity of our metrics and symbols, enabling readers to easily grasp their meanings while maintaining adaptability. However, we believe that oversimplifying definitions could limit their applicability in future work. For example, defining $t_{LAM}(q,i)$ and $\mathcal{E}(q,i)$ without considering $i$ (the $i$-th encounter of knowledge) would make measured values incompatible with future studies using different experimental setups, such as varying injection intervals.

𝛿ℓ(𝑞) is not defined in the main text, but appears in two figures.

$\Deltaℓ(q)$ is explained in Figure 1's caption where it's first introduced. We will add its definition to the main text as well.

𝜃𝑡 is the parameter before the 𝑡-th update?

Yes, we will clarify this in the revised version.

In Figure 1, why can the number of training steps be minus?

In Figure 1, negative training steps represent the period before the model encounters the injected knowledge. We set $t=0$ as the reference point where the model is exposed to the minibatch containing the injected knowledge. For $t\leq0$, the model is updated only with the Dolma corpus.

In convention, 𝐸 is usually used for error. The authors could consider using another symbol for effectivity.

Thank you for the feedback. We will use an alternative symbol in the revised version.

We hope our responses have addressed your concerns. We appreciate your thorough review and remain open to any further questions.

We sincerely thank reviewer fwEj for the detailed review and valuable feedback. We will respond to each weakness and question in order:

the model trained with paraphrased knowledge show lower performance after learning on the semantic probe and the composition probe compared to the model trained with duplicated knowledge. This is counterintuitive as more variations in the training data should lead to better generalization

This point stems from a misunderstanding between overall model performance (i.e., generalization) and rapid acquisition of specific knowledge (measured by effectivity). Our analysis focused on the gap between memorization and generalization improvements, not absolute increases. We didn't interpret immediate improvement magnitude as an indicator of model performance. While duplication shows higher effectivity, it leads to a widening memorization-generalization gap over iterations. This results in the model favoring memorized content, potentially harming generalization (L295-299). Previous research has shown that LLMs trained on duplicated datasets tend to generate memorized content, which can be mitigated through deduplication [1].

but why it also learns faster?

The slightly larger effectivity for duplication likely stems from the greater lexical overlap between injected knowledge and probes, as we used injected knowledge as probe templates.

the learning rate, is not explicitly considered in the experiments.

Agreed, as noted in Limitations (L504-506). Given the computational intensity of pretraining experiments (L690-691), exploring learning rate effects is currently infeasible in our academic setting. We acknowledge the need for further investigation with additional resources.

could the gradual learning of facts be an artifact of low learning rate?

Experimental data with an unusually high learning rate (10e-3, see attached Figure 1) demonstrate that the accumulating behavior remains consistent, indicating it's not an artifact of a low learning rate.

How to tradeoff between faster learning and slower forgetting is not obvious from the results (at the end of the training session in Figure 2, duplicated knowledge still has a higher performance than deduplicated knowledge).

We don't interpret absolute log probability increases as performance measures. For instance, higher probabilities for target spans in memorization probes in the duplication scenario (Figure 2) don't necessarily indicate better performance. We claim that deduplication improves overall performance by mitigating memorization-generalization gaps, given sufficient training with deduplicated data.

Also, deduplication increases the average interval between exposures, which could cause more knowledge to fall outside the learnability threshold

Deduplication can actually bring more knowledge within the learnability threshold. We might first consider a synthetic case in which every instance is tripled. In this scenario, deduplication does not alter the expected exposure interval, as the dataset size decreases proportionally. In real pretraining corpora, most instances aren't duplicated, while few are highly duplicated ([1] showed this with the C4 dataset). Thus, deduplication typically reduces the expected interval for most knowledge, and deduplication can bring more knowledge inside the learnability threshold.

the authors should provide more description of the generated fictional knowledge

We provided main statistics in L92-93 and addressed the detailed dataset generation process and target span selection in Appendix B (referenced in L98-99). Space constraints limited main text details, but we'll clarify the dataset design further in the revised version.

how is the composition probe constructed, and what does "composing factual knowledge in multiple sentences" exactly mean?

We defined compositional generalization as “the ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge”. Composition probes were created based on this definition, as detailed in Appendix C.4 (L577-59). We will clarify this in the main text.

“Some important experiment details are missing: in line 156, what does the "paraphrased knowledge" refer to?

This is addressed in Appendix B. “paraphrased knowledge” is constructed by prompting GPT-4 to paraphrase each injected knowledge (L517-518), using the prompt in Appendix C.2 (L553-556).

Is it the same as the passages in semantic probe?

No. Semantic probes are separate GPT-4 paraphrases of each memorization probe sentence, maintaining the target span (Appendix B, L523-525). The specific prompt is in Appendix C.3 (L557-576).

How many paraphrases are used for each fact?

We used 9 paraphrases per injected knowledge (10 total injections: 1 original + 9 paraphrased), as described in Appendix B (L517-518).

although the observed forgetting fits to a power-law curve, extra care should be taken when extrapolating the curve to extreme values, such as to the vicinity of the x-axis

We agree caution is needed in extrapolation. However, Figure 16 (Appendix E.5) demonstrates the power-law holds near the x-axis for 1B models, which show faster forgetting than 7B models, supporting our extrapolation's reliability.

The relationship between model size and forgetting is not sufficiently explored

We explored this through comparisons of decay constants for 7B models (Table 2, Section 4.3) and 1B models (Table 7, Appendix E.4). Results show larger models are more robust to forgetting, aligning with previous findings.

We thank Reviewer fwEj again for their time and effort. We welcome any follow-up questions and are open to further discussion.

References

[1] https://arxiv.org/abs/2107.06499

We sincerely thank reviewer fwEj for their time and effort in reviewing our work.

We appreciate the reviewer's acknowledgment of the contributions of our work:

• The creation of FICTIONAL KNOWLEDGE dataset for controlled experiments on factual knowledge acquisition dynamics in LLM pretraining
• The comprehensive evaluation framework we developed, including metrics for assessing knowledge acquisition and retention
• The practical insights provided by our research for improving LLM training practices through empirical analysis
• The potential impact of our work on improving model reliability and performance, as well as its contribution to a more nuanced understanding of LLM pretraining

Regarding the "poor" rating for contribution, we respectfully seek clarification. Given the reviewer's positive comments on the novelty of our dataset, evaluation framework, and insights, we believe that our work makes a significant contribution to the field. Any specific feedback will be greatly appreciated.

Although the reviewer did not provide specific weaknesses or questions, we are open to further discussion on our work. We have addressed concerns and questions from other reviewers, and we would be happy to discuss these if it can be helpful for a more comprehensive evaluation of our paper.

We thank the reviewer again for their time and valuable feedback.

We sincerely thank reviewer ZYtT for acknowledging our contribution to studying the pretraining dynamics in LLMs, and commendation to our method and analysis regarding this topic. Also, we appreciate the reviewer’s effort to improve our work.

We would like to discuss each point in order:

It would be interesting to understand how much overlap of the proposed dataset there is with the pretraining dataset.

This is a good point! Understanding the overlap between our FICTIONAL KNOWLEDGE dataset and the pretraining corpus, and its impact on acquisition dynamics, would indeed provide valuable insights. While we haven't conducted a quantitative evaluation of how lexical and semantic overlap affects the acquisition of each knowledge piece, we hypothesize that such overlap primarily influences the initial log probability the model assigns to each probe's target span before encountering the knowledge. As the reviewer noted in their second point, the overlap between given knowledge and follow-up tokens may also affect forgetting dynamics. However, we believe these individual effects are mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Thus, we don't anticipate a significant impact on our core findings and interpretations.

It would be interesting to understand if forgetting is more or less pronounced when follow-on tokens are similar or different to the injected knowledge.

We agree this is an intriguing direction! Analyzing the influence of given knowledge's similarity to follow-on tokens on the forgetting dynamics will provide valuable insights.

Log-prob metric is clear, the others are a bit vague.

We acknowledge that some of our metrics may not be immediately intuitive. However, we would like to note that our primary focus on designing the metrics was making them well-defined and easily adoptable in future works on analyzing fine-grained pretraining dynamics with different experimental setups. This is of particular importance as this is one of the first works to study this, although it may come at the slight cost of clarity. We will strive to improve the clarity of our metrics in the revised version, ensuring readers can easily grasp the meaning of each metric while maintaining their adaptability.

How could you actually measure the "forgetting period" for long-tail knowledge?

As discussed in the footnote in Section 4.4 (L270), we didn't explicitly measure the forgetting period (which we interpret as the learnability threshold) for long-tail knowledge as our concept of learnability threshold is derived from our simulated setup. Therefore, the theoretical period we discuss may not exactly match the estimated x-intercept in Figure 5. However, we believe our results provide a reasonable estimate for relative comparisons of learnability thresholds.

We thank the reviewer again for their time and valuable feedback.

Thank you for taking time to respond to the weaknesses and questions.

Your rebuttal, while generally clear, throws up a few more questions due to quite surface level answers.

(1) we averaged all metrics across the entire dataset, and

Averaging does not answer the question of how much overlap there is (only if it was a `perfectly' balanced dataset). Could you provide an analysis or study of overlap between pre-training data (PTD) and Fictional Knowledge (FK)? Could you then provide additional results on how metrics differ for these categories?

(2) our primary focus is on relative changes in log probability. Thus, we don't anticipate a significant impact on our core findings and interpretations.

Similarly, here it would be very interesting to see how relative log probability changes behave under varying overlapping conditions. As other reviewers noted some information could be easily inferred by the LLM in general and therefore the change might be small for such knowledge (and the forgetting rate as well).

focus on designing the metrics was making them well-defined and easily adoptable in future works on analyzing fine-grained pretraining dynamics with different experimental setups

This needs further clarification and concrete examples and demonstrations.

We will strive to improve the clarity of our metrics in the revised version

Could you give an attempt already now?

Once again thank you for submitting the rebuttal, please elaborate on your answers and add more details. At this stage some things have become less clear.

We sincerely thank reviewer ewnz for their positive and thoughtful feedback. We appreciate the recognition of our contributions, particularly our insights on factual knowledge acquisition dynamics in LLM pretraining, as well as the commendation on our writing clarity and experimental design.

We would like to address each point in order:

Traces of Facts in Fiction

We appreciate the reviewer’s thorough investigation of our FICTIONAL KNOWLEDGE dataset! This is a good point, and we acknowledge the trade-off between the designed knowledge being ‘fictitious’ to minimize overlap between knowledge in a pretraining corpus, and still being ‘realistic’ not to invoke a significant distribution shift, which can cause our investigation to deviate from the true dynamics of acquiring factual knowledge contained in the pretraining corpus. We agree there is room for improvement in controlling this trade-off during dataset construction. We believe these 'traces of facts' primarily affect the initial log probability the model assigns to each probe's target span before encountering the knowledge, while it may also influence the forgetting dynamics. However, we expect these individual effects to be mitigated in our analysis because: (1) we averaged all metrics across the entire dataset, and (2) our primary focus is on relative changes in log probability. Therefore, we don't anticipate a significant impact on our core findings and interpretations.

Release of Checkpoints

Unfortunately, we could not store the intermediate model checkpoints due to storage constraints. Instead, we logged the model's logits for every instance in our dataset at each training step. We understand the computational challenges in reproducing our experiments and will release these log files to facilitate further analysis and ensure transparency.

The Mechanism of Knowledge Acquisition and Recall

Thank you for highlighting recent works on knowledge acquisition and recall. We will incorporate mentions and citations to these works in the main text. We believe that combining the spatial aspects of locating factual knowledge within model parameters with our temporal approach presents an exciting direction for future research.

We again thank the reviewer for their time and valuable feedback.