How new data permeates LLM knowledge and how to dilute it

We thank the Reviewer for their insightful question and their appreciation of our work. We are delighted to answer their question with a “yes” as we have indeed explored the behavior of other models beyond PALM-2, specifically Llama and Gemma. Interestingly, we observed striking differences in their priming dynamics during learning, despite all models exhibiting a relationship between pre-learning keyword probability and the degree of priming. These distinct behaviors, while fascinating, were difficult to interpret due to the numerous architectural and training differences between these models.

Inspired by the Reviewer’s question, and to isolate the potential impact of fine-tuning on priming, we conducted additional experiments with an LLM fine-tuned on the FLAN dataset (new Fig. 20, new pdf attached). This allowed us to directly compare a pre-trained PALM-2 model with a fine-tuned version sharing the same fundamental architecture. This controlled comparison revealed a key finding: fine-tuning disrupts the correlation between priming and memorization. We hypothesize that this may be due to a collapse of the hypothesis space during fine-tuning, affecting how the model incorporates new information.

This new analysis, motivated by the Reviewer’s question, provides valuable insights into how different training procedures can influence priming dynamics. We believe these findings contribute significantly to understanding the complexities of priming in large language models and have implications for mitigating its potential negative effects.

We thank the Reviewer's appraisal of our study; their question already inspired further experiments which we discussed above. If there are any further questions about our work, or suggestions that may lead to further increases in score, we hope the Reviewer will feel free to let us know.

Thanks for your response. The response has addressed my main concerns. I will increase my score to 8.

3/3

Answers to Reviewer questions:

• Q1: We apologize for the typo. There are indeed 11 categories, as listed in Appendix A.1. We have corrected this typo in the manuscript in all places.
• Q2: Learning was terminated after 20 iterations for experiments using the Alpaca dataset and 40 iterations for experiments using Wikipedia. This decision was based on initial pilot experiments and maintained for consistency across all relevant comparisons. To ensure uniformity, we have conducted new experiments on Wikipedia with 20 iterations and observed similar results (see new Fig. 21 in the uploaded pdf).
• Q3: The three LM families (PALM-2, Llama, and Gemma) were selected based on their availability within the Pax framework for training neural networks and their compatibility with our hardware constraints (2x2 Viperfish TPU).
• Q4: Unfortunately, the internal policy of many industry labs no longer allows to explicitly share the number of parameters, so, similarly to PALM-2 research report, we not able to share them either.
• Q5: We appreciate the Reviewer's suggestion to group experiments by keywords. While this approach could potentially reduce the number of experiments, we were concerned about potential interference effects even between samples with different keywords. As our findings demonstrate, the surrounding context, including potentially abstract semantic or thematic elements, can influence priming. To minimize these potential confounds and maintain the integrity of our experiments, we opted to train each of the 1320 experiments individually.
• Q6: "3 presentations" refers to the total number of times the Outlandish sample appeared within the training data. Each presentation consisted of inserting the sample into a minibatch of 8, with a spacing of 20 minibatches between presentations. This resulted in a total of 480 samples being processed in the period of these 3 presentations of the Outlandish text.
• Q7: We acknowledge that priming is inherently measured post-learning. The phrase "priming post-learning" was used to emphasize the distinction between measuring priming after learning versus measuring the probability of keywords before learning. Our work highlights the predictive relationship between these two measures, a novel and useful finding that we believe would be of significant interest to the community.
• Q8: "Succinct real facts" are highly compressed versions of "real facts." We have systematically include examples of facts for each of the 11 categories in Appendix A.2 (along with the examples shown in Figure 1) .
• Q9: Examples of augmentations have now been added to Appendix A.6.
• Q10: The minibatch size was 8. While the range of K (1 to 50) may appear excessive, we aimed to thoroughly investigate the relationship between spacing and priming. This broad range allowed us to systematically explore the effects of varying the frequency of sample presentation.
• Q11: Yes, the 20 to 40 minibatches are consecutive in the default setup.

Altogether, we are very heartened by the Reviewer’s deep dive into our manuscript, and their resulting comments have sharpened the manuscript's clarity. These lead us to endeavour to complete a number of new experiments and analyses which we showcase here, and resulted in deeper insights on the nature of priming, its relation to other metrics, and deeper digging into the puzzling priming v memorization dynamics observations.

We hope the Reviewer will recommend acceptance for this paper on account of its novel, puzzling, and robust experimental results and mitigation algorithms, as well as the sharp insights and new experiments that we and they jointly thought of, which we believe will benefit the ICLR community. We stand ready to address any remaining questions.

2/3

W3: Clarity and Presentation

We thank the Reviewer for their critique on presentation and clarity. We acknowledge that the initial manuscript relied heavily on the Appendix for detailed training setups, which may have hindered readability. In response, we have now successfully and thoroughly made the following changes:

• Training Details: We will have moved relevant detailed descriptions of the training protocol (previously Appendix Section A.3) into the main text (Section 3.2), ensuring essential experimental setups are self-contained.
• Figure 2b: The axes and significance of blue dots are now clearly defined in the caption.
• Figure 7: The terms “matched” and “unmatched” are explicitly explained in the main text and the figure caption --with a new figure panel (7b) illustrating this.
• Figure 8: Replaced with a strip plot for clearer visualization, following the example of Figure 9.
• Figures 10 and 11: Enlarged for better readability, with t-test results, and significance of findings. Missing p-values have been added.

We hope these revisions address the Reviewer’s concerns and significantly improve the manuscript’s clarity and accessibility, and if there are further issues, please let us know.

We sincerely appreciate the Reviewer's careful reading of our manuscript and their valuable feedback. We apologize that certain portions were unclear and frustrating, and have made every effort to address the Reviewer's concerns with detailed revisions. Our goal is to ensure the clarity and accessibility of our work, allowing its novel findings and surprising insights to be fully appreciated by the Reader. We have addressed each of the Reviewer's questions below:

W1: Distinguishing Desirable and Undesirable Priming

We sincerely thank the Reviewer for highlighting these other metrics. While they were outside the initial scope of our work, we now recognize that engaging with related metrics, such as locality and portability, strengthens our contributions and connects our study more robustly with the knowledge-editing literature. To address this, we have conducted new experiments adapting the definitions of locality and portability to the free-flowing text forms in the Outlandish dataset. Specifically:

• Locality Experiment: We tested whether rewrites of Outlandish facts could induce recall, retaining the contextual link to the original text, inspired by the formulation used in Meng et al NeurIPS 2022 and Yao, et al 2023 EMNLP.
• Portability Experiment: We examined whether keywords could transfer priming effects when the inserted text was reversed in subject order, inspired by the equivalent metric used in Yao et al 2023, EMNLP where the portability measure was introduced.

These results demonstrate that our proposed priming metric correlates with both locality and portability measures, reinforcing its validity as a complementary metric. We have incorporated these findings into the manuscript, adding a new figure (Figure 29) in the newly attached pdf, and have completed their incorporatation into the Related works section of the manuscript (Section 2.4).

An important note, however, is that the priming metric captures factuality-independent notions of knowledge propagation and is applicable to free-flowing text, and therefore perfectly complements existing metrics that focus on canonical (subject, relation, object) forms -- most of these contemporary metrics including specificity, locality and portability, do within their original formulations. By focusing on statistical regularities in such a wide diversity of texts, priming opens avenues for elucidating LLM behavior in broader, real-world scenarios.

Finally, we emphasize that our metric addresses a novel research gap: predicting priming and such information spread, after learning using a metric before learning and our discovery that keyword probability before learning robustly predicts priming after learning is one of our showcased findings. This predictive focus, too, well complements existing metrics that (retrospectively) characterize information spread.

W2: Priming and Memorization Coupling

We appreciate the reviewer's comment regarding the coupling of priming and memorization. Our initial investigations revealed an intriguing phenomenon: while all examined models (PaLM-2, Llama, and Gemma) exhibited a relationship between pre-learning token probabilities and priming, their specific learning dynamics varied significantly. This suggested that factors like architecture or training procedures could play a crucial role in how priming and memorization interact, but the many differences between these models makes it difficult to pinpoint the underlying mechanisms.

To further explore this, and inspired by the Reviewer’s comment, we endeavored to conduct new analyses using a fine-tuned LLM based on the FLAN dataset. This new figure allowed us to directly compare a pre-trained PALM-2 model with a fine-tuned version sharing the same fundamental architecture. By controlling for architecture, we could effectively isolate the impact of fine-tuning on the relationship between priming and memorization. Our findings reveal that fine-tuning alters this dynamic, perhaps due to a narrowing of the hypothesis space during the fine-tuning process (new Figure 20). This highlights the candidate significant influence of fine-tuning on how models learn and retain information.

While our work provides compelling evidence for a robust correlation between pre-learning token probabilities and priming across diverse architectures, a complete mechanistic understanding of this phenomenon remains an open question. We agree with the Reviewer that further investigation is crucial, particularly in exploring how architectural variations and training procedures contribute to the observed differences in priming dynamics, and our conclusion is that these puzzling findings would be very intriguing for the ICLR community when opened up for further investigation by the field.

2/2 continued

... Our ignore-topk modulation strategy offers an approach that is potentially more nuanced than simply removing data, enabling us to address the challenges of priming in a wider range of applications, and we are excited to hopefully share this unexpected finding with the ICLR community.

Dataset Size and Utility:

We thank the Reviewer for highlighting the sample size of the 12 keyword tokens. While each token is associated with over 100 Outlandish texts, (since one of our highest curiosities was whether the same keyword token could cause differing amounts of hallucination under different circumstances - the answer is affirmative!), we absolutely agree that future work should expand the dataset both in the number of tokens (12) and the diversity of contexts (> 100 each). This is now acknowledge in the new Limitations section.

In response to the Reviewer’s discussion on how to compute a general measure of "pollution," we propose adapting the priming metric itself, which quantifies how newly learned information propagates to unrelated contexts. We have reported new experiments above how it can be explicitly adapted to indicate the amount of “pollution” i.e. undesirable priming. Here, we may report a 2nd compelling experiment: we analyzed priming across test prefixes of the same theme (e.g., colors) versus different themes (e.g., jobs). The results show significantly less priming for unrelated themes, suggesting a potential boundary for the extent of pollution, and therefore, a promising, sensitive metric for detecting and quantifying the extent of pollution. This analysis is now in Fig. 7b and believe it highlights the utility of the priming metric as a proxy for measuring pollution. We have added a sharper explanation of this in Section 4.

Altogether, we were very happy for the number of insightful issues that the Reviewer brought up and have been instrumental in refining and strengthening our work. Their feedback prompted us to sharpen our discussion of our design choices and delve deeper into their implications. This led to our new experiments and analyses that investigated the relationship between priming and knowledge pollution, and led us to uncover a fascinating connection to gradient clipping techniques used in differential privacy.

In light of these matters, we hope the Reviewer will recommend the acceptance of this paper. We believe that reporting the puzzling dynamics results we obtained will cause insightful discussions when we will hopefully report to the ICLR community. We hope that the robust predictive results, and useful, novel algorithms, as well as the sharp insights and new experiments that we and the Reviewer jointly thought of, will benefit the research community. We welcome and stand ready to answer any further questions.

1/2

We thank the Reviewer for their thoughtful feedback and appreciate them recognizing the value of our work in understanding knowledge updating in LLMs, for the research community. Our responses to their questions, which we found fruitful and led to several new experiments and analyses, are below:

Framing and Pollution:

We thank the Reviewer for their insightful feedback on the paper's framing, particularly concerning the term "pollution." We agree that a more neutral term would better reflect the dual nature of new data insertion, which can both enhance and degrade existing knowledge. Following the Reviewer’s suggestion, we have proposed title revision to "How new data permeates existing knowledge and how to dilute it" and replaced "pollution" throughout the manuscript with "permeates" to describe the unintended propagation of newly learned information into unrelated contexts (see attached new pdf). This acknowledges the inherent spreading effect of new data within the model's knowledge space.

Nevertheless, in new Fig. 10, we are excited to have completed a new analysis showing how newly inserted facts alter the model's certainty about unrelated test prefixes, often replacing previously high-certainty responses (e.g., "the color of sand is gray") with newly acquired information (e.g., "the color of sand is vermilion"). This demonstrates how priming can explicitly permeate well-formed knowledge and is a direct consequence of new data insertion.

We have added this novel finding into Fig. 10 in the newly attached pdf and have written these results into the final manuscript, with the aim to better communicate our findings while addressing the Reviewer’s concerns.

Natural Consequences of Priming:

We appreciate the Reviewer's observation regarding the consequence of priming being more pronounced for unsurprising tokens. We acknowledge that our initial presentation could have benefited from clearer definitions and sharper, illustrative examples, potentially leading to this interpretation, which we will now seek to clarify.

We'd like to emphasize a crucial point: the denominator in the priming score calculation, $P_{before}(x_{key} | X_T)$, remains constant across all Outlandish texts within the same theme. This is because the testing prefixes ($X_T$) are identical for each theme (e.g., all "color" prompts use the same set of prefixes asking about the color of various objects). Therefore, any variation in the priming score is solely attributed to changes in the numerator, $P_{after}(x_{key} | X_T)$, directly reflecting the impact of new data insertion over the course of learning.

Brainstorming further on this point, one can make a hypothesis that perhaps increases in numerator in the priming score could have been due to increases in the memorization score, dragged up along the way. To further investigate this hypothesis, we examined the correlation between the relative change in the numerator of the priming score and the relative change in the numerator of the memorization score. Surprisingly, we found that this correlation varied significantly across different language models (PALM-2, Llama, Gemma). This unexpected diversity in priming dynamics, despite the robust keyword probability vs. priming relationship across models, presents an interesting, novel puzzle.

These findings raise intriguing questions about the underlying mechanisms of priming in large language models. We believe that presenting these puzzling results to the ICLR community will stimulate further investigation and contribute to a deeper understanding of language model behavior.

Motivation for Strategies:

We appreciate the Reviewer’s comments on the motivation behind our proposed strategies. For the stepping-stone augmentation, the primary motivation was to test our hypothesis that pre-learning token probabilities drive post-learning priming. This hypothesis was validated experimentally, as detailed in Section 5.2.

Regarding the ignore-topk pruning strategy, we acknowledge that its initial discovery was serendipitous, and therefore, we ourselves were surprised!

However, the Reviewer’s suggestion prompted us to explore connections with gradient clipping techniques in the differential privacy literature, where some ideas are used to mitigate unintended learning effects (e.g., https://arxiv.org/pdf/1905.03871) which we now believe might share deep principles with ignore-topk. While these works do not explicitly address priming or hallucination, they potentially provide a link as well as theoretical grounding for our approach. We have added this to Section 2.3 to strengthen the motivation for this strategy.

Furthermore, we want to emphasize that simply removing data that one believes to be undesirable is not always feasible. Priming effects can be subtle, and even benign data can contribute to unintended priming, simply because the data is novel and therefore unexpected ...

2/2

Impact of Priming on Prediction Accuracy:

We appreciate the Reviewer’s request for a clearer demonstration of priming’s impact on prediction accuracy. To address this, we conducted a new analysis examining how priming alters the model's responses to common knowledge prompts (e.g., "the color of sand is __"). Before learning, the model’s predictions for these prompts had high certainty and aligned with real-world knowledge (e.g., "gray"). After learning new Outlandish facts, the probabilities for unrelated keywords (e.g., "vermilion") increased by several orders of magnitude! This new result is shown in Fig. 10 (newly attached pdf) and highlights how priming can permeate into high-certainty knowledge and lead to erroneous predictions.

Brainstorming further, we propose a future experiment to trace facts encountered during pretraining. By identifying ground-truth facts explicitly learned by the model, we can more precisely measure how priming distorts existing knowledge versus unfamiliar contexts. This analysis would further elucidate the implications of priming in real-world applications, such as knowledge retrieval or fact-checking tasks.

In sum, the Reviewer brought up a wide range of insightful discussion points which we found fruitful and inspired several new experiments and new analyses which we have now reported. We are humbly grateful for the Reviewer’s careful reading and appraisal of our study and for their praise regarding the robustness, generality, and practical utility of our experimental results and new algorithms, as well as even catching our subtle results like comparing priming in ICL vs gradient-based learning. If there are any further questions or curiosities, or suggestions to raise the study’s impact still further (or even to increase their already very illustrious score), we hope they will feel free to ask us.

1/2

We thank the reviewer for their thoughtful and constructive feedback. We appreciate the positive comments and the resonance they have with us on the novelty of our findings, the robustness of our results, and the practicality of our proposed strategies. We address the questions as follows:

Priming Mechanism:

We thank the Reviewer for highlighting the importance of exploring the underlying mechanisms of the priming phenomenon. While our main empirical finding—that pre-learning keyword probability strongly predicts post-learning priming—is robust across models and sizes, we agree that understanding the mechanisms behind these dynamics is a crucial area for further study. Our experiment comparing the priming dynamics of PALM-2, Llama, and Gemma models suggests that indeed, factors like architecture or training procedures could play a crucial role in how priming and memorization interact, but the many differences between these models makes it difficult to pinpoint the underlying mechanisms.

To further explore this, and inspired by the Reviewer’s comment, we have now conducted new analyses using a fine-tuned LLM based on the FLAN dataset. This allowed us to directly compare a pre-trained PALM-2 model with a fine-tuned version sharing the same fundamental architecture. By controlling for architecture, we could effectively isolate the impact of fine-tuning on the relationship between priming and memorization. Our findings reveal that fine-tuning alters this dynamic, perhaps due to a narrowing of the hypothesis space during the fine-tuning process (new Figure 20, attached). This new finding provides a further step toward uncovering how different training setups affect the spread of new information. This new result is shown in Fig. 20, and we have added these results and their implications to the final manuscript and believe they will inspire further exploration of these mechanisms by the research community.

More Sophisticated Experimental Setups:

We share the Reviewer’s vision for extending the experimental setup to better reflect real-world complexities. As a starting point, we intentionally focused on vanilla fine-tuning to study how the most widely used data insertion mechanism propagates new information. We believed this fundamental understanding to be essential before investigating more advanced methods. That said, we have already begun exploring more complex scenarios that the Reviewer envisions. First, the Outlandish dataset incorporates diverse, free-form text structures beyond the canonically studied (subject, relation, object) facts, aligning with real-world data diversity. Second, we have conducted preliminary experiments on multi-fact learning, as shown in Figure 14, which have begun to highlight the interactions between multiple facts. These results provide a foundation for future work, which we ultimately aim to extend to state-of-the-art techniques such as those proposed by Meng et al. (2022a; 2022b) and Ovadia et al. (2023b). We have clarified this in the Limitations section.

Comparisons with Related Works:

We appreciate the Reviewer’s suggestion to strengthen the paper’s connection to related works. In response, we have enhanced the dedicated section in the manuscript comparing our approach to studies on LLM knowledge editing (e.g., Meng et al., 2022a; Mitchell et al., 2022) and continual learning (e.g., Wu et al., 2024; Shi et al., 2024), as well as honed discussions on how injection of new texts into LMs can cause hallucinations (Gekhman et al., 2024; Wan et al., 2023; Yin et al., 2023; Huang et al., 2023), or cause mistakes in downstream reasoning (Huang et al., 2023; Cohen et al., 2023a). This discussion (Sections 2.1, 2.3, 2.4) situates our contributions within the broader research landscape.

Rationale Behind Sprime:

We thank the Reviewer for raising the important question about the suitability of the priming metric. The central motivation behind Sprime was to create a simple, scalable metric for quantifying the spread of new information in free-form texts, beyond the constraints of (subject, relation, object) facts. However, we agree that the metric could be extended to account for synonyms, hypernyms, or related terms that may also reflect priming.

As a concrete improvement, we agree very much with the Reviewer that the priming score would be extended to even greater heights if it can account for related words and synonyms. Brainstorming further, one possible way to do so is to utilize the ideas from the paper Farquhar et al, 2024 Nature. We have added this to the Related Works (section 2.4) outlining this as a direction for future work and thank the Reviewer for this suggestion.