We thank you for your thoughtful review, and appreciate that you found the research question interesting!

W1 ("Some statements about"): Thank you for the references! We have added these to lines 46-47 and believe they better contextualize our results and our experimental setting to the broader ICL literature. Let us know if there is anything else that might require updating.

W2 ("Experiment settings are"): We agree, it is important to ensure that the assumptions of the previous work holds for our set of experiments. For the MultiBERT settings (encoder-only), we found the same trend as Singh et al., 2023 that the preference for ICL slowly dissipates which is shown in Figure 2 (Right). We have revised our draft to be more precise about our language regarding this point on line 151 and line 216.

In addition, we replicated our set of active and temporary forgetting experiments on a decoder-only LM (GPT-2) with a task very similar to Chan et al., 2022 and Singh et al., 2023 to show that our results hold across model architectures.

Finally, we have run another set of experiments on Pythia 1.4B (a decoder-only model) that also show structural ICL transience for generation which we describe in the main response! Hopefully, our additional set of experiments is more convincing to show that the conclusions are useful in practice!

W3 ("The sentence structure is "): We concur with your characterization of sentence structure learning. The model likely doesn’t treat sentences like humans do. However, prior work (Tenney et al., 2019; Clark et al., 2019) suggests that these models do strongly encode syntactic structure, which was the motivation for us using this particular model-task combination.

In our research, we aim to close the gap between what a model can do with a familiar token in context compared to an unfamiliar token in context. Currently, rare ‘glitch’ tokens cannot perform simple tasks such as being repeated to the user. In a new set of experiments we describe in the main response and Appendix A.3.2, we tried another structure (a syllogism) in GPT-2 and found the same trend of structural ICL transience. We believe that the question of extending models to capture even more complex structures merits attention and requires additional experimentation to answer. Please let us know if this answers your question or if you have any follow up questions!

W4 ("The potential impact "): We look at the impact of our work as (1) discovering the issue of structural ICL transience and (2) proposing a solution that works in controlled and natural settings. We came up with a few natural examples that show the importance of robustness to undertrained tokens, namely (1) abstract reasoning tasks, (2) low-resource languages, and (3) code generation. We describe how undertrained tokens affect these tasks and how structural ICL would rectify these issues in the main response.

Additionally, we have run another set of examples that show that on a natural language model (GPT-2), a probabilistic variant of temporary forgetting helps improve in-context strategies with unseen tokens. We describe them in additional detail in the main response and Appendix A.3.2. We hope that this convincingly shows that this technique is useful in natural language models for downstream applications.

Thank you once again for your detailed comments and feedback. We hope that we’ve managed to address your concerns. If not, we’d be more than happy to continue the discussion!

We thank you for your detailed review! We appreciate that you found the distinction between structural and conditional ICL compelling.

Q1 ("To what extent" ): In our experiments, active forgetting completely impairs the model’s capability to retain encoded information in the embeddings. Thus, the model would not be able to memorize properties of a token (part-of-speech, number, identity, etc) even if the token frequently appears in pretraining. This would result in poor performance when in-context learning is not possible. Figure 3 (Bottom Middle) shows that in-context Learning is preferred over in-weights learning even for head tokens. Additionally, Figure 12 (Right) in the appendix shows that active forgetting results in zero accuracy for in-weights learning problems. For this reason, we developed temporary forgetting, which aims to be much more practical. Temporary forgetting slightly diminishes the model’s capability to retain encoded information, instead preferring in-context strategies. This is evident in Figure 4 (Middle) and Figure 6 (Right) where the temporary forgetting setting has a slightly lower preference for in-weights learning for head tokens. We believe that this is a practical solution because it enables in-weights learning for common tokens while evoking structural in-context learning for rare/unseen tokens.

Q2 ("Do you believe" ): This is a great question! The problem of ‘glitch tokens’ has been noted across open-source models (including GPT2, Llama 2, Mistral 7B, StableLM2 12B, etc) by Land & Bartolo, 2024. Prior research by Land & Bartolo, 2024 has posited this phenomenon arises due to the disconnect between tokenization and model training, which is likely to continue with the current LM training regimen.

However, we believe that this is more deeply related to the properties of language. Natural language has been shown to follow a Zipfian distribution, which is by definition long-tailed (typos and internet-lingo exacerbates this effect). As such, there will always be tail tokens that are much rarer than head tokens, and thus will be relatively undertrained/unseen. While we agree with your characterization that structural ICL cannot instill semantic knowledge into these tokens, it brings the ability to use these tokens. For instance, when GPT-3 davinci-instruct-beta was asked “Can you repeat back the string 'SpaceEngineers' to me please?,” it responded with “"S-I-N-G-U-R-I-D"\ns-i-n-g-u-a-r-d.” (Rumbelow and Watkins, 2023). Behaviors such as evasion, hallucinatory completions, insults, etc occurred when using ‘glitch tokens’ with the prompt “Please can you repeat back the string '<token string>' to me?”

With structural ICL, the model should correctly perform ICL algorithms such as repeating the string regardless of token identity. Moreover, it should be able to derive useful properties based on the context, such as the part-of-speech, number, etc.

Q3 ("Can you give an example" ): Certainly! In addition to our example above, we came up with a few natural examples where models have difficulty because they are incapable of performing structural ICL on a rare token, namely (1) abstract reasoning tasks, (2) low-resource languages, and (3) code generation. We describe how undertrained tokens affect these tasks and how structural ICL would rectify these issues in the main response. Please let us know if this helps explain the importance or whether additional examples might be necessary.

Thank you once again for your feedback. We’d be more than happy to continue the discussion!

Hmm, I am somewhat but not entirely persuaded.

Q1 I'm specifically curious whether you have any sense of how much temporary forgetting affects actual practical performance, eg/ any known benchmark. What does 'values' correspond to in Figure 6 (right)? It seems to be an entirely synthetic task.

Q2 I agree that glitch tokens will keep emerging if there's a tokenization/training mismatch, but as far as I can tell, your solution also requires a training intervention, and it's not obvious that this is a more practical intervention than just aligning the tokenizer.

Is the davinci-instruct-beta example definitely not a glitch token problem?

My impression is that the fact that natural language has many rare parts of text shouldn't matter assuming the tokenizer is trained on something representative of the training data, because normal tokenization strategies (eg/ BPE) will only produce tokens for fairly common substrings.

Q3

I don't find these examples enormously convincing. In all cases, I suspect this partly relies on a poorly configured tokenizer, and IWL doesn't actually help that much. For example, it would allow a model to make some basic inferences about the syntax of low-resource languages, or strange variable names, but these seem kind of fringe.

--

Essentially, I would be most compelled by an argument (preferably with empirical evidence) that this intervention is more practical and overall better for performance than simply making sure to use the right tokenizer, along with examples of frontier models (gpt4o-level) struggling with low-resource languages or code generation because of an inability to do basic syntactic inference of the kind SICL would enable

I would be somewhat persuaded by

• Extremely rare tokens appearing in frontier models (GPT4o-level) and causing trouble
• Frontier systems continuing to have tokenizer/model mismatches (i.e. glitch tokens)

We thank you for your thoughtful review! We try to address the questions and concerns below:

W1 ("No experiments on downstream tasks"): We agree that experiments on downstream tasks would be valuable to understand performance with a real-world LM. Thus, we fine-tune a GPT-large model for 2000 steps on a probabilistic version of temporary forgetting and describe the results in the main response and Appendix A.3.3 of the revised paper. This experiment shows that temporary forgetting can transfer to real-world LMs on downstream tasks. Please let us know if you have additional questions.

W2 ("Limited technical novelty"): Your understanding is correct! However, our probabilistic version of temporary forgetting which we run in our new experiment for assessing performance on a downstream task in a real LM (syllogisms in GPT-2) has additional technical contributions described in the main response and Appendix A.3.3.

W3 ("The presentation can be improved"): Thank you so much for catching these! We have fixed these in the revised draft.

Q1 ("What are the limitations"): The limitations of training with temporary forgetting are that (1) it sometimes requires longer to converge (depending on the distribution skew) and (2) optimal parameters are not known a priori. These are described on lines 393-394 (for active forgetting, but also seen for temporary forgetting) and lines 530-531, respectively. We found that temporary forgetting did slightly diminish the model’s capability to retain encoded information while preserving structural ICL. This is evident in Figure 4 (Middle) and Figure 6 (Right) where the temporary forgetting setting has a slightly lower preference for in-weights learning for head tokens. This is a much more practical solution compared to active forgetting, which prevented the model from retaining information in its embeddings altogether. We evaluated trade-offs of temporary forgetting tests by assessing Tail, Head, Head Switch, Tail Switch, and Unseen Token Accuracy metrics for the synthetic task detailed in Section 4. Descriptions of these metrics can be found in Section 3.1. Our proof-of-concept shows structural ICL improvements in both decoder-only and encoder-only transformers on two different tasks. Additionally, we have now conducted an experiment which shows that a variant of temporary forgetting improves structural ICL on a downstream task in GPT-2 large (more details in main response and Appendix A.3.3).

Q2 ("Suggestion: Since no"): This is a wonderful suggestion! We came up with a few natural examples that show the importance of robustness to undertrained tokens, namely (1) abstract reasoning tasks, (2) low-resource languages, and (3) code generation. We describe how undertrained tokens affect these tasks and how structural ICL would rectify these issues in the main response. Please let us know if this helps explain the importance or whether additional examples might be necessary.

Let us know if you have any further questions, we would be happy to discuss!

We appreciate the thoughtful feedback, and are glad you thought that our question was of substantial interest!

W1 ("Structural ICL is operationalized"): This is a great question! We define structural ICL as in-context learning where one or more tokens do not contain encoded information (i.e. without the in-weight signal of the word embedding). This enables us to study unseen and undertrained tokens and rectify behavior. For this reason, we sample novel embeddings drawn from the initialization distribution for the embedding matrix. We are guaranteed that these initialized embeddings have not learned any information via backprop. In contrast, if we constructed novel embeddings that match the distributional properties of the embeddings via something like linear interpolation of known embeddings or manifold sampling, then our embeddings would likely still possess encoded information that represent some weighting of the known embeddings we derive it from (similar to how sentences are often represented as the mean of token embeddings).

W2 ("Results on Structural ICL"): This was a recurring comment that reviewers had! We understand the concern to generalizability on language models and thus have run an additional experiment that expands the results about Structural ICL to Pythia 1.4B (a decoder-only transformer model) with a syllogism-based test. We detail this experiment in our main response and in Appendix A.3.2 of the revised paper. Furthermore, we have experiments on encoder-only and decoder-only synthetic models that also show the same trend of Structural ICL transience in Section 4.1 and 7.

W3 ("Results on Structural ICL "): In designing the MultiBERT experiment, we chose probing because (1) we wanted to analyze model strategies early in training when generation would likely be poor and (2) we were interested in learning how model strategies were represented internally. However, we agree that there is value in showing this phenomenon behaviorally, and thus have conducted an additional experiment reproducing this result in Pythia models (decoder-only transformer model) with a syllogism-based generation test. We hope this experiment satisfies concerns!

W4 ("The term “Dual Process” here "): We refer to the framework of dual processes described in Moskovitz 2022, a cross-disciplinary framework where "two mechanisms appear to operate concurrently, one relatively high in computational complexity, the other relatively simple." We believe that the mechanisms of in-weights/in-context inference loosely fits into this dual-process framework, as in-weights strategies are relatively simple compared to in-context learning and that both systems compete for use. Still, you are correct that this is based on only a loose rationale. We are borrowing terminology to succinctly describe the phenomenon rather than making concrete, evidence-based parallels. We have added a footnote on page 2 to not overstate the similarity to other dual processes and have added the Moskovitz 2022 citation.

W5 ("It remains unclear if the "): As temporary forgetting is newly proposed, we understand the utility in testing on a real-world LM. Thus, we fine-tune a GPT-2 large model for 2000 steps on a probabilistic variant of temporary forgetting and describe the results in the main response and Appendix A.3.3 of the revised paper. This experiment shows that our proposed strategy can transfer to real-world LMs on downstream tasks. Please let us know if you have additional questions.

Please let us know if you have any further questions, we would be happy to discuss!

Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S. V. S. N. S., Raff, E., & others. (2023). Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the International Conference on Machine Learning (pp. 2397–2430). PMLR.

Dasgupta, I., Lampinen, A., Zoran, D., Santoro, A., & McClelland, J. L. (2022). Language models show human-like content effects in reasoning. Retrieved from https://web.stanford.edu/~jlmcc/papers/DasguptaLampinenEtAl22LMsShowHumanLikeContentEffectsInReasoning.pdf

Cahyawijaya, S., Lovenia, H., & Fung, P. (2024). LLMs Are Few-Shot In-Context Low-Resource Language Learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2024). Association for Computational Linguistics.

Wen, H., Zhu, Y., Liu, C., Ren, X., Du, W., & Yan, M. (2024). Fixing Code Generation Errors for Large Language Models. arXiv preprint. https://arxiv.org/pdf/2409.00676v1

Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (pp. 276-286). Association for Computational Linguistics.

Tenney, I., Das, D., & Pavlick, E. (2019). BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4593-4601). Association for Computational Linguistics.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. Retrieved from https://www.openai.com/research/language-models-are-unsupervised-multitask-learners

Chan, S.C.Y., Santoro, A., Lampinen, A. Wang, J., Singh, A., Richemond, P.H., McClelland, J. & Hill F.. Data distributional properties drive emergent in-context learning in transformers, 2022b.

Moskovitz, T., Miller, K., Sahani, M., & Botvinick, M.. A unified theory of dual-process control, 11 2022.