Competition Dynamics Shape Algorithmic Phases of In-Context Learning

Dear Reviewer medY,

We thank you for your time and effort in giving us such extremely detailed and thoughtful review. We have just updated our manuscript, and your thoughtful feedback has really reshaped our paper, both in framing and technical analysis. We have spent a huge amount of time and effort these past two weeks running new experiments (e.g., see Appendices E and F) and performing rewrites of several parts of our paper (e.g., see Abstract and Introduction). We truly believe our paper is in a better position because of this.

However, since the window for paper revisions expires in a few hours, we have now updated our manuscript with all the changes we could implement, and several that we could not. In particular, we tried several variants in our writing that could incorporate your suggestions as much as possible. This process required a lot of subtlety and effort, and we would appreciate if you could allow us additional time to craft a detailed response specially for you that summarizes our extensive changes in the context of your recommendations.

We'll come back soon!

Summarizing rebuttals

Summary. We would like to again and show our sincere gratitude to the reviewer for their detailed feedback and the immense amount of time they must have put in to create such a precise review. We have tried our best to put in a proportionate amount of effort, and we are extremely excited to see the results of the same: our paper now uses a much more accurate terminology throughout, is noticeably more precise with its claims, and has several new experiments that provide further evidence to all claims. We hope our responses above address the reviewer’s concerns, and, if so, that they would consider consider raising their score to support the acceptance of our work!

W3. The motivation in terms of generalizing and unifying phenomena is inaccurate

We first note that we are excited to see the reviewer’s appreciation of our attempt at capturing several ICL phenomena in a singular setting. While we understand the primary concern the reviewer has flagged, we want to emphasize that getting most (if not all) known results around ICL in a single setup truly was a guiding motivation for our work. This motivation stemmed from wanting to take a step towards developing a unified or normative theory of ICL. Specifically, we believed that given the use of rather disparate setups for understanding ICL, and phenomenology that is at least seemingly in conflict (e.g., task-retrieval vs. task-learning notions of ICL), we should first develop a setup that is rich enough to capture most such known results in a single setting, but also simple enough to be amenable to modeling. Such a setting can then become fertile grounds for attempting a unified account of ICL.

The motivation above is what led to our proposed task: learning to simulate a finite mixture of Markov chains. However, we note that our final pitch in the abstract and introduction ended up getting awkwardly phrased, and hence did not accurately reflect our intentions faithfully. We have now rewritten several parts of the paper (especially abstract and introduction) to address this problem. While we strongly encourage the reviewer to read the abstract and introduction, we specifically highlight that we now explicitly state the following (paraphrased for brevity): our paper’s goal is to enable a step towards a unified account of ICL; to this end, we propose a setup that captures most (if not all) known phenomenology of ICL, and is simple enough to be amenable to modeling.

Questions

We thank the reviewer for the questions below. Before we answer them, we note that in the pursuit of making changes according to their comments above and running relevant experiments, we could not necessarily accommodate all requested edits. We have highlighted such cases below, and promise to perform relevant changes in the final version of the paper!

Q1. The four algorithms have unclear and misleading names. As I said, I am a believer in the importance of names. I felt strongly that the…

We thank the reviewer for this question! Based on this question and W1, we have made significant edits to the manuscript. In particular, we have changed the names of our algorithms from Uni-Bayes, Bi-Bayes, Uni-ICL, Bi-ICL to Uni-Ret, Bi-Ret, Uni-Inf, Bi-Inf, respectively. Below, we reply to the reviewer’s sub-questions.

A1.1: Indeed, the model is doing some form of in-context learning throughout all four phases. We have now changed the solution names so that they do not seem to suggest that some solutions are ICL and some are not. We thank the reviewer for motivating this important change!

A1.2: We thank the reviewer for this comment as well. We have now replaced “Bayes” to “Retrieval”, and we have also clarified the use of “Bayes” in App. B.1. For a replacement for “ICL”, after a long discussion, we converged on “Inference” based on the fact that these algorithms involve inference of the stationary distribution or the transition matrix from the context. We hope the new names better illustrate each algorithm without causing confusion.

Q2. Inconsistent summary of algorithms between main text and figure 4 caption: In the figure caption, the Uni-Bayes and Bi-Bayes descriptions talk about selecting a 'closest task' from the mixture. Based on the main text, my understanding is that they do not select a single task but rather they form a posterior distribution over all tasks and use the posterior predictive distribution to make their prediction.…

We thank the reviewer for pointing this out! The reviewer’s understanding is correct, our Bayesian (now renamed fuzzy retrieval) algorithms indeed perform a Bayesian averaging operation, i.e. an average of every hypothesis’ prior weighted by their likelihoods. We have now made major edits to the main text so that this point is clear. Furthermore, we removed the boxes in Figure 4 which used to describe selection. However, embarrassingly, editing Figure 4’s caption itself escaped our attention and the current version still mentions selection. (We have internally tagged this as an edit to make.) We should also fix this in Fig. 37’s caption as well.

While we will make these edits, we clarify two things partially alleviating this wording:

• In practice, Bayesian averaging and selection turns out to be extremely similar at context length of 15+ (which is most of our sequence of length 512).

• For the above reason, we concisely use the word “retrieval” in our task for “fuzzy retrieval” to make intuitive explanations of phenomena, especially in App. C.

[Continued below...]

• New explanation for Transience. Our explanation for transience builds on our demonstration that a specific algorithmic behavior slowly, but steadily, supersedes other ones; in particular, this is the algorithm that achieves better loss on the training data over time (what we used to call Bi-Bayes). As the reviewer noted, the contribution here is relatively simple; however, our novelty lies in the technique used to demonstrate it and the dynamic underneath it. To address reviewer’s concerns that we may perhaps have overclaimed this contribution, we note we have made the following changes to the paper. We also compare our contribution to the related work noted by the reviewer below.

  • Removing the adjective “new”. We agree that the precise dynamic here is driven by the fact that there is a certain algorithmic behavior that yields a better loss than others on the training dataset. Hence, to not state this contribution overly strongly, we have removed unnecessary adjectives and merely stated that we “offer an explanation for the transient nature of ICL” (e.g., see Contribution 3 in introduction).

  • Mechanistic evidence for a competition picture of Transience. As the reviewer pointed out, we chose to use LCA to offer an explanation for the transient nature of ICL. The use of this tool actively pits different algorithms in competition with each other, but we did not sufficiently justify why one should take this competition lens. To further back up this picture, we have now added a preliminary mechanistic analysis, wherein we demonstrate that at the Bi-ICL to Bi-Bayes phase boundary, we can precisely see that attention maps have a structure that is akin to an interpolation of the structures achieved in those two phases individually! As training further proceeds, these maps increasingly look like the Bi-Bayes phase’s maps. Furthermore, we find the model shows an intriguing memorization dynamic: the ability to reconstruct transition matrices from MLP neurons first increases during training, then decreases in the Bi-ICL phase, and then starts to increase again! These results, in combination, are highly suggestive (but do not completely vindicate) the idea that the different algorithmic behaviors are in competition with each other to explain the model.

  • Loss vs. accuracy comparison to Singh et al. Finally, we would like to highlight a subtlety that we slightly disagreed with in the reviewer’s comments. The reviewer mentioned that Singh et al.'s setting for studying transience likely has a different dynamic than our work’s, since their system is constructed to have equally performant in-weights learning (IWL) and in-context learning (ICL) solutions. However, we emphasize that performance in Singh et al.’s paper is defined via accuracy on the task, not via loss. That is, the two solutions to their in-context classification task are equally performant in terms of accuracy! In a recent work released after the ICLR deadline, it was in fact shown that IWL finally overtakes ICL because it yields a better final loss than ICL! The underlying learning dynamic uncovered in that work is exactly what we propose as well. In fact, we would also like to highlight a parallel ICLR submission that has made the same argument as ours, that competitive dynamics underlie transience, but in the setting similar to Singh et al.’s [2].

[1] https://arxiv.org/pdf/2410.23042 [2] https://openreview.net/forum?id=INyi7qUdjZ

[Continued below...]

• Summary of new experiments in mechanistic analysis: Appendix E. In Appendix E.1, we demonstrate that we can reconstruct transition matrices from the model’s MLP neurons in “fuzzy retrieval” phases (we note that is the name we now use for the phases we used to call “Bayesian”; we have made this revision to ensure further precision in our intended meaning). In Appendix E.2, we define a quantitative measure of memorization and evaluate it throughout training; as we find, if a model undergoes transience, memorization is first high, then it goes down when the model enters the Bi-ICL phase (or what’s now called the Bi-Inf phase), and then comes back again! We have also significantly expanded our attention heads analysis in Appendix E.3 and E.4, showing in fact that the attention maps at boundaries of different phases are a direct linear interpolation of attention heads from individual phases. We believe these latter set of results in fact provide some grounding to our approach of linearly combining individual algorithms in the LCA analysis.

Overall Summary of Response to W1

Overall, we note that we have made substantial edits to the paper to address reviewer’s comments: we took their suggestions seriously and have put in an honest effort to modulate our claims in accordance with our offered experimental evidence. Meanwhile, we have also run new experiments to add further evidence towards our claims, as summarized above. We note there are likely still parts of the writing that should have been addressed, but slipped through due to time constraints: we know of at least one such occurrence, i.e., the title of Section 4.2, which we plan to retitle “Analyzing OOD performance with LIA”. We promise to have these slips addressed by the time of final paper revision!

W2. LCA is not mechanistic and does not offer a new explanation

We believe the reviewer has raised three broad points in their comment: (i) whether LCA (which we have now renamed to LIA) is a technique that can offer mechanistic insights, (ii) whether our explanation of transience via LCA is in fact a “new explanation”, and (iii) whether our explanation is causal. We mostly agree with the reviewer on points (i) and (iii), though we slightly diverge on point (ii). Especially, a framework can provide an explanation of a phenomena at different levels and we can deepen our understanding without being fully mechanistic. We explain our reasoning in the discussion below, and summarize what changes we have made to address the reviewer’s concerns.

• On causality. Since this point is easiest to clarify, we discuss it first. We note we have removed the mention of the term “causal explanation” from introduction (the only place this term occurred). We believe this was mostly a slip during writing, and we agree it was unnecessary.

• On LCA being mechanistic. We agree that claiming LCA (or what is now called LIA in the paper) offers a “mechanistic decomposition” is inaccurate. We have hence taken active efforts to rephrase parts of the writing that suggested LCA decomposes the model into different mechanisms at any given experimental configuration (except in Section 4.2’s title, which we mistakenly forgot to edit). Instead, we now claim that LCA offers a protocol for explaining the model’s behavior at any given configuration via an interpolation of individual algorithms we claim to underlie our phase diagram. That is, we now try to pitch LCA as a reliable and useful tool that can help gain insights into the model’s learning dynamics if we can preemptively speculate which algorithms explain the model’s behavior in different phases.

  • Despite these writing edits, we do want to emphasize that given LCA’s OOD performance, we believe the technique can offer insights into a model’s behavior by offering an “effective substitute”. That is, one can think of LCA as a modeling technique that captures a trained model’s behavior, and can hence serve as an effective substitute for doing theoretical analysis or developing hypotheses to explain its behavior (e.g., why a model makes a decision for a given input). To the extent this modeling is accurate, which, as the reviewer noted, is likely true in our case, the developed insights will likely be effective too. We intend to write a longer note summarizing this in the final version of the paper; due to time constraints, we could not do so just yet.

[Continued below...]

• Acknowledged possibility of other algorithms. We completely agree with the reviewer that our analysis does not guarantee that the algorithms we use to describe our phase diagram are “unique”. We have now made significant attempts at ensuring that our writing does not suggest so. Specifically, we now explicitly state in both the introduction and in analysis (see L358) that we are able to identify at least four algorithms that can explain the model’s behavior. That is, we do not rule out the possibility of there existing other algorithms that can explain the model’s behavior, but we weakly suggest they’ll likely be able to do so because they are in some sort of a class of behavioral equivalence with our considered algorithms: i.e., these alternative algorithms would also produce the same next-token predictions as our considered algorithms, but they may arrive at that solution via a different set of rules or a different implementation. Due to time constraints, we could not weaken this phrasing further yet, but by the final revision we intend to tone the claim down to suggest that algorithms which generally produce the same next-token prediction (i.e., they are approximately in the class of behavioral equivalence) could also be considered for describing our phase diagrams.

(ii) New set of experimental evidence to further validate our algorithmic picture:

• KL divergence between model and algorithms associated with individual phases: Fig. 5(d). Having said the above, we note we still label our phases using our algorithms’ names. We believe this is justified because (i) we do not claim these algorithms are a mechanistic explanation of the model or the phase diagram itself corresponds to a mechanistic decomposition of the model, and (ii) because we are in fact able to almost perfectly predict the model’s next-token predictions via our procedurally defined algorithms on OOD data. For this latter point, we have now added a new panel to Figure 5 to make the argument quantitatively, showing the KL between our model and predefined algorithms’ next-state transition probabilities is very low in phases associated with these algorithms.

• Effects of alternative algorithms on LCA: Figs. 40, 41. Arguably, one could consider the possibility of alternative algorithms that are completely out of our considered algorithms’ “equivalence class”, i.e., they produce somewhat reasonable next-token predictions to take away some mass in our phase diagram analysis, yielding a new phase. To address this situation and demonstrate the robustness of our analysis to (at least) naively defined ad-hoc algorithms, we have now added new experiments in Appendix H (Figs. 40, 41). Our results show that such algorithms in fact do not get assigned their own individual phases. We emphasize that we do not claim these experiments are sufficient to suggest one could not come up with alternative algorithms to adversarially attack our analysis, but it does provide further evidence towards our claims’ validity.

Further (but nevertheless preliminary) mechanistic analysis. As the reviewer noted, our mechanistic analysis demonstrating mechanistic evidence towards the considered algorithms being “implemented” by the model was relatively weak. We completely agree with this assessment! As noted above, we have partially tried to address this issue by rephrasing several parts of the paper. However, we note that we have also substantially expanded our mechanistic analysis now, yielding evidence towards the model at least possessing relevant information to implement an algorithm akin to the ones we consider in this work. We emphasize we have made efforts to ensure our writing reflects this evidence is still preliminary though, i.e., we do not claim this evidence is sufficient to make outright claims about models implementing algorithms considered in our analysis.

[Continued below...]

We thank the reviewer for their detailed feedback, extremely precise comments, and a multitude of suggestions to help improve our work! We are very happy to see the reviewer’s excitement about our work, for example that they found our work to be “well-motivated…priority for science of deep learning”, that our proposed task offers “a neat setting…with interesting and rich collection of idealised solutions”, that our experiments form “a comprehensive study of the numerous axis of variations”, that our LCA analysis is “thought-provoking” and “an elegant idea for behavioural analysis of models in general”, and that our work overall “makes a valuable contribution that enriches our understanding of ICL phenomenology”.

Below, we respond to specific comments by the reviewer. Before we begin, we want to emphasize that this review was, at least for us, unprecedented along several axes: in its detail, in its precision, and in its length. We truly appreciate the amount of time and effort the reviewer invested in giving us this detailed feedback and providing us concrete recommendations that have already improved our work. Indeed, we stress that we have worked very hard these past two weeks to accommodate these recommendations and address the reviewer’s concerns. Specifically, we want to highlight that there was a lot in the reviewer’s comments that we agreed with and tried to address: e.g., we agree that we could have been more precise on our use of the word “mechanistic”, especially given the term’s modern connotations in deep learning; we have hence tried to remove any phrases that could have suggested we offer a complete mechanistic interpretation of our models. We have also made concrete attempts at actually reverse engineering our models and providing further (but nevertheless preliminary) experimental evidence for our claims.

We hope our edits and efforts sufficiently address your concerns, and we look forward to engaging with you to further identify paths for improving the quality of our paper!

Response to weaknesses

W1. Insufficient evidence for confidence in algorithmic phase identification

We first note that we are glad to see the reviewer found our contribution of creating a phase diagram of ICL to be impressive—we found this result to be fascinating ourselves, and it updated how we (i.e., the authors) think about in-context learning.

As we understand, the reviewer’s primary apprehension with our analysis is the possibility of alternative algorithms that can describe individual phases in the diagram, and, relatedly, insufficient mechanistic evidence for the algorithms we do associate with said phases (please correct us if we are overly simplifying this!). We completely understand the reviewer’s concern, which we believe stems from a mixture of imprecise writing on our end and some concrete experimental evidence that we could have offered to further back our claims.

Below, we summarize the efforts we have made to revise our manuscript to address the reviewer’s concerns, including both (i) writing changes and (ii) new experiments.

(i) Summary of our rewrites to be more precise

• Emphasis on behavioral equivalence and removal of remarks on “implementation”. The first change we have made to our writeup is the removal of any phrases that suggested a model “implements” the four algorithms we use to define our phase diagram. Instead, we now say the four algorithms explain the model’s behavior in individual phases, where we define behavior as the next-token predictions. That is, we now merely claim that the algorithms can predict what output a model would produce, i.e., how it would behave, for a given input. Given that we are able to do this on both ID and OOD data, we feel confident the model likely implements some mechanism that is behaviorally equivalent to our predefined algorithms, but we neither claim to know nor do we try to speculate what this mechanism is. We put explicit remarks inline with these statements at the very beginning of our phase diagram analysis (see L252–L256).

[Continued below...]

Q3. Questionable choice to use a nonuniform task prior: If I understand correctly from the setting description, the authors sample a prior vector from a uniform Dirichlet distribution over task priors. ... This seems to me to be a significant departure from the use of task diversity by of Raventos et al., for whom, if I remember correctly, the task prior is always uniform…..

We agree to this point for the purpose of the conclusions drawn in this paper. Here is the honest reason: at the experimental design phase, we also aimed to investigate how models treat rare chains vs frequent chains. For this reason, we also explored uniform and Poisson priors on the tasks.

However, we did not deeply investigate into the effect of this prior since:

• The setup was rich enough that this question didn’t make it to our priority list.

• In general, the effect of the prior term turned out to be very weak. This is because the posterior is heavily dominated by the likelihood for retrieval solutions and the prior has minor effects to the inference solutions.

• Our initial experiments didn’t show significantly different results when we changed the prior.

• With a uniform prior, we expected at best a small translation of the phase diagram towards low data diversity as we make all data distributions effectively more diverse.

• We observed that the random seed on the Dirichlet prior had minor effects on the phase diagram.

We are currently constrained by space to describe this in the main text, but we will try our best to fit these points in future edits to the paper.

Q4. Convoluted phase isolation tests: These test seems very intricate. It is not clear to me that they are the clearest ways of isolating phases, and I wonder if you have considered and ruled out simpler alternatives...

1. We have indeed considered this test! However, simply comparing the rows of the transition matrix will not fully reveal whether the model is able to use bigrams. The model could very well be using bigram statistics to retrieve a chain, but output next state probabilities from the stationary distribution of that chain irrespective of the last state. Please see Appendix B.2 for a discussion of these “unigram posterior” solutions. In other words, our test tries to quantify the presence of bigram modeling in the whole model and not just whether the output probabilities are last token dependent.

2. For proximity, we have also considered simpler tests, e.g., solely based on the KL divergence of the model and the training set matrices. However, we converged to this method as this method provided a clear expectation value when the method is operating in a fully retrieval mode vs. a fully inference mode. Motivated by the reviewer’s comment we added a detailed explanation of this normalization procedure in Appendix A.3.3.

Q5. Missing details for proximity test: I was left wondering about some details of the proximity test. Unless I missed something, I would recommend clarifying the following points, if possible in the main text or otherwise by expanding on the 'additional details' in the appendix.

We answer the sub-questions below.

1. Yes, it is defined at the distance (KL divergence) to the closest member! We have now clarified this in Appendix A.3.3, and wrote down the formula for proximity explicitly in Eq. 11.

2. As mentioned above in response to Q3, the effect of the prior is minimal, especially since we uniformly average over sequence length, and thus most sequences are in the long context limit (15+ states).

3. Each row of this chain is sampled from the Dirichlet prior described in Appendix A.1, similarly from the training set and the random (control) set.

4. We thank the reviewer for this comment! We have added the formula and evaluation details in Appendix A.3.3.

Q6. Missing details about figure colour schemes: The phase isolation methodology and the LCA analysis are two distinct methods for colouring a point on a phase diagram. ... Please consider clarifying this in each figure's caption.

All figures except Fig. 5 use the LCA (now LIA) phases. We will clarify this in the manuscript.

Q7. Why is LCA formulated in terms of L2 distance for probabilities? ....

We used numerical optimization to minimize Eq. 7, and when using KL divergence we encountered small fitting problems and thus used the L2 distance. We do not expect this to make a big difference. We also point out that this metric was recently used in scaling studies of ICL as well [Anil et al 2024 https://www.anthropic.com/research/many-shot-jailbreaking], [Arora et al 2024 https://arxiv.org/abs/2410.16531].

[Continued below...]

Q13. Paper structure: On first read I found it slightly difficult to follow the paper's first few sections. ... I personally found understanding the phases helpful to my understanding of the remainder of the paper. I wonder if the authors have considered promoting section 4 to come before section 3? Of course, this is up to the authors.

This is a fair question. Our main motivation for the paper structure stemmed from the intuition that, despite its commonplace use in literature, ICL does not have a formal definition. Hence, to study the concept, we should develop a setup that captures as much phenomenology as possible, using the phenomenology itself as the definition of ICL: i.e., if there is a setup that yields a model that captures most known results on ICL, then that setup becomes worth of studying the concept of ICL in a unified manner. Motivated by this then, our paper layout became: (i) propose our task, (ii) demonstrate it reproduces phenomenology of ICL to argue that the setup is worth studying, and (iii) develop insights into ICL.

We have tried to make this narrative more fluid in the updated version of the paper; we’d be grateful if the reviewer can take a look.

Q14. Terminology and notation: A small number of minor notes. 1. Have you thought about whether the 'phases' are indeed phases in the sense of physics? 2. A bold "1" is overloaded as both a vector of ones (when describing the configuration of the Dirichlet distribution) and also an indicator function (line 232)....

Response to 14.1. Thank you for this question! As our title suggests, we indeed drew inspiration from the term "phases" in physics. While establishing an exact analogy is challenging due to thermodynamics' focus on "equilibrium states," careful examination of this analogy was the key inspiration that shaped our work. Below, we elaborate on this point, which may bring further clarity to the issues we discussed earlier, such as behavioral metrics, mechanisms, and the nature of explanations.

The first key insight from physics is how scientists defined phases in the early days of thermodynamics, even without knowing that atoms existed! Notably, in physics, new invention of "order parameters" defines new phases.

• Macroscopic Order Parameters (Thermodynamics): Historically, thermodynamics developed through the identification of macroscopic variables like pressure, temperature, and volume, and their relationships in describing systems' bulk properties. Phase transitions were identified by discontinuities or abrupt changes in these variables or their derivatives. For instance, water's volume changes abruptly when it freezes into ice at 0°C. Our use of behavioral metrics to differentiate algorithmic solutions in Transformer models parallels the employment of macroscopic order parameters in thermodynamics. These evaluations detect changes in the model's output behavior without examining internal mechanisms. Thus, we acknowledge that our analysis is not "mechanistic" at this stage!

• Microscopic Order Parameters (Statistical Mechanics): With the emergence of atomic theory, statistical mechanics provided a microscopic "mechanistic" understanding of thermodynamic phenomena. Microscopic order parameters, such as molecular arrangement and orientation in crystal lattices, offered mechanistic insights into phase transitions. To achieve a similar mechanistic understanding, indeed, we need to examine the network's internal variables, including weights, activations, and neural representations. These internal variables function as microscopic order parameters, and it would Their analysis could reveal how different algorithmic strategies are implemented within the network's architecture.

To contextualize in the modern deep learning landscape, for example, a remarkable recent sequence of developments can be seen in the behavioral characterization of hidden progress measures [1], followed by the formulation and discovery of mechanistic analogues [2]. Overall, the parallel between studying physical systems and neural networks has helped us understand the reasoning behind your comments on naming and framing. Like all early scientific endeavors, considerable work remains to develop these ideas further, and your thoughtful feedback has brought clarity to our thinking about this topic!

Response to 14.2. Thank you! We changed this to the Kronecker delta.

[1] Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

[2] Progress measures for grokking via mechanistic interpretability

Q15. Typos

Thank you for highlighting these typos; we have fixed them in the revised manuscript!

[Continued below...]

Q8. What is the relationship between LCA and delta metrics from Raventos et al.? ... Have you thought about the relationship between these metrics and LCA?

This is a great question! Unfortunately, we haven’t dived deep into making a metric level comparison with Raventos et al. recently. We would have ideally found a chance to re-read the paper and identify a relation during the rebuttals stage, but the paper edits kept us extremely occupied. We promise to attempt a reconciliation of their metrics and LCA before the final version of the paper is to be submitted.

Q9. How close is the LCA fit? LCA weights are defined via a least squares optimisation problem. In figures 6 and 7 the authors plot the argmin weights. What is the min? In other words, how large is the irreducible component of the least squares loss representing the distance of the transformer from the simplex spanned by the four algorithms in function space?...

Thank you for these suggestions! We have now added these quantifications to the manuscript.

1. The residual probability space L2 (the argument of argmin in Eq. 7) is shown in Fig. 38 in Appendix H.1. The residual L2 is at the 0.001 scale, which is very small given that probability vectors are normalized to sum to unity. Thus, our four algorithms indeed span a simplex which can model the Transformer accurately. Interestingly, we find the phase boundaries show up as regions with increased—but still very low in an absolute sense—L2.

2. This is another important quantification, and we now show this in Fig. 39 in Appendix H.1. We show that the KL divergence between the model and the LCA predictions are very low in either direction of KL computation, compared to the KL divergence seen in Fig. 3, 5, 6, 7. Here again, we find a small increase of KL at the phase boundaries.

We thus conclude that the four solutions do provide a good basis to model the trained Transformer.

Q10. Non-ICL algorithms early in training: Figure 6 shows shifts in the model's development across training. One lesson from Hoogland et al. ... Have the authors considered adding such non-ICL algorithms into the LCA analysis? I would be curious whether doing so reduces the residual at all.

This is a great question and we would have been really curious to attempt it! Unfortunately, there was only so much time and we ended up prioritizing other experiments for now. However, we would like to note this parallel ICLR submission that has performs an analysis that may be inline with what the reviewer is suggesting (it is not LCA exactly, but the authors demonstrate a competition dynamics w.r.t. a memorization solution that can dictate learning dynamics earlier in training).

Q11. Questions about evaluation: Two small questions about the methodology for evaluation. 1. During ID evaluation, what prior do you use for sampling the tasks from the set of tasks? Do you use the training prior or a uniform prior? You just say you 'choose one from a set.’ 2. During evaluation (for both ID evaluation and OOD evaluation), how many tasks do you sample?

1. We sample from the training prior to keep the distribution exactly the same. We have clarified this in Appendix A.3.2.

2. We sample 30 tasks and k (=10) sequences from each task. This detail is available in Appendix A.3.2.

Q12. Sample implementation of data generating process: Could the authors please clarify the relationship between the sample implementation of the data generating process and the actual code used in experiments?...

The sample implementation and the actual DGP used in experiments produces exactly the same data. The DGP we actually used simply has more class hierarchy, methods, attributes, tokenizing tools, etc. We also clarify that Appendix A.1 does mention the initialization of samples from the stationary distribution. This code block is mostly added since our DGP is very simple and some readers would prefer to read code instead of equations. The codebase will be open sourced, but we intend to keep this code block in the manuscript for clarity.

[Continued below...]

Question 2:

First, we thank the reviewer for pointing out this paper again! We had missed it during our initial submission, and agree that this paper is indeed more similar to our proposed task. We now cite it in Appendix C and Figure 27 (will be reflected in the next version of the paper).

Main answer. We certainly think the "forgetting" (term used by Panwar et al. for transience) is a much more similar phenomenon to what we see, modulo the sequence modeling aspect that we have, which adds another dimension of analysis: solution complexity (unigram vs bigram). And with our study, we can now interpret there are competing solutions underlying their observations!

Quoting Panwar et al.’s Section 6.1:

We group them into the following 4 categories based on OOD loss and describe the most interesting one in detail (full classification in §D.3): (1) 2 1 to 2 3 : no generalization; no forgetting; (2) 2 4 to 2 6 : some generalization; no forgetting; (3) 2 7 to 2 11: full generalization and forgetting [...] (4) 2 12 to 2 20: full generalization; no forgetting.

These results can be almost one-to-one related to: (1) no generalization; no forgetting -> Fig. 33 a (A Uni-Ret solution emerges which doesn’t generalize and thus cannot show transience) (2) some generalization; no forgetting -> Fig. 33 b (From a mixture of Uni-Ret and Uni-Inf, we observe an initial increase of KL followed by a decrease) (3) full generalization and forgetting -> Fig. 33 d, e (Bi-Inf solution being overtaken by Bi-Ret) (4) full generalization; no forgetting -> Fig. 33 f (The Bi-Ret solution not appearing within the computation budget)

We again note we have added a discussion of this paper in Appendix C, where we discuss the similarity of our results to Panwar et al.'s, and also discuss the relation between their setup and ours. We thank the reviewer for bringing this paper to our attention!

Thank you for continuing the discussion!

We would first like to note that your argument makes perfect sense to us, and we had not considered this perspective: the distinction between the idealized algorithms and the corresponding implementation a model arrives at is a really good idea! In fact, rereading Singh et al., we now realize that they explicitly state in their paper something to the effect of: (i) an induction head is (likely) needed for their task to be solved entirely in-context; (ii) since attention is used to implement this, and attention learned via gradient descent can likely implement induction in only a fuzzy way, the ICL implementation will always have some uncertainty (and hence non-trivial loss); (iii) the IWL solution, as it is implemented, does not need to be fuzzy and can hence have lower uncertainty / loss; and (iv) this likely leads to IWL overtaking ICL eventually. While this argument does make explicit the distinction between idealized and implemented algorithms, we really like this vocabulary and the nuance you are suggesting!

Overall, then, we see your point now! We are more than happy to revise the next version of the paper to discuss in detail the difference with respect to Singh et al.'s setup for Transience and our own setup. Specifically, we will emphasize the distinction between the idealized and implemented algorithms, but also clarify that the dynamics of transience are led by one solution having a better loss than the other. This loss imbalance can emerge either because of how the problem is specified (as in our case) or how the solution is implemented (as in Singh et al.'s case).

Finally, thank you again for this message! This was very a really clarifying note for us.

Questions

This paper identifies a number of these interesting phenomena, but it would be nice if the authors could discuss a bit more about why these phase transitions occur. ...

Great question! Based on our current understanding of the learning dynamics, we believe the following picture is likely at play under these transitions.

• Transition with training time. When the model begins to utilize bigram statistics of the context, we see a rather sharp drop in KL for bigram-based algorithms. Interestingly, this sharp transition correlates with the form of a precise attention head structure: specifically, the induction head. Please see the newly added Fig. 19, 20, 21 for a mechanistic analysis of the attention layers supporting this point. This structure allows the model to compute bigram statistics, and hence enables the learning of solutions that can achieve a better loss (specifically, a Bayesian solution that uses bigram statistics). We also note that this induction head formation takes place at generally the same number of training iterations in our phase diagrams, i.e., this transition is likely optimization-limited (with a weak dependence on data diversity, which must just be higher than some critical value). We note we have also added this discussion to Appendix J.

• Transition with data diversity. As we increase data diversity, we find the model behavior is explained by an algorithm that involves inferring the transition matrix from context, instead of relying on chains seen during training (we now call this algorithm Bi-Inf; its earlier name was Bi-ICL). Our newly added mechanistic analysis provides hints for what is going on in this transition. Specifically, we find that at low data-diversity values, the model memorizes the transition matrices of Markov chains seen during training in its MLP neurons (see Appendix E.1). Then, as we increase data diversity, we find the model starts to represent these transitions across several neurons (i.e., in superposition). While this superposition works well at medium data-diversity values, at very high values, it is difficult for the model to achieve it quickly; i.e., it can likely still memorize all transition matrices, but it can take a long time to do so. Meanwhile, as the model goes through training, the ability to utilize bigram statistics kicks in. Since the model has still not memorized the transition matrices well by this point, this renders the Bi-Inf (i.e., what used to be called Bi-ICL) algorithm the best solution to the task! However, with further training, the Bi-Ret algorithm (i.e., what used to be called Bi-Bayes) takes over since it finally achieves a lower loss on the training set. Intriguingly, we find that we can again start to get decent reconstruction of the Markov chains in this phase! That is, this cross-over of Bi-Inf to Bi-Ret is driven by a persistent pressure to memorize the training distribution.

We hope the above summary is helpful, and we are happy to expand further if needed!

Summary: We thank the reviewer for their detailed feedback, which has helped us improve the rigor of our linear interpolation of algorithms (LIA) analysis by adding several interesting baselines, and also helped us contextualize the practical value of our work. We also appreciate the nudge to better justify the algorithmic phases seen in our results. We hope the responses above address the reviewer’s concerns, and, if so, they would consider championing our paper during the discussion period!

I think I want some kind of control condition where you use 3 or 4 different “silly” algorithms just to confirm that these kinds of phase transitions aren’t some artifact of fitting some linear combination of algorithms and that, if your algorithms are not related to the task, you don’t always see these interesting algorithm phase transitions.

This is a great suggestion! Following your comments, we have now added Fig. 40, 41 in Appendix H, where we run the following two experiments. These results provide validation to both: (i) the algorithms we claim to underlie our phase diagram, and (ii) the methodology of linearly interpolating these algorithms’ predictions to describe model behavior to identify which phase the model is in when using a given experimental configuration.

• Experiment 1: We define four relatively ad-hoc algorithms and try to decompose model next-token probabilities across them using LCA (now renamed to LIA–linear interpolation of algorithms). Results, shown in Fig. 40, demonstrate that there are no interesting transitions when using the four ad-hoc algorithms for LIA! The largest weight for each algorithm simply remains roughly constant throughout checkpoints.

• Experiment 2: We also perform another version of the above experiment, where we mix the four ad-hoc algorithms to our four core algorithms identified using tests on bigram utilization and Bayes proximity (yielding a total of 8 algorithms). We then perform our analysis on these 8 algorithms simultaneously. Results are shown in Fig. 41, and we find that our four core algorithms again dictate model behavior in precisely the same regions we claimed to be their corresponding algorithmic phases. The four ad-hoc algorithms are never assigned a phase (or for that matter any mass in the interpolation) of their own.

Further, we also refer the reviewer to updated Fig. 5d, and the new Fig. 35 in Appendix F.1.4 These figures show KL divergence between our model’s next token predictions and the four algorithms that we claim to underlie our phases. We find the phase boundaries very closely match the boundaries of low KL regions in these diagrams, further validating our algorithms and phase diagram!

I think I also wanted a bit of better understanding on how well LCA predicts the performance of the transformer relative to some naive baselines just to confirm that LCM well approximate the transformer.

This is also a great idea! Following your comment, we have now run the following experiment.

Defining the naive baseline. In general, when doing LCA (now called LIA in the paper), we optimize the weights for linear interpolation at a given checkpoint. Now, to define our baseline, we run the optimization process over all checkpoints simultaneously––that is, we try to identify weights such that a linear interpolation of our algorithms would best predict the model’s behavior when averaged over time. We note that to induce some diversity and make this baseline somewhat difficult, we do allow the other control variable in our experiments (i.e., data diversity) to vary.

Results. See Fig. 42 in Appendix H. We find that when we fix the weights across checkpoints, the error on the fit KL divergence increases substantially. Thus, we demonstrate that the “basis” of four solutions are indeed well modeling the transformer’s implemented solution.

[Continued below...]

We thank the reviewer for their careful reading of the paper and valuable feedback! We are excited to see they found our work to be "thoughtful and creative", “nice, comprehensive”, and that they "like the idea of approximating the transformer's behavior as a mixture of algorithms", which "provides a compelling explanation for phenomena like transience."

We have made significant changes to our manuscript:

• A major revision of the writing, revising some terminologies, improving the flow and highlighting contributions.
• A large set of mechanistic experiments to confirm the existence of the algorithms we delineate. (Fig. 15-26)
• Additional quantifications of experimental results to verify our findings. (Fig. 5d, 35, 38-42)
• Additional visualizations of results for completeness. (Fig. 32-34)

Below, we address specific comments from the reviewer.

First of all, I want to emphasize that I think this line of work is valuable and scientifically meaningful. However, it would be nice to discuss how these insights translate into practical design choices (e.g., predicting OOD performance using a similar approach to this paper).

Thank you for this comment! While we do believe our work suggests directions that may be worth pursuing for practical design choices, we would like to first emphasize that the very fact we are engaging in this discussion indicates our study has offered an important practical takeaway: our results suggested that models may be undergoing a competition of algorithms when performing ICL, updating our perspective on how one should think about ICL, and also prompting the reviewer to ask how one can tame the competition. We believe this discussion could not have been had before, and hence the opening of this discourse can likely lead to useful work down the line. For example, we may find research work motivated towards design of contexts that bias the model towards an algorithm over another one in the competition.

That said, we summarize what we believe are the most crucial takeaways for practice.

• Doing better evals: Effects of context scaling. One of the crucial axes that our work explores is context-size scaling, demonstrating that a larger vs. shorter context results in very different algorithms succeeding the competition. Most current benchmarks primarily focus on use of 2–32 exemplars for few-shot capabilities evaluation, but we believe this may be underestimating model capabilities, e.g., by focusing on retrieval / Bayesian algorithms. There is a chance that merely scaling the context and adding more exemplars will significantly improve model performance on downstream tasks, without further training advancements. Results in this spirit have started to emerge (e.g., see [1]), but our work offers a formal model for why context scaling should be considered more seriously in capability evaluations. In fact, the sudden learning of capabilities via context-scaling, as has been shown in some recent work (e.g., see [2, 3, 4]) may even be explainable by our competition of algorithms picture!

• Linear combination of algorithms may offer a tool for OOD performance prediction. Assuming one can identify what algorithms are engaged in the competition to dictate model behavior during ICL, our results show that it is likely possible to predict model performance on OOD samples by merely using in-distribution inputs! We do not claim this finding will necessarily be of relevance to LLMs, where the pretraining data is too vast in its scope, but Transformers and their ability to perform ICL are nowadays used as backbones for several applications. For example, in the application of in-context RL [5, 6], one hopes to in-context learn a world model and then have the Transformer engage in goal-directed behavior on novel environments. We believe such tasks are sufficiently simple to speculate what policies (algorithms) the model may be following, decompose its behavior across them, and then predict behavior for novel environments!

[1] https://arxiv.org/abs/2403.05530

[2] https://www.anthropic.com/research/many-shot-jailbreaking

[3] https://arxiv.org/abs/2310.17639

[4] https://openreview.net/forum?id=pXlmOmlHJZ

[5] https://arxiv.org/abs/2306.14892

[6] https://arxiv.org/abs/2210.14215

[Continued below...]

In a Bayesian approach ... Given this, shouldn’t the KL divergence be examined between the model prediction and the true posterior or posterior predictive distribution (according to Bayes' rule and the correct prior) at each context length...?

Thank you for this suggestion! We certainly agree that behavioral equivalence between our model and predefined algorithms will be better demonstrated by computing the KL between model predictions and algorithms’ posterior predictive distribution. To this end, we have now made the following updates to the paper.

• KL between model and algorithms in Figure 5d. We have now updated Figure 5 by adding a panel (d) that shows the KL between our model’s predictions and the algorithms’ posterior predictive distributions. We find a clear validation of our claims: specifically, we see the identified algorithmic phases for different experimental configurations (Figs. 5a–5c) perfectly match the regions of low KL with respect to the corresponding algorithms in Fig. 5d!

• KL with context-scaling. We have now added Figure 34 where we chose 4 checkpoints in the phase space where the model is expected to be dominated by one algorithmic solution and show the model and solution’s expected KL divergence as we change the input context length. We find a clean verification of our claims: the model prediction robustly follows the KL expected from each algorithmic solution for different context sizes, both for ID chains and OOD chains. This figure is actually very convincing, thank you very much for suggesting this!

We thank the reviewer again for suggesting this experiment!

I understand that you may be viewing task-retrieval vs. task-learning ... this distinction could also be seen as reflecting a finite, discrete prior versus a continuous or uniform prior. ...

As we note in our previous answers, we used terms that indicate a distinction between task-retrieval vs. task-learning primarily to stay consistent with prior work (especially the paper by Raventos et al.). We had a footnote in the paper explaining, however, that we do not believe this dichotomy is real. To repeat, we had written: “We refer to these solutions as non-Bayesian, in line with Ravento ́s et al. (2023), while acknowledging that they can be interpreted as a form of Bayesian inference with a relaxed prior (see App. F.1).” We then clarified in the referenced appendix that in fact the non-Bayesian solutions have a continuous or uniform prior, and the Bayesian ones had a discrete prior over chains seen during training.

Overall, we believe that this consistency with prior work has not been useful and caused confusion. To this end, we have now updated the names of our algorithms and emphasized in Section 3 that, in fact, they can both be deemed as Bayesian inference with different forms of priors: one over just the seen training chains, and one a continuous prior over the entire data distribution.

Summary: We thank the reviewer for their detailed feedback that has motivated us to contextualize our work with respect to prior results, further emphasizing its novelty. We are also grateful for reviewer’s comments to better clarify the distinction between families of algorithms considered in this work, helping us significantly update Section 3 and add several new experiments mechanistically grounding these algorithms in Appendix E. We hope our responses have addressed the reviewer’s concerns, and, if so, they would consider raising their score to support the acceptance of our work!

Questions

The Bayesian vs. Non-Bayesian comparison: Generally, I think of Bayesian learning as a form of "learning to learn," where a posterior is updated based on observed data according to Bayes' rule, with a prior typically established during pre-training. Therefore, it is unclear why the comparison between ID and OOD performance specifically reflects a shift between Bayesian and non-Bayesian paradigms...

Thank you for raising this point! We first emphasize that we used the terms Bayesian vs. non-Bayesian primarily to be consistent with prior work by Raventos et al., who deem these two approaches to be different in nature. While we had mentioned this motivation in Footnote 3 in the paper, i.e., that both approaches can in fact be understood as Bayesian inference with different forms of prior, we realize now that this terminology actually induced confusion and the footnote did not adequately address the issue. To this end, we have now significantly improved the paper’s writing to address this ambiguity. Specifically, we made the following updates.

• Update names to remove the Bayesian vs. Non-Bayesian dichotomy. We have now updated the names of our algorithms to remove any ambiguity that one algorithm is Bayesian and another is not. Specifically, we call the two approaches “fuzzy retrieval” versus “inference” of the relevant transition matrices. We have updated the discussion around these terms to emphasize, however, that they are both Bayesian inference approaches under the hood (as discussed below).

• Updated Section 3 to emphasize that both approaches involve a Bayesian averaging operation. We have updated Section 3’s writing to strongly emphasize in the main text itself that, in fact, both the retrieval and inference approaches involve a Bayesian averaging operation: the precise prior involved in this operation is quite different though, limiting the generalization abilities of one family of algorithms over another. Specifically, in the fuzzy retrieval approaches, the prior covers merely the Markov chains seen during training, while in the latter it is a relaxed prior that uniformly covers the entire Dirichlet distribution. This affects their ID and OOD generalization abilities, as discussed next.

• Improving the discussion for why Retrieval approaches underperform on OOD inputs. As the reviewer correctly points out, in Bayesian inference the posterior is indeed updated based on observed data (i.e., at inference time) and the prior over training chains is established during pre-training. However, our results show that the form of these priors can be drastically different depending on the data diversity in the experimental setup: this can be seen, e.g., in Eq. 2 and 3 where the likelihood calculation (and thus the posterior calculation) requires knowledge over the training set matrices. The model is still "learning to learn" in the sense that it is trying to ascertain which seen Markov chains best explain the observed data (i.e., the context), and using a weighted average of those chains to continue the sequence (hence performing “learning”). However, the two inference solutions (i.e., what used to be called “non-Bayesian solutions”) in Eqs. 5, 6 do not involve any information about the training set, and thus will have no performance gap between ID and OOD. That is, since their prior covers the entire training distribution (i.e., the Dirichlet prior), they guarantee the same performance on both ID and OOD data. We have updated Section 3.1 to emphasize the above.

• Mechanistic evidence to justify poor OOD performance of retrieval algorithms. While the low (and at times literally zero) KL to our predefined algorithms indicates the model is behaviorally equivalent to them, we note that we have now provided crucial mechanistic evidence to justify these claims. Specifically, in Appendix E, we now demonstrate that we can literally reconstruct transition matrices from MLP neurons when the model is engaging in behavior similar to retrieval algorithms (see App. E.1)! We show that this does not work when the model is in “inference” phases, i.e., is relying on a relaxed prior.

We hope the above changes help address the reviewer’s concerns, and please let us know if there are any further clarifications that can be of help!

[Continued below...]

• Analyzing implementation of behavioral algorithms (New Appendix E). To further address your feedback on how to best leverage our system for extracting novel insights, we have added a comprehensive new section with four key analyses (See Appendix E.1-E.4). First, we successfully reconstruct training transition matrices from MLP weights in low data diversity settings, demonstrating a direct neuronal implementation of retrieval solutions (Appendix E.1). Second, we track neuron memorization dynamics by measuring KL divergence between neuron outputs and transitions, revealing systematic changes in memorization that align precisely with our algorithmic phase transitions (Appendix E.2). Third, we analyze attention maps to demonstrate distinct architectural signatures for each algorithm - uniform attention for Uni-Inf, induction head patterns for Bi-Inf, and patterns for Bi-Ret (Appendix E.3) combines information from two subsequent tokens, in line with our behavioral observations. Finally, we validate our Linear Interpolation of Algorithms (LIA) framework by showing that attention patterns at phase boundaries smoothly interpolate between adjacent algorithms (Appendix E.4). Together, these analyses provide mechanistic evidence for how identified behavioral algorithms are implemented and compete, providing a mechanistic foundation and novel insights!

We thank you for your valuable feedback, which prompted us to thoroughly reconsider and enhance how we present our work's goals, approach, and insights. We have expanded our discussion of specific contributions in Appendix C, newly created Appendix E, and revised the manuscript to better emphasize how our empirically grounded system advances beyond simple reproduction. The revised manuscript now offers refined insights into the algorithmic nature of In-Context Learning, including new results that analyze how transformers implement these algorithms through our minimal, yet rich experimental setup.

[Continued below...]

We thank the reviewer for their thorough feedback! We are glad they found our study "very detailed and sound", "very empirically rigorous" and offering “a valuable unifying study that synthesizes and reproduces previous findings in a single framework”. We also appreciate the feedback for better clarifying our novel contributions, and we have made substantial improvements in our updated manuscript to address this.

We have made significant changes to our manuscript:

• A major revision of the writing, revising some terminologies, improving the flow and highlighting contributions.
• A large set of mechanistic experiments to confirm the existence of the algorithms we delineate. (Fig. 15-26)
• Additional quantifications of experimental results to verify our findings. (Fig. 5d, 35, 38-42)
• Additional visualizations of results for completeness. (Fig. 32-34)

Below, we respond to specific comments by the reviewer.

My main concern with this paper is its novelty: ... this paper offers a valuable unifying study that synthesizes and reproduces previous findings in a single framework, the setting itself has also been examined in prior work (e.g., Edelman et al.)....

Thank you so much for raising this important point for clarification! We'd first like to emphasize that while reproducing ICL phenomena in a singular model system was a crucial step for validating our framework, our ultimate goal was to harness this system’s simplicity and richness to analyze the mechanisms underlying ICL’s phenomenology and characterize their precise implementations in transformers. Below, we clarify three key novel contributions of our work.

• Unified setup for studying ICL. Our primary motivation for reproducing ICL’s phenomenology in a singular setup was to demonstrate the remarkable richness of our proposed task. Despite its simplicity, our task captures most (if not all) known phenomenology of ICL, providing a first step towards a unified understanding of the concept. This is precisely why we spent only half a page on those experiments, instead devoting most of our paper to our novel analysis of linear interpolation of different ICL algorithms. While Edelman et al. studied a setup similar to ours, i.e., learning to simulate Markov chains, their work involves learning with infinite chains. Our strategic choice of finite mixtures enabled us to capture an array of crucial ICL phenomena, including data diversity effects, transient nature of ICL, model width scaling effects, and several others listed in Appendix C. We emphasize these results could not have been shown in Edelman et al.’s setup, restricting the generality of their claims on how ICL works.
• Novel analysis of linear interpolation of different ICL algorithms. Our thorough grounding of the system via reproduction of phenomenology was not an end in itself, but rather a crucial step toward deeper mechanistic insights. The interpretability gained through simplification led us to identify four distinct algorithmic phases with linear interpolation of different ICL algorithms. To the best of our knowledge, this analysis has never been attempted before! With this approach, we characterized the transitions between these algorithmic phases as a function of data diversity, optimization steps, and context size. This interpretability offered us a novel framework of "competitive dynamics of algorithms", revealing how different algorithmic strategies are promoted or suppressed by varying factors of data diversity, optimization steps, and context size. Through this lens, we have been able to not just reproduce, but also refine our understanding of six distinct phenomena of ICL previously reported in the literature. For example, we can now explain the origin of the non-monotonic nature of ICL as a process where a certain algorithm gets promoted during training, but then suppressed by another algorithm driven by the training loss. We can also explain the impact of model width scaling, tokenization, and other architectural design choices, and we in fact demonstrate that there was a confounding factor in the effect of model width scaling in prior work by Kirsch et al.! (See Appendix C.2.2 and C.2.3 for discussion on the latter.)

[Continued below...]

We thank the reviewer for their thorough review and feedback! We are glad that they found our paper “well-written", our experiments “sound, novel and unifying”, and our analysis “strong; revealing new insights”! We also appreciated reviewer’s emphasis on what we believe to be our main contributions: 1) unification of ICL phenomenology with a novel setup and 2) downstream insights from algorithmic decomposition.

We have made significant changes to our manuscript:

• A major revision of the writing, revising some terminologies, improving the flow and highlighting contributions.
• A large set of mechanistic experiments to confirm the existence of the algorithms we delineate. (Fig. 15-26)
• Additional quantifications of experimental results to verify our findings. (Fig. 5d, 35, 38-42)
• Additional visualizations of results for completeness. (Fig. 32-34)

Below, we respond to specific comments by the reviewer. Please note that we changed the names of algorithms from Uni-Bayes, Bi-Bayes, Uni-ICL, Bi-ICL to respectively Uni-Ret, Bi-Ret, Uni-Inf, Bi-Inf in the manuscript. However, in our responses below we will continue using the original names to ensure ease of readability.

(eg, why Dirichlet?)

Great question! We note that the Dirichlet distribution is a commonly used prior for categorical or Multinomial variables in Bayesian inference. Given we are working with discrete state spaces, a Dirichlet prior thus makes intuitive sense for our problem setting as well. This is also likely the reason for the prior’s common use in recent work exploring Transformers trained on Markov chains [1, 2]. That said, we emphasize we did not see any noticeable effects in our preliminary experiments with a uniform prior and a Poisson prior. Hence, to stay consistent with prior work, we preferred to use a Dirichlet prior in our experiments.

[1] https://arxiv.org/abs/2402.11004

[2] https://arxiv.org/abs/2407.17686v1

[Continued below...]

Minor comments

The paragraph on l. 341 is not detailed enough, and doesn’t explain what the Fig. 5 shows. It also doesn’t help that Fig. 5 doesn’t have clear legend on the heat map scales — is darker more Bayesian or less Bayesian for 5.b?

Fair point! To address your comments, we have now made the following updates to Fig. 5 and the corresponding paragraph.

• Figure 5: We now clearly mark the meaning of individual colorbars to help identify what a brighter or darker color means for a given heatmap. We have also added another panel to this figure, wherein we demonstrate that our identified algorithmic phases in panel (a)–(c) yield very low KL from the model’s next-token probabilities. This helps further ground our analysis.
• Paragraph corresponding to L.341 (now found at L.345): We have expanded the discussion in this paragraph substantially, discussing first the algorithmic phases identified using our evaluations, then the impact of different control axes (data diversity and training) that drive transitions between the different algorithmic phases, and then emphasizing several other experiments that corroborate this analysis. These additional experiments are available in Appendix E,F Fig 15~35.

Some parameter choices in the analysis are weird: l. 187, why plot at train step 839 specifically? l. 192, why at n = 2^7? Can we have similar figures for later training steps and for larger n’s in the Appendix?

We apologize for any confusion caused by missing details! To address your comments, we have now added notes on why we use specific values for the main results, and added plots ablating those values to show our claims remain consistent regardless of the precise value chosen.

• Why step 839: Since our goal at L.187 (and hence Fig. 3.b) was to demonstrate the data diversity threshold needed for OOD generalization similar to Raventos et al. 2023, we chose the checkpoint at step=839 as the threshold is most easy to visualize there. This is because step=839 is a checkpoint where we can observe the data diversity dependent transition without confounding factors from the transient nature of ICL. However, we note that the plots look qualitatively similar for checkpoints from step 600 to 2200. To this end, we have now added plots for checkpoints from these ranges in App. F.1.3 Fig 32.
• Why n=2^7: (the manuscript now uses $N$ instead of $n$) Similar to above, we chose $𝑛=2^7$ as the data diversity to plot since that value best illustrates the transient nature of ICL. Again, plotting for $𝑛=2^6$ or $𝑛=2^8$ also yield us the same qualitative effect, but the the Bi-ICL phase is respectively too short and too long for these values, hence not yielding the clearest visualization of transience. However, to emphasize that our results are robust, we have now added additional plots in App. F.1.3 Fig. 33 for different values of $n$ and noted in the main paper that results for alternative settings can be found in the appendix.

Summary: We thank the reviewer for their detailed feedback, which has helped us significantly improve the clarity of our writing and helped us demonstrate the robustness of our claims via addition of several new plots that ablate control variables! We hope our responses address the reviewer’s concerns and, if so, they would consider championing our paper during the discussion period.

Lack of actionable takeaway message. Because the benchmark is synthetic, it’s unclear how much it says about ICL for real-world LLMs. ... I liked the flavor of paragraph on l. 301 but it’s unclear how to replicate this study on real-world LLMs. ... What do we do with the discovery that ICL is best thought of as a mixture of algorithms?

Thank you for this comment! While we do believe our work suggests directions that may be worth pursuing from a practical standpoint, we would like to first emphasize that the very fact we are engaging in this discussion indicates our study has offered an important takeaway: our results suggested that models may be undergoing a competition of algorithms when performing ICL, updating our perspective on how one should think about ICL, and also prompting the reviewer to ask how one can tame the competition! We believe this discussion could not have been had before, and hence the opening of this discourse can likely lead to useful work down the line. For example, we may find research work motivated towards design of contexts that bias the model towards an algorithm over another one in the competition.

That said, we summarize what we believe are the most crucial and actionable takeaways of our results:

• Doing better evals: Effects of context scaling. One of the crucial axes that our work explores is context-size scaling, demonstrating that a larger vs. shorter context results in very different algorithms succeeding the competition. Most current benchmarks primarily focus on use of 2–32 exemplars for few-shot capabilities evaluation, but we believe this may be underestimating model capabilities, e.g., by focusing on retrieval / Bayesian algorithms. There is a chance that merely scaling the context and adding more exemplars will significantly improve model performance on downstream tasks, without further training advancements. Results in this spirit have started to emerge (e.g., see [1]), but our work offers a formal model for why context scaling should be considered more seriously in capability evaluations. In fact, the sudden learning of capabilities via context-scaling, as has been shown in some recent work (e.g., see [2, 3, 4]) may even be explainable by our competition of algorithms picture!

• Linear combination of algorithms may offer a tool for OOD performance prediction. Assuming one can identify what algorithms are engaged in the competition to dictate model behavior during ICL, our results show that it is likely possible to predict model performance on OOD samples by merely using in-distribution inputs! We do not claim this finding will necessarily be of relevance to LLMs, where the pretraining data is too vast in its scope, but Transformers and their ability to perform ICL are nowadays used as backbones for several applications. For example, in the application of in-context RL [5, 6] , one hopes to in-context learn a world model and then have the Transformer engage in goal-directed behavior on novel environments. We believe such tasks are sufficiently simple to speculate what policies (algorithms) the model may be following, decompose its behavior across them, and then predict behavior for novel environments!

[1] https://arxiv.org/abs/2403.05530

[2] https://www.anthropic.com/research/many-shot-jailbreaking

[3] https://arxiv.org/abs/2310.17639

[4] https://openreview.net/forum?id=pXlmOmlHJZ

[5] https://arxiv.org/abs/2306.14892

[6] https://arxiv.org/abs/2210.14215

[Continued below...]

The Bayesian vs non-Bayesian discussion is unclear.

Thank you for raising this point! The distinction between these two sets of approaches (Bayesian vs. non-Bayesian) is subtle and indeed worthwhile being clear over. To this end, we have now significantly updated Section 3, highlighting how one set of algorithms involves Bayesian averaging with a prior defined over the set of Markov chains seen during training (we now call this set the fuzzy retrieval algorithms), while another set of algorithms involves Bayesian averaging with a relaxed prior, i.e., the Dirichlet distribution (we now call this set the inference algorithms). We note we have also improved the discussion in Appendix B.1, where we discuss in more detail how these two sets of approaches are related.

In regards to the effect of choosing test chains differently, we highlight our preliminary results in Appendix G on test chains that do not come from a Dirichlet prior: we find that models in the Bi-ICL phase in fact generalize perfectly well to an entirely OOD chain!

[Continued below...]

Dear Reviewer vTGM,

Thank you for your thoughtful and insightful feedback which has helped us significantly improve our manuscript. As the extended discussion period ends tomorrow, we wanted to follow up on our response and the substantial updates we've made to address your concerns.

We recognize that we have made numerous changes which may take time to digest. Thus, we would like to concisely summarize the key updates addressing your main points:

1. Regarding the "lack of actionable takeaway message":

   • We have added extensive mechanistic experiments (new Fig. 15-26) that validate the existence of the algorithms we identified
   • New quantitative results (new Fig. 5d, 35, 38-42) provide concrete evidence for the importance of context-scaling effects and potential for OOD performance prediction

2. On the "Bayesian vs non-Bayesian discussion":

   • We have significantly revised Section 3 to clarify the distinction between fuzzy retrieval algorithms and inference algorithms, updating these terminologies throughout the paper
   • Added preliminary results in Appendix G demonstrating perfect generalization to OOD chains in the Bi-ICL (now called Bigram Inference) phase

3. Addressing presentation concerns:

   • Added clear colorbar annotations to Fig. 5 and expanded the corresponding discussion
   • Included comprehensive ablation studies (new Fig. 32-34) showing robustness across different parameter choices
   • Added extensive explanations in Appendix E,F justifying our parameter selections
   • Demonstrated the consistency of our findings across different model architectures and sizes (new Fig. 38-42)

We would greatly appreciate if you could review these updates and consider updating your score if you find that we have adequately addressed your concerns. If you have any remaining questions or would like clarification on any point, we are happy to respond promptly.