In-Context Occam’s Razor: How Transformers Prefer Simpler Hypotheses on the Fly

Thank you for your time, careful reading and feedback. We appreciate your recognition of our paper's contribution in investigating, clearly experimenting with, and theoretically supporting a novel hypothesis.

Link to PDF with additional results.

“Results seem to not clearly support the claim that on order-1 chains the model resorts to using the simpler hypothesis.”

We appreciate your careful reading, and we recognize the importance of clearly demonstrating the results. We maintain that our argument for the Occam's razor-like inductive bias is strongly supported by a comprehensive look at our experimental evidence. Let us clarify our interpretation with further detail.

1. Figure 1 (Order-1 vs. Order-3 Comparison): For order-1 contexts, we do observe a clear separation in Figure 1, where KL (bigram ‖ model) ≈ 0.004 is an order of magnitude smaller than KL (tetragram ‖ model) ≈ 0.05. We agree the figure is somewhat cluttered, and we will add these numerical values to the figure caption to emphasize this significant gap, which indicates the model's stronger alignment with the simpler bigram statistics.

2. Effect of Context Length (Figure 7 in App. A.2): Your point about Equation (5) and statistics becoming equal for long sequences is crucial. Our experimental design accounts for this prerequisite. Fig. 7 in App. A.2 clearly illustrates this: on order-1 sequences, KL(bigram ‖ model) stays essentially flat and low, indicating a consistent fit to the simplest statistic. In contrast, KL(tetragram ‖ model) is large for short contexts (where higher and lower-order statistics are less similar), and steadily drops as the context grows (as expected by Eq. 5). The clear gap for short contexts validates our hypothesis that the model prioritizes the simpler explanation.

3. Figure 2 (Order-1 vs. Order-2 Comparison): You rightly note that the gap is smaller in Fig. 2 when comparing order-1 and order-2 statistics. This is natural, as the complexity difference between these two categories is minimal and from the point above we cannot expect a large gap for large sequence lengths. We acknowledge that this might not be our most indicative comparison. We are running an additional experiment with shorter-length evaluation which we will report and use to replace this figure.

4. Failure to Generalize from Complex-Only Training (Appendix B): Our finding in Appendix B is central to our argument. Here, we found that a model trained exclusively on complex (order-3) sequences fails to infer the correct statistics of an order-1 sequence when presented with one (Fig. 5, left, in App. B). If the model in Fig. 1 (trained on both order-1 and order-3 sequences) was also merely applying its most complex learned hypothesis, it would behave similarly as Fig. 5, whereas we observe in Fig. 1 that it has lowest KL divergence with the bigram statistics on order-1 sequences. These observations demonstrate a selective preference rather than a passive fit when training on a mixture of orders, strongly supporting our Occam's razor hypothesis.

5. Occam's-Razor Inductive Bias Consistent Across Testbeds: The same implicit Occam's razor-like bias clearly reappears in our second suite of experiments with linear regressors. In Figures 3a (left) and 3b (left) in the paper, when the test-context length $T$ is such that $d>T>d/2$ (meaning both hypotheses perfectly fit the context data), the transformer's predictions align far more closely with the "simpler" $d/2$-dimensional least-squares solution.

Across both testbeds, the empirical evidence and our Bayesian analysis consistently point to the same behavior: the transformer balances fit against complexity and selects the simplest adequate explanation.

To summarize, based on your feedback, we will make the following specific revisions to the manuscript: We will add numerical values for KL(bigram ‖ model) and KL(tetragram ‖ model) to the caption of Figure 1 to highlight the clear separation on order-1 contexts. We will further emphasize the nuance of context length in the main text, explaining how Fig. 7 in the paper (reproduced as Fig. 6 in the attached PDF) directly accounts for Equation 5 and validates our hypothesis in the relevant short-context regimes.

Thank you for the feedback. We are encouraged that you view our work as a theoretically strong contribution to an important topic and that you appreciate our experimental design. We address specific comments below.

Link to PDF with additional results.

“It is not clear that the results fully reflect real-world environments, which is the primary motivation of this paper.” “Findings may not generalize to more realistic, real-world settings.”

As stated in the introduction, our goal is not to replicate every facet of real-world languages, but to isolate and study one dimension that existing synthetic setups overlook: hierarchical task complexity. As you rightly point out in your summary of our paper, prior work has largely examined fixed-complexity tasks; in contrast, we carefully construct two (Markov-chain and linear-regression) testbeds that deliberately span multiple, nested hypothesis categories. This setup systematically shows whether transformers lean towards simpler or more complex explanations in a clean, controlled setting, uncovering our finding of an Occam's razor-like inductive bias. We believe this controlled approach is essential for rigorously establishing the existence of this inductive bias.

While our primary focus is on uncovering and explaining an underlying mechanism of in-context learning in the most transparent terms, we acknowledge the importance of demonstrating the relevance of our findings to more realistic settings. To that end, motivated by your questions, we have conducted additional experiments on pretrained LLMs (please see response 3 in the attached PDF, where we include results for a pre-trained GPT-4 model). These new results provide evidence that similar inductive biases can also be observed in larger, pre-trained models.

“The experiments themselves are not thoroughly described”

Given your acknowledgment elsewhere of our "two testbeds carefully described" (referring to our main Markov Chain and Linear Regression testbeds, which are indeed detailed in the main body), we suspect this concern primarily refers to the probabilistic context-free grammar (PCFG) experiment. You are correct that, in optimizing for space in the main paper, we were brief in describing this particular experiment, deferring additional details to Appendices A.4 and C. To address this, we will enhance the description of the PCFG experiment in the main body of the revised manuscript to ensure its design is clearer. If you have specific questions about the setup after reviewing these appendices, please let us know; we are happy to provide further explanations.

Q: Why select GPT2? Effect of changing model architecture?

We chose GPT-2 style transformers as our primary architecture due to their widespread adoption in prior studies on ICL with linear-regression and Markov-chain tasks (e.g., Garg et al., 2022; Park et al, 2024, Edelman et al. 2024, Akyürek et al., 2023). Adopting this commonly used architecture allows for direct comparison and builds upon existing insights within this research line.

To assess the impact of different model architectures and scales, we have conducted additional experiments on LSTM architectures and a model scale ablation. Please see response 1 and 2 in the attached PDF for details. We find that LSTMs exhibit a slightly weaker form of Occam’s razor-like inductive bias, as they require significantly more capacity to consistently select the correct underlying Markov chain order from the input sequence at inference time compared to transformers. Regarding model scale, we find that larger models also exhibit the Occam’s razor-like inductive bias, and converge faster as the number of parameters is increased. We will add these experiments and discussions in the revision.

We hope that these responses help address your concern and that you will consider increasing your score.

Thank you very much for your follow-up and for increasing your score! We appreciate your positive assessment and thank you again for your time and helpful feedback.

Thank you for your detailed review and positive comments on our paper. We are encouraged that you recognize the interesting nature of the research problem we study, and that our results shed light on the model's inductive biases. We also appreciate your acknowledgment of the strong empirical evidence supporting our findings and the good theoretical exposition.

Your comments have been very useful and motivating for additional experiments, which we discuss below and will add to the revision.

Link to PDF with additional results.

Effect of model scale

Please see response 2 in the attached PDF, where we include results for larger transformer models by increasing the number of layers, attention heads, and the embedding dimension. We find that larger models also exhibit the Occam’s razor-like inductive bias, and converge faster as the number of parameters is increased.

Varying proportion of simple and complex categories during training

Thank you for this great suggestion. We train transformers while systematically varying the fraction of order-1 and order-3 sequences in each mini-batch (order-1 fraction $\in [0.1, 0.9]$). For each setting, we evaluate at inference time: (i) $\text{KL}(\text{bigram} \|\| \text{model})$ and (ii) $\text{KL}(\text{tetragram} \|\| \text{model})$ on both order-1 and order-3 test sequences. The results are shown as heatmaps in Fig. 4 of the attached PDF.

We find that the transformer reliably learns bigram statistics for order-1 sequences, even when order-1 examples make up a small fraction of the training mix. In contrast, learning tetragram statistics for order-3 sequences becomes increasingly difficult as the fraction of order-1 sequences rises—under such skew, the model tends to default to bigram-like behaviour. Moreover, the number of training steps required to acquire tetragram statistics increases as the training distribution becomes more imbalanced.

Increasing the batch size helps offset this effect: larger batches ensure that more order-3 sequences are seen per step in absolute terms, which raises the threshold at which the model can still learn the correct higher-order statistics.

Effect of curriculum-style scheduling

That’s another good suggestion. We actually used curriculum learning in the linear regression setting in the paper, following previous linear-regression ICL studies (e.g., Garg et al., 2022 and follow ups). To test its impact, we reran the experiment without any curriculum (Fig. 5, left, in the attached PDF). In that setting, the transformer eventually converges to the $d/2$-dimensional least-squares solution, but it takes substantially longer than with curriculum scheduling (Fig. 5, right). We will include this comparison and discussion in the revised version.

Q1: Key implications for real-world downstream tasks

Motivated by your questions, we have added experiments for a pre-trained GPT-4 model testing whether similar Occam's razor-like inductive biases are observed in larger, pre-trained models (please see response 3 in the attached PDF). Our findings suggest that the pretrained model also exhibits a strong preference to the simplest possible explanation among those that explain the context.

This aligns with and provides a fundamental understanding for observations in other relevant empirical work on pretrained LLMs, such as studies on ambiguous examples (Si et al., 2023; Bartsch et al., 2023). Specifically, Si et al. (2023) found that LLMs consistently favor one hypothesis over another when presented with ambiguous sentence-label pairs. Our investigation, using carefully crafted synthetic tasks, allows us to systematically vary task complexity and directly attribute inductive biases to this complexity. This can be seen as a first-principles approach to explaining such empirically observed behaviors in larger models.

While our primary aim is to uncover and explain an underlying ICL mechanism in the most transparent terms, demonstrating broader relevance is important. The new results on pretrained LLMs support the generality of our findings and help bridge our mechanistic insights to these more empirical works.

Overall, we can interpret our findings as suggesting the importance of carefully selecting in-context examples that truly reflect the desired task complexity. If simpler patterns are present, models might favor those in their predictions

Thanks a lot for the time and effort to review our work. We appreciate your positive endorsement and support.

References:

Si et al., “Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations”, ACL 2023.

Bartsch et al. “Self-Consistency of Large Language Models under Ambiguity”, BlackboxNLP Workshop, ACL 2023.

Thank you very much for your follow-up and for increasing your score! We appreciate your positive endorsement and support. Thank you again for your great suggestions that helped improve our work!

Thank you for the detailed comments and your overall positive feedback. We are encouraged that you found the paper well-written with theoretically motivated synthetic tasks for investigating hierarchical task learning in transformers, the insights interesting, and the experimental setup replicable and extensible to new tasks.

Your comments have been very useful and motivating for additional experiments, which we discuss below and will add to the revision.

Link to PDF with additional results.

Other Architectures such as LSTMs

Thank you for the suggestion. Please see response 1 in the attached PDF, where we include results for LSTMs with different number of layers and embedding dimensions. We find that LSTMs exhibit a weaker form of Occam’s razor-like inductive bias, as they require significantly more capacity to consistently select the correct underlying Markov chain order from the input sequence at inference time compared to transformers.

You rightly point out that our theoretical framing is not tied to any specific architecture. In that sense, the fact that LSTMs eventually, with sufficient scaling, can exhibit similar behavior is consistent with our analysis. However, the observed difference in efficiency and the pronounced nature of the bias in transformers confirms a stronger architectural inductive bias in transformers for this type of hierarchical task learning.

Effect of Model Size

Please see response 2 in the attached PDF, where we include results for larger transformer models by increasing the number of layers, attention heads, and the embedding dimension. We find that larger models also exhibit the Occam’s razor-like inductive bias, and converge faster as the number of parameters is increased.

Furthermore, these ablations confirm that a smaller transformer with 6-layers, 6-heads fails to learn tetragram statistics for order-3 sequences, directly supporting your intuition that a minimum model size is required to handle a given task complexity. The LSTM experiments support a similar conclusion. Thank you for suggesting this experiment, it will be a valuable addition to the revised version.

Examination of how results may generalize to real-world LLMs

Thank you for the suggestion. Please see response 3 in the attached PDF, where we include results for a pre-trained GPT-4 model testing whether similar Occam's razor-like inductive biases are observed in larger, pre-trained models. Our results suggest that the pretrained model also exhibits a strong preference to the simplest possible explanation among those that explain the context.

Overall, as you note, the ideas and insights in the paper can extend to other tasks that exhibit a hierarchy of complexity levels, beyond the settings we consider. Several recent works leverage synthetic setups (e.g., linear regression, Markov chains, Boolean functions), to study various aspects of ICL in transformers, such as which function classes can be learned in-context, what algorithm is used for ICL of linear regression, the transient nature of ICL, phase transition based on diversity of training tasks, and so on (as discussed in Section 2.2 in the paper). These studies, though grounded in synthetic settings, provide important insights into the ICL behaviour of small-scale transformers, and help inform our understanding of ICL in larger, real-world models. Our work follows a similar philosophy, focusing on ICL in settings with hierarchical task complexity, where we uncover an Occam's razor-like inductive bias.

While our primary aim is to uncover and explain an underlying mechanism of ICL in the most transparent terms, we acknowledge that demonstrating broader relevance is important. To that end, the new results on pretrained LLMs, which provide evidence that similar inductive biases can also be observed in larger, pre-trained models, will be included in the revision to support the generality of our findings.

Moving Mathematical Exposition to Appendix

We appreciate this suggestion. We agree on the importance of maintaining clarity and accessibility for the COLM audience while still providing the necessary transparency into our findings. To balance things, we will move Section 3.2.3 (Analysis for the Linear Regression Setup) to the Appendix. In its place, we will expand the analysis and discussions related to the new experiments prompted by your questions—specifically, our evaluations on pretrained models and comparisons to LSTMs.

Thanks a lot for the time and effort to review our work. We appreciate your positive endorsement and support.