We thank Reviewer LvzK for the thoughtful feedback and insightful questions. We are encouraged that the reviewer found our work to be a "meaningful contribution" to the important area of understanding ICL. We address the specific weaknesses and questions below.

W1: "The stylized setting is quite specific, the claim is more general."

We agree with the reviewer on the importance of clarifying the scope of our claims. While our experiments are necessarily conducted with a regression problem in a stylized setting to benchmark ICL against the Bayes optimal estimator, our central claim about the diminishing sample efficiency of ICL is grounded in a general theoretical analysis that is not specific to our experimental setup.

Our information theoretic analysis hinges on a general condition on a shape of the excess risk curve (Assumption 4.1) sufficient to cause the diminishing efficiency in long context. This condition is not tied to a specific model size or data type. Besides, the sufficient condition stated in Assumption 4.1, which is about the existence of lower bound of the excess risk curve, align well with empirical observations in large-scale models. For instance, Anil et al. (2022) and Zhou et al. (2024) show that state-of-the-art LLMs struggle when the context length at test time significantly exceeds that seen during training. These challenges transformers face with length generalization prove the existence of the excess risk’s lower bound in the length generalization regime.

Therefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.

(Line 317) “We remark that the deficiencies of state-of-the-art LLMs beyond length generalization regime (Anil et al., 2022; Zhou et al., 2024) prove the existence of the excess risk’s lower bound within the length generalization regime.”

Anil et al. (2022). Exploring length generalization in large language models. In NeurIPS.

Zhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.

W2: "I understand that the authors aim to illustrate the phenomenon, and I consider this a valuable contribution. But I think the abstract could be more informative about the learning problem considered."

Thank you for this valuable suggestion. To provide readers with a clearer problem setup and the complexity involved in the learning task, we have revised the abstract (Lines 6-7) as follows:

“To investigate this, we adopt a meta ICL setup where each prompt corresponds to a regression problem, with the target function sampled from a hierarchical distribution, requiring inference over both latent model class and task parameters. In this setting, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under diverse performance requirements.”

W3: "The information-theoretic analysis is interesting, but it is not directly compared to experimental results. Perhaps mine is an unpopular opinion, but I feel that the technical details do not add much to the experimental results here. I'd rather see more experimental results on different learning problems. "

We appreciate this insightful comment. While more experiments on diverse tasks would certainly add empirical breadth, our theoretical analysis in Section 4 provides a unique and vital contribution that experiments alone cannot. The theory does not just support the experimental findings; it provides a causal explanation for them. It demonstrates that the observed diminishing sample efficiency is not an artifact of our specific model, task, or training procedure, but rather an inherent limitation of ICL mechanism when faced with a large number of demonstrations. By identifying the formal conditions under which this inefficiency occurs, our theory correctly attributes the phenomenon to the ICL mechanism itself. We believe this provides a more fundamental insight than additional experiments on other tasks would.

To clarify this crucial role of our theory, we have added the following sentence to the manuscript (Line 276):

“This theoretical grounding is critical, as it establishes that the diminishing inefficiency is an intrinsic property of the ICL mechanism, rather than an artifact of a particular experimental setup. This correct attribution is essential for guiding future work aimed at overcoming this limitation.”

W4: "The related work section is rather light. I note that the most recent reference is more than a year old (essentially an epoch in modern ML research). Expanding this section would allow the authors to better place their results in the growing body of research seeking to characterize and understand ICL with "physics-style" or "synthetic benchmark" approaches."

​​

Thank you for the great suggestion! We agree that contextualizing our work within the rapidly evolving literature is essential. We have updated our related work section to include several relevant recent papers that adopt similar stylized settings to understanding ICL.

(Line 400): “More recently, these stylized settings have been used to probe other sophisticated behaviors of ICL. This includes analyzing transformers' in-context model selection and preference for simpler hypotheses (Deora et al., 2024; Elmoznino et al., 2025), their ability to infer causal structures (D’Angelo et al., 2025), and the implicit connection between ICL and low-rank updates to MLP layers (Dherin et al., 2024).”

(Revised Line 402) This new perspective unveils a unique insight: the fundamental inefficiency of ICL in the many-shot learning regime, a critical limitation that was not the focus of these prior analyses.

Deora et al. (2025). In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly. arXiv.

D'Angelo et al. (2025). Selective induction heads: How transformers select causal structures in context. In ICLR.

Elmoznino et al. (2025). In-context learning and Occam's razor. In ICML.

Dherin et al. (2025). Learning without training: The implicit dynamics of in-context learning. arXiv.

Q1: "Raventós et al (NeurIPS 2023) argue that transformers achieve better generalization performance by failing to be Bayes optimal (with respect to the empirical pretraining distribution). My understanding is that in the current submission, transformers never outperform Bayes optimal algorithms. I would appreciate a discussion on this point. "

We appreciate the reviewer’s important point and clarify the difference between the notion of Bayes optimality in our work and in Raventos et al. (2023).

• Raventós et al. (2023): Optimality w.r.t. the finite pretraining data. They show that as pretraining data diversity increases, transformers deviate from the Bayes-optimal predictor defined on that finite pretraining set. This deviation actually improves generalization because it moves the transformer’s behavior closer to a ridge estimator that is optimal for the true underlying data distribution.
• Our work: Optimality w.r.t. the true data-generating distribution. We define the Bayes-optimal estimator with respect to the ground-truth hierarchical distribution from which tasks are sampled. By definition, no algorithm can achieve a lower average risk on this distribution. For a side note, the notion of Bayes optimality here is the same as in the optimality of the ridge estimator in Raventós et al. (2023).

In short, Raventós et al. show that transformers are not optimal with respect to their limited training data, which allows them to be more optimal with respect to the true data source. Our work compares transformers to the theoretical ceiling of performance—the true Bayes-optimal estimator. Thus, there is no contradiction: transformers never outperform our Bayes-optimal baseline, which is consistent with the findings of Raventós et al. when properly interpreted.

To clarify this in the paper, we have revised Line 118:

“The Bayes-optimal estimator defined in (2) minimizes the expected risk with respect to the true hierarchical data-generating distribution. This is distinct from the notion of optimality with respect to an empirical pretraining distribution with finite samples, as considered in Raventós et al. (2023), where deviating from empirical optimality can improve generalization.”

We hope these revisions and clarifications have fully addressed the reviewer's concerns. We thank the reviewer again for the constructive engagement.

We thank Reviewer pLhk for their time and effort in reviewing our manuscript and providing constructive feedback. Below, we carefully address the reviewer’s comments, clarify certain misconceptions, and emphasize the novelty and significance of our findings with support from recent relevant studies.

W1: "From expressivity point of view the transformer can implement many principled algorithm in its forward pass, an idea popularized by Oswald et al.[1] and many other follow-ups. So, if that's the case then I do not see the central point of this paper. The context length for fixed error level should be exactly the same for transformer or base algorithm."

We agree that transformers, in theory, have the capacity to implement principled learning algorithms in their forward pass (von Oswald et al., 2023). However, the crucial distinction—and the central point of our paper—lies between this theoretical potential and the algorithms that are actually learned through standard meta-training. The work by von Oswald et al. is an existence proof, showing that a specific weight configuration can mimic an algorithm like gradient descent; it does not guarantee that pre-training will converge to this or any other optimal implementation.

Our work investigates this exact gap between theory and practice. We ask: What kind of algorithm do transformers learn, and what are its limitations? Our key finding is that while the learned algorithm appears near-optimal in the few-shot regime (a finding consistent with previous works, e.g., Garg et al. (2022) and Panwar et al. (2024)), its sample efficiency degrades significantly in the many-shot regime. This reveals a critical limitation of what is learned in practice, which is the novel insight of our work.

This also clarifies the reviewer's comment on sample complexity. If a transformer did perfectly implement the Bayes-optimal algorithm, its sample complexity would indeed be the same. But since our findings show it implements a suboptimal algorithm, its sample complexity is naturally different and, as we demonstrate, worse. To clarify this in the manuscript, we have added the following: (Line 54) "While transformers are theoretically capable of implementing principled algorithms (von Oswald et al., 2023), we show that ICL, actually learn by training, deviates significantly from optimality in the many-shot regime.”

Garg et al. (2022) What can transformers learn in-context? a case study of simple function classes. In NeurIPS.

Panwar et al. (2024) In-context learning through the Bayesian prism. In ICLR

W2: "Related to 1, ICL is nothing special from the traditional learning paradigm. We have some demonstration and we want to predict a new query, so the transformer can be thought of as an algorithm potentially."

We respectfully disagree with the characterization that "ICL is nothing special." While ICL can be viewed as an algorithm, its underlying mechanism is fundamentally distinct from traditional learning paradigms, and this distinction is critical to our paper's contribution.

Traditional supervised learning involves updating a model's explicit parameters (weights) via an optimization process like gradient descent. In contrast, ICL performs learning purely through inference within a fixed-weight network. The "learning" happens as information provided in the demonstrations is processed without any weight updates.

This mechanical distinction is not trivial; it imposes unique constraints that do not exist in the same way for traditional learners. Investigating the properties and limitations of this unique inference-based learning mechanism is an active and important area of research (e.g., Brown et al. 2020; Xie et al., 2022; Akyürek et al., 2023; Jeon et al., 2024; Elmoznino et al., 2025). Our work contributes directly to this by providing a new, formal understanding of ICL's sample efficiency in long context.

Brown et al. (2020) Language models are few-shot learners. In NeurIPS.

Xie et al. (2022) An explanation of in-context learning as implicit Bayesian inference. In ICLR.

Akyürek et al. (2023). What learning algorithm is in-context learning? investigations with linear models. In ICLR.

Jeon et al. (2024) An information-theoretic analysis of in-context learning. In ICML.

Elmoznino et al. (2025). In-context learning and Occam's razor. In ICML.

W3: "I think the main problem of this paper is Assumption 4.1, basically as the model gets larger and more expressive, and also the number of pre-training increases we can get closer and closer to the best possible error. And this is aligned with scaling-law common wisdom."

We thank the reviewer for raising this point, as it allows us to clarify the scope and basis of Assumption 4.1. Reviewer pLhk’s concern appears to stem from a misunderstanding, connecting our assumption to scaling laws.

Our assumption is not about performance scaling with model size or pre-training data. Instead, Assumption 4.1 simply states that the excess risk curve is lower bounded when it confronts a sufficiently long context exceeding the context length during pretraining. Far from being "restrictive or unrealistic," this is a robust and widely observed empirical phenomenon. It is a core finding in the literature on length-generalization limitations in transformers (Anil et al., 2022; Zhou et al., 2024) and is a pattern we explicitly validate in our own experiments (Figures 3 and 4). Therefore, we think that Assumption 4.1 is a reasonable, realistic, and empirically grounded foundation for our theoretical analysis of why ICL's efficiency degrades in the many-shot regime.

Anil et al. (2022). Exploring length generalization in large language models. In NeurIPS.

Zhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.

W4: "Related to 3, because the transformer is meta-trained across many tasks, it can, in principle, exploit cross-task regularities that a hand-crafted Bayes estimator (tuned per task) cannot access. The analysis ignores this potential advantage."

As Reviewer pLhk pointed out, transformers' meta-learning capabilities allow them to exploit cross-task regularities, potentially offering advantages over handcrafted estimators learned for each task.

First, our empirical setup explicitly accounts for this advantage. We pre-train the transformer on a diverse, hierarchical distribution of tasks. The ICL performance we report is therefore the result of the model already having exploited these cross-task regularities.

Second, and more fundamentally, Reviewer pLhk might misunderstand the role of the Bayes-optimal estimator in the benchmark. The Bayes-optimal estimator is not a "hand-crafted Bayes estimator (tuned per task)." It is a theoretical performance ceiling derived analytically from the true, known data-generating distribution (cf. Eq. 2). By definition, no algorithm, whether it is a transformer using meta-learning or any other method, can achieve a lower expected risk on this distribution.

Therefore, our comparison is fair: we compare the transformer's meta-learned performance against the theoretical best-case performance on the same task distribution. The fact that the transformer falls short, particularly in the many-shot regime, is a significant and non-obvious finding.

We hope this clarifies that our assumptions are well-grounded and our findings provide a novel and important insight into the fundamental limitations of in-context learning.

I thank the authors for the clarification. However, my concerns are still not addressed:

About W1, W2: What I meant by saying that ICL is not special is that, at test time after the pre-training phase, it can be viewed as implementing a fixed algorithm that consumes the in-context demonstrations and returns an estimate for the query. How far this algorithm falls from the optimal one depends on several factors, including:

1. The training dynamics of transformers

2. The number of pre-training samples (e.g., in the hypothetical extreme of training on all possible samples, one would expect optimal performance)

3. Context length

I agree that context length matters, but transformers are highly expressive (even a two-layer MLP is quite expressive), so in theory they can implement any algorithm. If they do not, it must be due to either an insufficient number of pre-training samples or limitations in the training procedure. Hence, I do not understand the abstract’s claim: “Through an information-theoretic analysis, we show that the diminishing efficiency is inherent to ICL.” Why is it inherent?

W3: I still do not see why Assumption 4.1 is reasonable. Length generalization issues may simply reflect limitations of the training dynamics. If a transformer could implement the optimal algorithm (acknowledging this is unrealistic), it should generalize perfectly to longer contexts. This is therefore a limitation of the training process, not of ICL itself.

In addition, although your experimental setting has been extensively studied in prior work, it is far from the realistic ICL observed in LLMs. In this setup, the models are deliberately trained on explicit ICL tasks, unlike the natural emergence of ICL, making the setting less relevant.

Overall, I remain unconvinced by the paper’s main message and take-aways and thus keep my score unchanged.

We sincerely thank Reviewer N11X for the positive assessment and insightful, constructive feedback. We have revised our manuscript to incorporate the excellent suggestions, which we believe have strengthened the paper.

W1: “The authors draw connections between their results and large language models. However, it is unclear whether the decreased efficiency of ICL in their particular setup and model size persists in LLMs, where the models are significantly larger and trained on a wide range of demonstration sizes. In the authors' setup, they only considered a fixed largest demonstration size”

This is a crucial point. While our experiments use a controlled setup, we argue that the phenomenon we identify—diminishing ICL efficiency with long context—is a fundamental issue likely to generalize to large-scale models. Our reasoning is twofold:

• General Theoretical Foundation: Our theoretical analysis in Section 4 is formulated with a general setting. The core finding relies on Assumption 4.1, which is not tied to a specific model size or data distribution. As a result, the theoretical insights we offer remain broadly applicable, transcending the specifics of our experimental setup.
• Alignment with Large-Scale Empirical Evidence: The practical consequence of our theoretical finding is that Assumption 4.1 is a sufficient condition for the diminishing efficiency in long context. Crucially, Assumption 4.1 generality aligns well with empirical observations from recent large-scale transformer studies. For instance, Anil et al. (2022) and Zhou et al. (2024) both report that even state-of-the-art LLMs struggle to extrapolate when test-time context length exceeds that seen during training. Given that the Bayes risk improves with more demonstrations, this means that the excess risk curve is deteriorating in the extrapolating regime.

Therefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.

Anil et al. (2022). Exploring length generalization in large language models. In NeurIPS.

Zhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.

W2: "The analysis is based on the Assumption that there exists a uniform lower bound on the excess risk after a certain demonstration length, which can be too strong to hold."

We thank the reviewer for highlighting this critical detail, and we have revised the manuscript to clarify the scope of Assumption 4.1. The assumption is not intended to be a single universal bound that holds for all models and tasks. Rather, it posits that for a given transformer and a given task environment, a lower bound of the excess risk exists (especially when generalizing to the demonstration size significantly larger than that of used in training).

This model-and-environment-specific assumption is strongly supported by empirical evidence. We observe this performance plateau directly in our own experiments (Figures 3 and 4). Furthermore, this behavior aligns well with empirical observations in large-scale models. For instance, Anil et al. (2022) and Zhou et al. (2024) show that state-of-the-art LLMs struggle when the context length at test time significantly exceeds that seen during training. These challenges transformers face with length generalization prove the existence of the excess risk’s lower bound in the length generalization regime.

To ensure this is clear, we have explicitly revised the statement of the assumption in the manuscript to be model and environment-dependent (Line 314):

“For an environment $\mathcal{E}$ and a transformer $TF_\theta$, there exist constants … ”

W3: “This work only considered in-context supervised regression problems, leaving classification problems and other learning paradigms, such as reinforcement learning, unexamined.”

This is a correct observation, and we thank the reviewer for suggesting we make our scope and its justification clearer. We have now explicitly addressed the focus on regression and other limitations in the revised manuscript.

Our focus on regression is a deliberate methodological choice. This paradigm allows for the analytical derivation of the Bayes-optimal performance, providing a "gold standard" benchmark. This is essential for our goal of rigorously and quantitatively measuring the sample efficiency of ICL relative to a theoretical optimum. While extending this analysis to classification or RL is an exciting future direction, it poses challenges in analytically deriving the optimal baselines.

To improve transparency, we have made the following additions to the manuscript:

(Line 6) “To investigate this, we adopt a meta ICL setup where each prompt corresponds to a regression problem, with the target function sampled from a hierarchical distribution, requiring inference over both latent model class and task parameters. In this setting, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under diverse performance requirements.”

(Line 81) “Following common practice in the meta ICL literature, we focus on regression problems for which the Bayes-optimal performance can be derived analytically, allowing us to precisely quantify the sample complexity of in-context learning and principled learning algorithms.”

(Line 554) “Further, the model size and training data diversity used in our meta ICL setup are significantly smaller and less diverse than those used in modern LLMs, such as GPT-4 and Gemini 2.5, which typically possess hundreds of billions or even trillions of parameters trained on vast, heterogeneous datasets.”

Q1: “I see the authors use the Transformer as their default model architecture to examine the drop in sample efficiency for in-context learning with long contexts. I wonder what special properties of the Transformer they leveraged for their analysis? I don't see any architectural requirement in their assumptions. Thus, why don't the authors also consider other sequence modelling models, such as the LSTM, as ICL is not exclusive to the Transformer?”

This is an excellent question that prompted us to conduct new experiments to strengthen our claims. Our initial focus on the Transformer was due to its dominance in ICL research and state-of-the-art large language models. However, the core of our theoretical analysis is not specific to the attention mechanism but rather to the general properties of sequence models, as long as it satisfies the conditions of our analysis (e.g., Assumption 4.1).

Motivated by the reviewer's suggestion, we ran additional experiments with LSTM and Mamba architectures. The results are highly informative: The Transformer consistently outperformed both LSTM and Mamba in overall ICL performance, aligning with recent comparative studies (Akyürek et al., 2024; Lee et al., 2024). Crucially, the central phenomenon of diminishing sample efficiency in long context persisted across all three architectures. This finding is also consistent with recent work analyzing the length extrapolation limits of Mamba (Ben-Kish et al., 2025). These new findings strongly reinforce our central claim: the diminishing efficiency of ICL in long context is a general property of the ICL paradigm in sequence models having a non-vanishing excess risk, not an artifact of the Transformer architecture. This underscores the broad applicability of our theoretical framework. We will add a discussion of these results to the revised manuscript.

Akyürek et al. (2024). In-context language learning: Architectures and algorithms. arXiv

Lee et al. (2024). Is attention required for icl? exploring the relationship between model architecture and in-context learning ability. ICLR

Ben-Kish et al. (2025) Decimamba: Exploring the length extrapolation potential of mamba. In ICLR.

Limitations: See the responses to W3 above.

We thank the reviewer again for the insightful suggestions, which have significantly improved the rigor and clarity of our work.

We are grateful to Reviewer kMFD for the encouraging and exceptionally clear feedback. We are pleased that the reviewer found the few-shot vs. many-shot dichotomy "clear and compelling" and our analysis "solid." The questions raised are thoughtful and address the core foundations of our work as well as its broader implications.

W1: “The primary limitation, which the authors acknowledge, is the use of a stylized setting (regression on a mixture of Fourier series). Although it's a clean setup, it raises questions about the generalizability of the findings to the complex, high-dimensional, and often discrete tasks that real-world LLMs handle.”

This is a crucial point. While our experiments use a controlled setup, we argue that the phenomenon we identify—diminishing ICL efficiency with long context—is a fundamental issue likely to generalize to large-scale models. Our reasoning is twofold:

• General Theoretical Foundation: Our theoretical analysis in Section 4 is formulated with a general setting. The core finding relies on Assumption 4.1, which is not tied to a specific model size or data distribution. As a result, the theoretical insights we offer remain broadly applicable, transcending the specifics of our experimental setup.
• Alignment with Large-Scale Empirical Evidence: The practical consequence of our theoretical finding is that Assumption 4.1 is a sufficient condition for the diminishing efficiency in long context. Crucially, Assumption 4.1 generality aligns well with empirical observations from recent large-scale transformer studies. For instance, Anil et al. (2022) and Zhou et al. (2024) both report that even state-of-the-art LLMs struggle to extrapolate when test-time context length exceeds that seen during training. Given that the Bayes risk improves with more demonstrations, this means that the excess risk curve is deteriorating in the extrapolating regime.

Therefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.

Anil et al. (2022). Exploring length generalization in large language models. In NeurIPS.

Zhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.

W2: “The theoretical results hinge on Assumption 4, which posits a non-vanishing lower bound on the excess risk. While this assumption is well-supported by the paper's own experiments, it is still an assumption about the behavior of transformer-based ICL. I would like to hear more discussion about this assumption.”

See the response to Q1.

Q1: "The core of the theoretical argument rests on the non-vanishing excess risk. Your experiments (Fig 3b, Fig 4) convincingly show this behavior. Do you have a hypothesis for why the Transformer architecture, trained with this objective, leads to a non-vanishing excess risk? Is it related to the fixed-capacity nature of the attention mechanism or something else?"

This is an excellent question that prompted us to conduct new experiments to strengthen our claims. Our initial focus on the Transformer was due to its dominance in ICL research and state-of-the-art large language models. However, the core of our theoretical analysis is not specific to the attention mechanism but rather to the general properties of sequence models, as long as it satisfies the conditions of our analysis (e.g., Assumption 4.1).

Motivated by the reviewer's suggestion, we ran additional experiments with LSTM and Mamba architectures. The results are highly informative: The Transformer consistently outperformed both LSTM and Mamba in overall ICL performance, aligning with recent comparative studies (Akyürek et al., 2024; Lee et al., 2024). Crucially, the central phenomenon of diminishing sample efficiency in long context persisted across all three architectures. This finding is also consistent with recent work analyzing the length extrapolation limits of Mamba (Ben-Kish et al., 2025). These new findings strongly reinforce our central claim: the diminishing efficiency of ICL in long context is a general property of the ICL paradigm in sequence models having a non-vanishing excess risk, not an artifact of the Transformer architecture. This underscores the broad applicability of our theoretical framework. We will add a discussion of these results to the revised manuscript.

Akyürek et al. (2024). In-context language learning: Architectures and algorithms. arXiv

Lee et al. (2024). Is attention required for icl? exploring the relationship between model architecture and in-context learning ability. ICLR

Ben-Kish et al. (2025) Decimamba: Exploring the length extrapolation potential of mamba. In ICLR.

Q2: "You conclude by motivating a new generation of 'on-the-fly' adaptive methods without the diminishing efficiency. Could you elaborate on what such methods might look like?"

Thank you for this excellent forward-looking question. Our paper's theoretical analysis provides a clear target for designing the next generation of methods: they must directly address the non-vanishing excess risk, which we identify as the root cause of diminishing ICL efficiency and is widely observed in many transformer-based models and other sequential models.

One promising direction is a method we might call Residual In-Context Learning. This approach augments the fixed base model with a lightweight, adaptive module (e.g., a linear model on the base model's features) trained at inference time. Using only the in-context examples, this module's goal is to learn the base model's residual error for the current task. Because this adaptive module is a principled learning algorithm, its error in estimating the residual would vanish as the number of demonstrations grows. Then, the final output can be obtained by correcting the original ICL prediction with the on-the-fly estimate of the excess risk, which could make the excess risk vanishing and therefore restore efficient learning behavior in long context that the base ICL model lacks.

We emphasize, however, that this is a conceptual direction, and realizing such a hybrid system would require significant further investigation.