Intention Model: A Novel Explanation for In-context Learning

We would like to express our gratitude to the reviewer for the time and effort dedicated to reviewing our work. In response to your comments, we have provided detailed responses below.

Response to weaknesses:

W.1: The paper's theoretical framework largely follows the derivation approach from Xie et al. (2022). The contribution seems more like an incremental step rather than a major theoretical innovation.

Ans for W.1): We apologize for the misunderstanding. To clarify our contribution, we highlight the difference between our work and the mentioned work.

• Xie et al.'s work provides an outstanding framework to explain the phenomenon of ICL [r1]. However, their work uses some strong assumptions: LLMs can exactly fit distributions and perfectly delineate task intention, under which they prove that the prediction of ICL can be asymptotically optimized as the number of examples increases.

• We propose a novel theoretical framework, i.e., the intention model. This allows for common models and demonstrations and derives the no-free-lunch theorem of ICL. Namely, whether ICL emerges depends on the prediction error and prediction noise, which are determined by i) LLMs’ prediction error of the next token, ii) LLMs’ prediction smoothness, and iii) the quality of demonstrations.

W. 2: The use of the term "No Free Lunch" for the presented theorem seems a bit off

Ans for W. 2): We apologize for the misunderstanding.

The use of the term "No Free Lunch" may be confusing to readers. We chose this term to show that no demonstration works best for all problems. However, in our theorem, we are not emphasizing a universal algorithm but rather a specific condition for ICL We will revise the terminology in the revised paper to make it clearer and more accurate.

W. 3: The experimental section lacks clarity on how each of the theoretical components manifests in practice

Ans for W. 3): Thanks for your valuable suggestions.

Our theorem shows that the ICL capacity of an LLM emerges depends on the prediction error and the prediction noise. However, it is challenging to calculate or estimate the related values, i.e., the error of next-token prediction $\delta_3$, LLM’s prediction smoothness, and demonstration shift. To validate our theorem, we design experiments to test these factors' impact implicitly.

For instance, the error of next-token prediction $\delta_3$ could be related to the LLMs' performance under general tasks. The prior error of LLMs can be regarded as the difference between the LLMs’ prediction and the actual distribution.

Thus, we could conclude that the errors of GPT-4 are less than those of LLaMa-7B. Applying an external transition matrix can increase the demonstration shift, which would lead to larger prediction noise according to the prediction noise in Eq. (14). To verify these theoretical insights, we evaluate the ICL performance of different LLMs under the scenario where the transition matrix is realized by an addition operation i.e., realizing $\mathcal{T}$ by y -> (y + 1) mod 5 or by a more complicated one y -> (3y + 1) mod 5. The results shown in Table 2 are consistent with our theoretical analysis.

Reference

[r1] An explanation of in-context learning as implicit bayesian inference.

Dear Reviewer #b9hR,

We sincerely appreciate your dedicated time and effort in reviewing our work.

If you have any additional questions or issues that require further clarification, please do not hesitate to let us know. We would be more than happy to address them promptly.

Thank you once again for your invaluable support and contributions to improving our work. We greatly appreciate your feedback.

Best regards,

Authors of #9067

W. 4: Experiment section is too small and severely cut (defered to the appendix). Also, the evidence is circumstantial.

Ans for W. 4): Thanks for your valuable comments. We will revise the paper to include more details and results in the main text. Accordingly, we will add the following descriptions to the revision.

Our intention model shows that learning behaviors, e.g., learning with flipping labels, can be modeled by multiplying a transition matrix $\mathcal{T}$ by the original transition matrix $\theta$, as shown in Eq. (14), $y_{t}(\mathcal{T}\theta_g) = {\rm \mathop{arg\; max}\limits_{y}}\; p(y| x_{t},\mathcal{T}\theta_g), \text{with} \ y(\mathcal{T}\theta_d) = {\rm \mathop{arg\; max}\limits_{y}}\; p(y| x,\mathcal{T}\theta_d)$. Based on our intention model, we have three conclusions:

• Conclusion 1: introducing an external $\mathcal{T}$ to modify original outputs makes ICL more challenging. This is because the transition matrix $\mathcal{T}$ would lead to $KL(p(x,y|\mathcal{T} \theta_d) || p_M(x,y|\mathcal{T}\theta_g))=\epsilon^\prime \geq \epsilon=KL(p(x,y|\theta_d) || p_M(x,y|\theta_g))$. Consequently, this results in 1) larger prediction errors, as shown in Eq. (7); and 2) larger prediction noise, as shown in Eq. (13). Thus, introducing an external matrix $\mathcal{T}$ will degrade ICL performance, which is consistent with the results shown in Table 2, i.e., changing $y$ to $(y + 1)\ mod \ 5$.

• Conclusion 2: an LLM with smaller error of next-token prediction $\delta_3$ performs better in overriding semantic priors under flipped label scenarios. This is because smaller prediction error $\delta_3$ can reduce the prediction noise as shown in Eq. (13). Intuitively, $\delta_3$ is related to the LLMs' performance under general tasks, i.e., $\delta_3$ of GPT4 could be less than that of GPT2. Thus, we employ three LLMs to verify the point and report their performance in Table 2. Our results verify the theoretical insights, and fortunately, this conclusion aligns well with the experimental observations [r1].

• Conclusion 3: increasing the number of demonstrations $n$ under the random label scenario lead to decreasing performance. This is because larger $n$ would magnify the impact of demonstration shift and the LLMs’ error of next-token prediction, as shown in Eq. (13). This conclusion aligns well with the experimental observations [r3]. Similarly, a small $n$ leads to good performance, which aligns with the experimental observations [r2].

W. 5: The paper is very hard to read and follow.

Ans for W. 5): We apologize for the clarity and writing issues in the paper. We will revise it to make it easier to read and follow and address all of the specific issues you pointed out.

W. 6: Presentation can be improved by removing some theoretical proofs by putting it into the appendix and describe a stronger link between the theoretical and empirical results.

Ans for W. 6): Thanks for your kind suggestion. We will put some theoretical proof into the appendix to improve the presentation. For instance, we will put Eqs. (10) and (12) in the appendix. We will add the corresponding experiments after the theoretical analysis to highlight the connection between the theoretical analysis and experiments.

We would like to express our gratitude to the reviewer for the time and effort dedicated to reviewing our work. We appreciate that you find our theoretical framework generally interesting and reasonable, and the provided link between next token prediction ability and ICL is valid and interesting. In response to your valuable comments, we have provided detailed responses below. We hope that our responses have satisfactorily addressed your concerns, thereby enhancing the overall quality of our work.

Response to weaknesses:

W. 1: The assumption 4 is strong. Moreover, there is no way to get predictions from the LLM given the same context and some different intentions \theta_different.

Ans for W. 1): We apologize for the potentially confusing assumption. Accordingly, we will add the following descriptions to the revision.

Assumption 4 only considers cases where the model infers a relatively correct intention, and if the model infers a wrong intention, i.e., $\theta \notin \varTheta_{\epsilon}$, we expect this probability to be small and bounded by Eq. (12). So this assumption is loose.

Under our theoretical framework, task demonstrations are generated based on a given task intention. It is a tiny probability event to generate the same demonstration under very different intentions, so the model will have different tendencies for the intention distribution $p_M\left(y|S^n\left(\theta_d\right), x_{t},\theta_g\right)$ generated by demonstrations generated by different intentions.

W. 2: The no-free lunch theorem in this paper does not explain the task learning capabilities of LLMs on completely novel tasks (\theta not in the intention family) unrelated to the pretraining text.

Ans for W. 2): Thanks for your valuable comments. Trained on large amounts of data, LLMs generate amazing emergence capabilities, allowing the model to handle previously unseen tasks. We will discuss potential limitations and future work to address novel tasks.

W. 3: The whole external mapping thing does not make much sense to me. Users do not provide an external mapping when getting the model outputs; they directly present demonstrations with this transformation. If the LLM infers this mapping, it can only be implicit. Making it a part of the original intention family. It is hard to tell if a mapping like flipped labels is present in the intention family learnt by the model during pretraining. If the mapping is randomly generated, this becomes a contradiction as it is indeed not present in the pretraining corpus.

Ans for W. 3): Thanks for your valuable comments. We present this external map to show more clearly how difficult it is for the model to infer different tasks. In the multiple choice test, the flipped label can be expressed by the transition matrix, which weakens the model's ability to distinguish task intention. The external mapping matters because different mapping makes a difference in the task's difficulty. e.g. The model performed worse on task $y->3y+1(\text{mod}5)$ than on task $y->y+1(\text{mod} 5)$

Response to questions:

Q. 1: Why is it called the no-free lunch theorem?

Ans for Q. 1):

The no-free lunch theorem in machine learning states that no learning algorithm is naturally met and universally superior to all others for all possible problems. In the context of this paper, the no-free lunch theorem refers to the fact that the ICL performance of an LLM will depend on the specific task and the instructions provided to the model.

Q. 2: Why do we need to have a neighborhood of intentions, which are exactly modeled, compared to Xie et al.’s exact intention, which may have some modeling error?

Ans for Q. 2):

We propose a framework that models a neighborhood of intentions rather than a single exact intention. In practice, a user's true intention may not be perfectly specified, and there may be multiple plausible interpretations of the user's input. By modeling a neighborhood of intentions, we can capture a range of possible interpretations and make predictions that are robust to variations in the user's input.

Q. 3: What is the difference between Assumption 3 and 5?

Ans for Q. 3): Assumption 3 describes that the difference in probability of similar intention producing the same next-token prediction in the real task space is bounded. Assumption 5 characterizes that the probability difference of similar intention in the distribution predicted by the model produces the exact next-token prediction is bounded. Actually, a significant improvement in our work is to incorporate the model's predictive power into the factors influencing ICL.

Q. 4: Why is it difficult to estimate next-token error of LLMs? Where is the causal link that implies that the model is figuring out this new matrix T?

Ans for Q. 4): Thanks for your valuable comments. We present this external map to mainly show how difficult it is for the model to infer different tasks. In the multiple choice test, flipped label can just be expressed by the transition matrix, which will lead to the weakening of the model's ability to distinguish task intention, i.e., $KL(p(x,y|\mathcal{T} \theta_d) || p_M(x,y|\mathcal{T}\theta_g)) \geq KL(p(x,y|\theta_d) || p_M(x,y|\theta_g))$, implicitly leads to an increase in $p_M(\varTheta_{\epsilon})$, and thus the model's performance under the flipped-label task will deteriorate.

Reference

[r1] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.

[r2] Larger language models do in-context learning differently.

[r3] In-context learning learns label relationships but is not conventional learning.

We extend our heartfelt thanks to the reviewer for the time and effort invested in evaluating our submission. It is gratifying to learn that you acknowledge the novelty of our perspective on explaining ICL. Given your constructive comments, we have prepared comprehensive responses below to address each point raised.

Response to weaknesses:

W. 1: The amount of detailed mathematical analysis in Sections 3 and 4 is dense and obscures the key take away messages from the theory. One suggestion to the authors is to reconsider whether to keep all of the technical details in the main paper, or describe the main takeaways and the theorem, but move the rest into the appendix

Ans for W. 1): Thanks for pointing out this potentially confusing manner of writing. There are many empirical studies on ICL phenomena, but few theoretical studies on the actual mechanism and influencing factors of ICL exist. Our study provides a possible theoretical framework and can be used to explain some of the ICL phenomena. Accordingly, we will add detailed descriptions to our revision. And we will reconsider the placement of some mathematical analysis and consider moving some of them to the appendix to focus more on the main takeaways and theorem in the main paper.

W. 2: The paper lacks direct empirical confirmation of its theoretical findings.

Ans for W. 2): Thanks for pointing out this potentially confusing experimental configuration. Accordingly, we will add the following descriptions to the revision.

Our intention model shows that learning behaviors, e.g., learning with flipping labels, can be modeled by multiplying a transition matrix $\mathcal{T}$ by the original transition matrix $\theta$, as shown in Eq. (14), $y_{t}(\mathcal{T}\theta_g) = {\rm \mathop{arg\; max}\limits_{y}}\; p(y| x_{t},\mathcal{T}\theta_g), \text{with} \ y(\mathcal{T}\theta_d) = {\rm \mathop{arg\; max}\limits_{y}}\; p(y| x,\mathcal{T}\theta_d)$. Based on our intention model, we have three conclusions:

• Conclusion 1: introducing an external $\mathcal{T}$ to modify original outputs makes ICL more challenging. This is because the transition matrix $\mathcal{T}$ would lead to $KL(p(x,y|\mathcal{T} \theta_d) || p_M(x,y|\mathcal{T}\theta_g))=\epsilon^\prime \geq \epsilon=KL(p(x,y|\theta_d) || p_M(x,y|\theta_g))$. Consequently, this results in 1) larger prediction errors as shown in Eq. (7); and 2) larger prediction noise as shown in Eq. (13). Thus, introducing an external matrix $\mathcal{T}$ will degrade ICL performance, which is consistent with the results shown in Table 2, i.e., changing $y$ to $(y + 1)\ mod \ 5$.

• Conclusion 2: an LLM with smaller error of next-token prediction $\delta_3$ performs better in overriding semantic priors under flipped label scenarios. This is because smaller prediction error $\delta_3$ can reduce the prediction noise as shown in Eq. (13). Intuitively, $\delta_3$ is related to the LLMs' performance under general tasks, i.e., $\delta_3$ of GPT4 could be less than that of GPT2. Thus, we employ three LLMs to verify the point and report their performance in Table 2. Our results verify the theoretical insights, and fortunately, this conclusion aligns well with the experimental observations [1].

• Conclusion 3: increasing the number of demonstrations $n$ under the random label scenario lead to decreasing performance. This is because larger $n$ would magnify the impact of demonstration shift and the LLMs’ error of next-token prediction, as shown in Eq. (13). This conclusion aligns well with the experimental observations [3]. Similarly, a small $n$ leads to good performance, which aligns with the experimental observations [2].

W. 3: The intent recognition experiment is not totally convincing. Details of the task setup are also missing.

Ans for W. 3): Thanks for pointing out this potentially confusing configuration description.

Our experiments are mainly inspired by the observation that induction heads are shown to be closely related to general ICL in LLMs [r1]. Thus, we aim to identify a set of induction heads for a given intention and verify the impact of these intentions on the generation process under a specific intention. Due to limited space, we defer the detailed setup to Appendix F. In response to your valuable comments, we will highlight these details on the main page as follows.

For the algorithm to locate induction heads, we draw inspiration from the work [r2]. Namely, we construct counterfactual examples to compare the induction heads when the intention of interest is activated and when it is not. Specifically, given a reference sample used for activating a certain intention, we construct a corresponding counterfactual example to deactivate the intention with minimal changes to the reference sample. Subsequently, we replace the induction heads of the reference sample with those of the counterfactual example. Thus, we can record the output changes when replacing each head. Consequently, induction heads that cause drastic changes in outputs are located as the candidate heads related to the current intention. Following previous work, we employ the model LLaMA-2-7B and the dataset SST-2 for the employed models and datasets.

We will highlight these details on the main page in response to your valuable comments.

W. 4: Please discuss these experiments in the main body of the paper, state their conclusions and how they support the theory.

Ans for W. 4): Thanks for your valuable comments. Accordingly, we will add the following descriptions to the main text. We will also provide more details on the dataset preparation.

Our experiments are designed to assess intention recognizability, pinpoint intention instantiation, and verify theoretical insights.

• Intention recognition. To make the concept of intention more vivid, we design experiments to verify the recognizability of intention. The intuition is straightforward, namely, the basic idea of our intention model is that LLMs can infer intentions from the demonstrations. This implies that intentions could be recognized when LLMs are conditioned on demonstrations. Thus, we collect $10,150$ prompts with $50$ distinct intentions and extract the features of these prompts using different LLMs, leading to numerous pairs of features and the corresponding label representing intentions. Some of these samples are used to train a classifier, and the left are used to evaluate the prediction accuracy of this classifier. The results are shown in Table 1.

• Intention localization. It shows that induction heads are the mechanistic source of general ICL in LLMs [1]. This motivates us to take a step further beyond recognizing intentions, namely, we aim to identify a set of induction heads for a given intention and verify the impact of these intentions on the generation process under a specific intention. Thus, we design experiments to locate intentions with results are given in Figrue 1. These results show that we can pinpoint intention instantiation, making the concept of intention more vivid.

• Insights verification. Our theorem shows that the ICL capacity of an LLM emerges depends on the prediction error and the prediction noise. However, it is challenging to calculate or estimate the related values, i.e., the error of next-token prediction $\delta_3$, LLM’s prediction smoothness, and demonstration shift. To validate our theorem, we design experiments to implicitly test the impact of these factors. For instance, the error of next-token prediction $\delta_3$ could be related to the LLMs' performance under general tasks. Thus, we could conclude that the $\delta_3$ of GPT4 is less than that of LLaMa-7B. Applying an external transition matrix can increase the demonstration shift, which would lead to larger prediction noise according to the prediction noise in Eq. (14). To verify these theoretical insights, we evaluate the ICL performance of different LLMs under the scenario where the transition matrix is realized by an addition operation, i.e., realizing $\mathcal{T}$ by y -> (y + 1) mod 5 or by a more complicated one y -> (3y + 1) mod 5. The results shown in Table 2 are consistent with our theoretical analysis.

W. 5: The writing of the paper needs significant editing and proofreading.

Ans for W. 5): Thanks for your kind suggestions. We apologize for any writing errors or unclear phrases in the paper. We will carefully rewrite and edit the paper to address these issues.

Response to questions:

Q. 1: What is “n” in Table 1? How does Table 1 “show that larger LLMs can capture the intention”?

Ans for Q. 1): “n” refers to the number of examples in the demonstrations in Table 1.

Our theorem shows the no-free-lunch nature of ICL, involving prediction error and noise. Intuitively， The error of next-token prediction $\delta_3$ could be related to the LLMs' performance under general tasks, i.e., $\delta_3$ of GPT-4 is less than that of GPT-2. Thus, we compare the ICL capability of different models, i.e., LLaMa-7B, Mistral-7B, and GPT-4, with results in Table 1, verifying that LLMs with smaller $\delta_3$ exhibit higher ICL performance.

Q. 2: Can you provide more details or samples for the dataset preparation?

Ans for Q. 2): Details, such as the dataset description and experimental settings, are deferred to the Appendix. We are glad to provide further details.

Q. 3: F.2 do the induction heads identified here affect intent recognition in section F.1? I.e, if you “knock out” the heads then extract features, does the intent prediction performance degrade?

Ans for Q. 3): Yes. Knocking out heads usually leads to performance degradation, especially for the heads related to the current intention. However, limited performance degradation is observed if we randomly knock out heads. This is consistent with our results shown in Figure 1.

Reference

[r1] In-context learning and induction heads

[r2] Localizing model behavior with path patching

The reviewer would like to thank the authors for taking the effort to submit detailed responses to my comments and revising the paper. I've raise my score to reflect that some of my concerns have been address and questions clarified.

We would like to express our gratitude to the reviewer for the time and effort dedicated to reviewing our work. We appreciate your encouraging remarks on the applicability of our proposed theoretical framework. In response to your valuable comments, we have provided detailed responses below. We hope that these responses can satisfactorily address your concerns.

Response to weaknesses:

W. 1: A crucial missing reference seems to be Lin and Lee ("Dual operating model of ICL", ICML 2024). It would help if authors can delineate how their contributions differ from that of Lin and Lee.

Ans for W. 1): Thanks for your constructive comments. Accordingly, we highlight the difference between our work and the mentioned work.

• Lin et al.'s work [r1] considers a specific regression task and assesses the performance of ICL [1]. Their work focuses on context length's impact on ICL and explains two Real-World phenomena.
• We propose a novel theoretical framework, i.e., the intention model. This allows for common models and downstream tasks and derives the no-free-lunch theorem of ICL. Namely, whether ICL emerges depends on the prediction error and prediction noise, which are determined by i) LLMs’ prediction error of the next token, ii) LLMs’ prediction smoothness, and iii) the quality of demonstrations.

In response to your kind suggestions, we will add the above discussions to the revision.

W. 2: I hence encourage authors to take a prediction-centric perspective: what predictive claim does your theory offer, and can you demonstrate that said claim checks out experimentally?

Ans for W. 2): According to your suggestions, we have highlighted the theoretical claims verified by experiments in the revision.

Our theorem shows the no-free-lunch nature of ICL, involving prediction error and prediction noise. Thus, a straightforward approach to validate the theorem is to calculate the prediction error and noise. However, it is challenging to calculate or estimate these related values, i.e., the error of next-token prediction $\delta_3$, LLM’s prediction smoothness, and demonstration shift. Thus, providing a quantitive analysis to verify the theorem is challenging.

Fortunately, some related factors can be controlled implicitly.

• The error of next-token prediction $\delta_3$ could be related to the LLMs' performance under general tasks, i.e., $\delta_3$ of GPT4 is less than that of GPT2. Thus, we compare the ICL capability of different models, i.e., LLaMa-7B, Mistral-7B, and GPT-4, with results in Table 1, verifying that LLMs with smaller $\delta_3$ exhibit higher ICL performance.

• The demonstration shift is captured by $\epsilon = KL(p(x,y|\theta_d) || p_M(x,y|\theta_g))$. According to Eq. (14), the demonstration shift would vary with the external transition matrix $\mathcal{T}$, leading to larger demonstration shift, i.e., $KL(p(x,y|\mathcal{T} \theta_d) || p_M(x,y|\mathcal{T}\theta_g)) = \epsilon^\prime \geq \epsilon = KL(p(x,y|\theta_d) || p_M(x,y|\theta_g))$. According to Eq. (13), applying $\mathcal{T}$ leads to larger prediction noise. Thus, we evaluate ICL performance under different matrix $\mathcal{T}$ with results in Table 2, where $\mathcal{T}$ is realized using different mappings, i.e., $y=y$, $y = (y+1)\ mod\ 5$, and $y = (3y+1)\ mod\ 5$. This verifies that a larger demonstration shift degrades ICL performance.

Aligning with your valuable comments, we failed to provide direct experimental validation related to the LLM’s prediction smoothness. Thus, we will explore some implicit approach to reflecting the nature of LLMs in our future work.

W. 3: I found the writing quite convoluted in several parts of the paper.

Ans for W. 3): We apologize for the convoluted writing and grammatical errors in our paper. We will carefully revise the paper to make it more straightforward and more concise and correct all citation and grammatical errors.

Response to questions:

Q. 1: Can you better explain why the first term has additive influence of next-token error and prediction smoothness (which I would have expected to themselves have a multiplicative effect)?

Ans for Q. 1): Following your valuable question, we have added detailed explanations of the additive influence of next-token error and prediction smoothness. We agree that your intuition about next-token error and prediction smoothness is correct. Within the intention neighborhood inferred by LLMs, the actual effects of these two factors accumulate gradually by multiplying, leading to a more complex error term. We want this error to be easier to understand, so we use a binomial expansion and shrink the higher-order term to a constant. This provides a more intuitive understanding of the actual effect of the error term.

Reference

[r1] Dual Operating Modes of In-Context Learning, ICML 2024