Understanding In-context Learning with a Pelican Soup Hypothesis

Thank you for your comments.

With Regard to the Weakness

1. We consider one main contribution of our work to be explaining the ICL ability in the semantically-unrelated label ICL (SUL-ICL) setting [8], which is a step forward from the theorem by [7]. Indeed, our work can not explain the random label setting in [1]. However, we would like to point out that as far as we know, there has been no theorem able to explain it. We suggest that explaining it requires making more assumptions on the relationship between the input-label mapping and the domain/topic/theme distribution in the training data.

2. We are sorry for the confusion. The justification mainly follows the argument in Section 3. We will make the connection clearer in our later revision. For now, please allow us to elaborate it below:

   • Assumption 1: It follows the efforts made by early linguistics on linguistic formalism and cognitive psychology theories such as language of thought.
   • Assumption 2: This assumption follows the argument in Section 3.1 that general text is usually organized in a similar way as logical induction processes. We assume that “a paragraph is a traversal of the nodes of the proving tree in the topological ordering” because in general text, statements usually follow some causal and/or chronological order, so it can convince readers or be easy for readers to understand.
   • Assumption 3: In other words, this assumption assumes that every paragraph in an article mentions the topic of the article.
   • Assumption 4: In other words, this assumption assumes that some pronouns are used in the article and each pronoun in the article is associated with a single entity.
   • Assumption 5: This assumption follows the argument in Section 2.

   In our opinion, these assumptions are relatively mild compared with previous theoretical works. We are happy to discuss more if you still have some concerns about the realism of these assumptions.

3. To make our main response short, please allow us to discuss it below.

4. Thank you for the list of relevant works. We will include some of them in our later revision (as we have some concerns about the technical soundness of some of them).

Atomic elements of NLP tasks in [2]

If we understand [2] correctly, [2] does not specify what characteristics the “atomic skills” of NLP need to have. We think our work could be an instantiation of the “atoms” proposed in [2].

It seems that defining a set of “atomic skills” is not trivial and can’t be done arbitrarily. “Atomic skills” must be defined in a way such that the skills that work for the training data can generalize to the downstream task. To understand why it is non-trivial to define a set of atomic skills, we can probably think about a trivial case where we define the set “atomic skill” as the 26 skills to predict the words starting with A-Z. With this setting, the argument in [2] is greatly simplified. It seems that Theorem 14 only tells us that for most of the letters, the model can do the cloze problems in the training set well if the answer starts with that letter. It doesn’t seem to be guaranteed that the model can do the cloze problems in the testing set well.

On the other hand, defining the atomic skills with our atomic concepts seems to be more generalizable. Still, we posit it is necessary to discuss the discrepancies between the training set and the downstream task. That’s why in this work we explicitly characterize the distribution shifts in Section 4 and demonstrate that LMs are able to generalize under these distribution shifts in Section 5 and 7. Surely our settings do not fully replicate the real-world data distribution, but we think it still complements some aspects of [2].

Please let us know if you still have any questions. We are more than happy to discuss further.

[7] Xie, Sang Michael, et al. "An explanation of in-context learning as implicit bayesian inference." arXiv preprint arXiv:2111.02080 (2021).

[8] Wei, Jerry, et al. "Larger language models do in-context learning differently." arXiv preprint arXiv:2303.03846 (2023).

Thank you for your positive review.

We would like to point out that producing an environment, with which in-context learning still works, is a practice used in most (if not all) theoretical analyses such as [1] and [2] and even some empirical analyses such as [3]. We consider one of our main contributions to be proposing an environment that is closer to the real-world scenario than previous works.

With regard to the questions you stated:

• The yes/no questions in a Pelican Soup riddle have a similar function as the demonstrations for in-context learning, because based on the answer of those yes/no questions, the participants are able to figure out the latent story and thus are able to answer other following questions. Note that we use Pelican Soup riddles just to motivate the importance of the role of commonsense.
• Our argument is that solving these two problems (ICL and modeling coreference) requires similar capabilities. This argument is to draw the intuition for the framework/assumptions in Section 5.1. Surely the two problems are not identical, thus we characterize the difference between these two problems with the distribution shifts discussed in Section 4. To show that LMs may be able to generalize under these distribution shifts, we replicate these distribution shifts in our experiments, as discussed in Section 5.3 and results in Section 5.5 are aligned with our hypothesis.
• We understand ICL as generating verbalizers based on the meaning of the verbalizers, which is recovered according to the preceding context, so we think it is similar to modeling anaphora. Could you please elaborate more on why you think ICL is more like handling catephora?

Please let us know if you have any other comments. We are happy to discuss more!

[1] Xie, Sang Michael, et al. "An Explanation of In-context Learning as Implicit Bayesian Inference." International Conference on Learning Representations. 2021.

[2] Hahn, Michael, and Navin Goyal. "A theory of emergent in-context learning as implicit structure induction." arXiv preprint arXiv:2303.07971 (2023).

[3] Chan, Stephanie, et al. "Data distributional properties drive emergent in-context learning in transformers." Advances in Neural Information Processing Systems 35 (2022): 18878-18891.

We thank you for your review. We would like to address the weakness below:

1. Please let us know what are the arguments that you think are not substantiated. For the example you provided, we apologize for not including proper citations. We think that LMs are able to acquire some knowledge about commonsense (though imperfectly) has been widely accepted. Many studies can be found in previous works, e.g. [1][2].
2. Validating theories with a synthetic dataset is a common practice. We admit that our setting cannot explain domain bias. It may require more assumptions to theoretically explain. We would like to mention that, we think it will make the construction of the toy data even more complicated, which based on your 4th point, is not acceptable.
3. We justify our assumptions made in Section 5.1 in Section 3. As argued in Section 3.1, we posit that training data usually contain text that has some logical structure and/or follows chronicle order, which can contribute to LMs’ reasoning capability. The dataset Calcutec is a formal abstraction of such structure.
4. Could you be more specific on what details are missing and how we should improve? We will appreciate your suggestion. The construction of Calcutec is more complicated than the toy data used in previous theoretical works such as [3] because Calcutec relies on milder assumptions and is more realistic.
5. We apologize that we didn’t make the purpose of Section 3 clearer. We treat Section 3 as the justification/intuition for the abstraction/assumptions/framework we made in Section 5.1. We then instantiate the abstraction/framework with Calcutec and show that LMs trained with it can do in-context learning.
6. We use Pelican Soup riddles only to motivate the intuition that commonsense plays an important role for ICL. We would not say our hypothesis is based on this similarity.
7. Please let us know if you think any experiment is important but missing. Also, could you elaborate more on the possibility that the models solve the task by learning shortcuts (and maybe also your definition of shortcuts)?

Our answer to the question

1. We think mesa-optimization mainly explains why models of Transformer architecture trained with autoregressive loss have the capacity to model data that involves ICL-like phenomena. However, in works such as [4], they only study on some toy data. It is unclear how their toy data is relevant to natural language data. Our work starts from another direction, aiming to understand what characteristics of training data lead to the ICL ability.

[1] West, Peter, et al. "Symbolic knowledge distillation: from general language models to commonsense models." arXiv preprint arXiv:2110.07178 (2021).

[2] Li, Xiang Lorraine, et al. "A systematic investigation of commonsense knowledge in large language models." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[3] Xie, Sang Michael, et al. "An explanation of in-context learning as implicit bayesian inference." arXiv preprint arXiv:2111.02080 (2021).

[4] von Oswald, Johannes, et al. "Uncovering mesa-optimization algorithms in transformers." arXiv preprint arXiv:2309.05858 (2023).

Thank you for your thoughtful comments. We would like to address the weakness below:

1. In Section 2, we propose a formalism for NLP classification tasks while the main purpose of Section 3 is to justify the assumptions made in Section 5. We see the assumptions in Section 5 as the formal statement/abstraction of our theorem in Section 3. We are sorry that we didn’t describe it more clearly. We will improve this part in our later revision. Please let us know if we need to make more clarifications.
2. As far as we know, all theoretical analyses on ICL depend on some repetitive nature of prompts or training data, e.g. Xie et al and Hahn & Goyal. One difference in our work is that we focus on the repetitive usage and the consistent meaning of pronouns, which is very realistic in real-world data. In Figure 2, we also show that the model can still do ICL when the task description is more complicated than the meaning of the pronouns seen in the training set, i.e., when the task description is composed of 3 atomic concepts (the green lines for “triple” in Figure 2).
3. We discuss this point below:
   • The digit addition task: Please refer to Figure 8 for the distribution of the number of steps in the datasets. For the 5-digit addition task, when the dropping rate is 0.2, there are only 265 (0.265%) training samples where all the intermediate steps are dropped. We think it is reasonable to say the testing samples are out-of-domain in this case. (And GPT-2-sized model can achieve ~80% accuracy with only 265 training samples in the testing domain.) Meanwhile, considering that the real-world data also has a small fraction of text that drops all the reasoning steps, we think this setting is not unrealistic.
   • Calcutec: Please refer to Figure 5. It shows that even when no steps are dropped in the training set, the model still can do ICL, though the performance is not as good.

Question:

• The prompt is as the one in Table 1. We only include the reasoning steps of some (3) randomly selected examples and the premise of the testing input in the prompt. (For the example in Table 1, the testing input is x55 x76 x84 x99). We unroll until the first verbalizer is generated.

Please let us know if you have any other questions. We are more than happy to discuss with you.

We thank you for the positive feedback. With regard to the weakness:

1. While Section 3 is purely conceptual, we provide rigorous characterization in Section 5.1. We consider this as one of our main contributions, an instantiation for the assumptions made in previous theoretic works, such as [1] and [2]. For [1], our framework constituted with the assumptions we made in Section 5.1 satisfies the requirements of Corollary 4.2, therefore we can use their analysis to get an $O(1/T)$ regret bound ($T$ is the number of examples in the demonstration). For [2], please refer to our reply to reviewer rNJR.
2. Thanks for the suggestion! We can run more experiments. Could you let us know what experiments you think are important are missing?
3. We admit that we only focus on classification tasks in this work. However, we would like to stress that even for the classification setting, the mechanism of ICL is not yet clear. Future work may also extend our framework to generation tasks.

With regard to the questions:

1. By preprocessing the training data for LLMs, we think it’s possible. However, training an LLM requires lots of computational resources. We leave it for future work.
2. The formalism in Section 2 can handle arbitrarily difficult reasoning tasks as long as there is a logic system that can handle it and the induction searching process is computationally feasible, e.g. not NP-hard. We believe that most human-solvable tasks are in this class.
3. We think this is an important open question. A possible solution would be augmenting the training data, e.g. [3].
4. Our conjecture would be that for many commonsense rules, LLMs are trained with many instances for each of them. Because the training data may cover a large portion of common surface forms, even though it’s possible that LLMs are doing no more than pattern matching, they are still able to do “reasoning” following the commonsense rules.

[1] Zhang Y, Zhang F, Yang Z, et al. What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization[J]. arXiv preprint arXiv:2305.19420, 2023.

[2] Sanjeev A. and Anirudh G. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023.

[3] Zhou, Wangchunshu, Ronan Le Bras, and Yejin Choi. "Commonsense Knowledge Transfer for Pre-trained Language Models." arXiv preprint arXiv:2306.02388 (2023).