Understanding In-Context Learning from Repetitions

The paper examines the phenomenon of token co-occurrence reinforcement, whereby tokens occurring in the context have a higher chance of being predicted by an LM. An interesting consequence is that In-Context Learning (ICL) can go wrong when a label that is different from the target label is repeated often in the in-context examples, potentially helping explain some failure modes of ICL.

It first shows that self-reinforcement occurs with randomly constructed sequences of words. The longer the repeated part, the stronger the reinforcement effect (Figure 3). It then shows that even discontinuous repeated subsequences give rise to the reinforcement effect. Section 4 studies the effect of the reinforcement effect on ICL. The effect can help constrain the output space and the desired pattern, but it can also lead to incorrect results.

The authors delve into an investigation of the impact of token repetition on in-context learning (ICL) performance, revealing both beneficial and detrimental effects through empirical analysis. On the positive side, repetition aids in narrowing down the output space, fostering consistency in subsequent predictions. However, the downside is evident when non-informative or incorrect messages are repeated in the demonstrations, resulting in diminished prediction accuracy.

This paper studies the emergent in-context learning (ICL) ability of LLM. The authors try to probe the ICL performance through surface repetitions, and establish a theory called token co-occurrence reinforcement, which explains the reasons for possible failures.

This paper quantitatively investigates in-context learning in terms of surface patterns. It shows that there is an inherent correlation between surface patterns, self-reinforcement, iterative generation and their important role in text generation. In particular, it shows the role of surface surface patterns in text generation and the existence of token co-occurrence reinforcement that strengthens the relationship between two tokens based on their contextual co-occurrences. The experiments on MMLU and GSM8K show that the reinforcement helps to constrain the output space and format the output according to a demonstration such as ‘Let’s think step by step’ .

To all reviewers,

We thank all the reviewers for taking the time to review our paper and their valuable comments. We are grateful that reviewers appreciate the effort of our work in analyzing the In-Context Learning of LLMs from a new perspective with comments such as Reviewer d42T "novel framework", Reviewer iqcg "meaningful explanations, comprehensive and convincing", Reviewer Ydk8 "systematic study, interesting results”.

We notice that a common concern raised by reviewers is the cause and inner-working responsible for the token reinforcement.

We have delved into the reasons underlying token reinforcement and conducted three important experiments to enhance our paper.

As for the reasons, we believe that token reinforcement could be a result of the model's effort to maximize the likelihood of the training data. Token reinforcement is hidden in the nature of pre-training corpus. In light of this, we conduct experiments to validate our hypothesis.

Given a commonly used large pre-training corpus (i.e., wikipedia-english(https://huggingface.co/datasets/olm/olm-wikipedia-20221220), 5.7B tokens), we compute the following statistical probabilities:

Exp_w P(w occurs | w is never observed in the context)

Exp_w P(w occurs | w has been observed before in the context for 1 time)

Exp_w P(w occurs | w has been observed before in the context for 2 times)

…

where w is any word, and context is the autoregressive context window of pretraining data, usually 1024 tokens.

Conceptually, these probabilities reveal whether a particular word $w$ is likely to recur if there have already been $n$ instances of $w$ observed in the context, resonating with our analysis of token reinforcement.

The statistical probabilities are shown below (n=[0,9]):

The probabilities of a specific word occurring with an increasing number of instances perfectly align with the token reinforcement on LLMs.

The LLM manages to learn this inherent feature from the training data and generalize it to longer phrases and distant connections. This also elucidates the scaling with reinforcement, where larger models more effectively capture this feature from the training corpus.

To further understand the existence and inner workings of token reinforcement, we provide more experiments from two perspectives. The first one shows from learning process that leveraging repeating features are effective in the maximizing likelihood. We pretrain three conditional LMs from scratch and compare the converged loss of three strategies: (1) full context (2) random mask (3) mask repetitive features. Our results show that (3) mask repetitive features achieves the worst converged loss, which validates our hypothesis. The second experiment analyzes the trained LLMs and shows that the attention mechanism of attending preceding/adjacent token is responsible for the inner-workings of token reinforcement. After masking out the adjacent token, the strength of reinforcement is reduced by 10 times.

For more details and experimental settings, we refer all reviewers to Appendix C. We highlight the revised parts of the paper with blue.

Finally, our main contributions are:

1. We propose a novel perspective to understand icl with repetitive text generations, where the key effect causing repetitions appears to be an important feature in ICL.
2. We empirically establish the existence of token reinforcement across various LLMs, alongside with the inevitable reason behind learning process and inner workings responsible for it.
3. We show that token reinforcement plays crucial role in ICL for both beneficial and detrimental effects. It constrains output space and desired patterns, but it is also responsible for spurious connections and possible failure of ICL. Our findings provide a novel research basis for analyze the mechanism of in-context learning.

Based on our results, instead of the traditional understanding of input-label mappings, we view ICL as a combination of token reinforcements, even though some of them only appear once and do not get reinforced much.

Due to the time limit in the discussion period, we are committed to incorporating these new results about the reasons and inner workings of token reinforcement into the main content, as well as the highlighting our contributions and implications in the next version.

We hope we have addressed all your concerns and would greatly appreciate it if you could re-consider your score about our paper based on our responses.

This paper investigates an interesting aspect of the in-context learning ability of LLMs. They show that LLM behaviors are sensitive to the repetitive demonstrations in the context. Built upon this observation, they take a close analysis on the beneficial and detrimental effects of reptition through empirical analysis. On the positive sides, repetition helps constrain the output space and enhance consistency; on the downside, repeating wrong information can be very harmful and leads to incorrect answers. Extensive experiments are carried out to justify these claims.

The reviewers generally find the problem interesting and delivers new insights to the community. During rebuttal, some concerns rise on the lack of theoretical explanations of the discovered phenomena, and the authors provide some insights with new experiments. Another reviewer point a limitation of novelty since repetition is only a special case of distribution shift, which does make sense since repetition brings change to the distribution by upweighting some examples.

Summarizing these opinions, I find the discovered phenomena reveal certain risks of LLMs and the authors have put great efforts to empirically understood their behaviors and influence on LLM outputs. Given the timely interest in LLMs in the field, I would recommend accepting this paper to facilitate better understanding. I highly encourage the authors to incorporate the new results in the final version.

Accept (poster)