Many-Shot In-Context Learning

Details about Reinforced ICL: (1) how many rationales , (2) selected per problem, and (3) what happens when no generated rationales, and is it detrimental?

• (1) Number of rationales generated was based on available problems for ICL. MATH has 7.5K problems and we generated one rationale per problem, resulting in correct rationales for about 3.5K problems (but we see plateauing with about 500-shots). For tasks with a much smaller number of inputs (250 for GPQA, 150 for BBH), we generated 10 rationales per problem at a temperature of 1, to maximize having at least one correct rationale per problem.

• (2) For each problem, we randomly pick one of the correct rationales for the K-shot prompts.

• (3) Inputs without any correct rationales are thrown away entirely, which is a limitation of Reinforced ICL to be unable to use such inputs. Moreover, as these inputs correspond to the harder problems which the model cannot currently solve, we might be throwing away valuable information. As shown in Figure A.17, doing another iteration of Reinforced ICL using only these “harder” inputs can further improve many-shot performance.

l.194 "to reduce impact of false positives" --- why is this important?

Typically, we can only evaluate final answer correctness and not verify the CoT rationale. As such, BBH tasks with binary choices can result in rationales that obtain the right answer “by chance” with wrong reasoning (“false positives”) – our manual inspection indicated that model-generated rationales on such tasks were of poor quality.

This is an inherent limitation of methods that rely on model-generated rationales, including Reinforced Self-Training method, which inspired Reinforced ICL. We discussed this limitation in L134-137, and will clarify it in the revision.

fig 6: reinforced ICL at K=250 missing

We generated rationales with correct answers for only 129 problems (this was mentioned in L180-181, will further clarify).

Unable to find prompts for reinforced or gt ICL for unsupervised ICL tasks

Figure A.9 shows the zero-shot GT prompt for GPQA and Figure A.11 shows the 4-shot GT prompt for MATH and GSM8K. Reinforced ICL prompts contain the same problems as GT prompts but use model-generated solutions – we’ll add an example to show the solution differences.

Sec 4.4: why GT ICL may be lower NLL than Reinforced ICL .. rejects some high-prob model generated responses .. raising NLL from filtering step?

Certainly, the filtering step in Reinforced ICL can result in solutions that the base model considers as high NLL. Another hypothesis is that model-generated solutions can look very different from human-written ones, resulting in higher NLL on GT solutions. Notably, Reinforced ICL has lower NLL than GT ICL on model-generated solutions on test problems.

sec 3.1 Fig 5: what K=4 corresponds to in ICL GT. Are these 4 at the end of the prompt counted in the K, and is this same for all three prompt methods?

The 4-shot GT ICL prompt uses only 4 examples, which correspond to the same examples as used by the 4-shot formatting preamble of Unsupervised ICL (Figure A.11). However, Unsupervised ICL prompt also used an instruction preamble (“You will be provided Problems similar to the ones below:”), leading to differences in results from 4-shot GT ICL Prompt.

Reinforced ICL uses the same problems but model-generated solutions instead of GT solutions, resulting in different performance than GT ICL, even in the 4-shot setting.

Machine translation could use unsupervised ICL with the English phrases .. it wouldn't help at all for MT .. and evaluating it just to verify this. Also, unsupervised ICL on xsum?

As expected, Unsupervised ICL on low-resource MT doesn’t help, as shown in Figure A.16. This was mentioned on L146-147 about limitations of unsupervised ICL.

We also ran Unsupervised ICL on XSum but only observed a maximum rouge-L score of about 23.95 (with 250-shot unsupervised prompt), which is slightly higher than just using the 1-shot prompt with an article and summary. Looking at the generated summaries, they were more verbose than the abstractive target summaries. This might be mitigated to some extent with a better zero-shot instruction, but would be unfair as no such instruction was used for many-shot ICL with ground-truth examples.

sec 4.1 Fig 8: the shape of these curves as a function of K is what one would expect for a model preinitialized with the original task labels, as a function of training step

Agreed, this is a nice connection and we’d mention it in text.

With more shots .. more likely to include noisy or incorrect data

If many-shot examples in the prompt contain biases (e.g., stereotypes, unfair representations), the model can possibly amplify these biases. Moreover, many-shot ICL can be used for overriding safety-training biases, manipulating LLMs to behave in unintended or harmful ways. We’ll include this in the discussion.

Thanks for the detailed comments and questions, which we address below and would strengthen our work.

In the rebuttal pdf, we have added results on runtime differences, many-shot performance of 1.5 Flash and frontier LLMs, ablated impact of new information vs context length, re-evaluated results on Logistics, and added analysis for hallucination on xsum. Our detailed response follows:

Only Gemini 1.5 was explored .. are there model differences that impact many-shot

Indeed, our work serves as an existence proof for the huge potential of many-shot ICL. Nevertheless, we provided preliminary results for GPT-4-Turbo and Claude-3-Opus in Figure A.2, indicating that different models benefit from many-shot ICL to varying degrees.

We also added Figure 1 in the rebuttal pdf to evaluate many-shot ICL performance for Gemini 1.5 Flash, a smaller LLM than 1.5 Pro, and show that it can match or surpass Claude-3-Opus and GPT-4-Turbo with enough shots, despite having worse few-shot performance. We’ll move this to the main paper.

Recently, follow-ups [1, 2, 3] have exhibited many-shot ICL with other open-weights and closed-source models on different tasks. Our work contributes to this growing body of evidence, and contributes several analyses of the phenomenon.

[1] Many-Shot ICL in Multimodal Foundation Models. Jiang et al, 2024.

[2] Many-Shot ICL for Molecular Inverse Design. Moayedpour et al, 2024.

[3] ICL with Long-Context Models: An In-Depth Exploration. Bertsch et al, 2024.

Runtime differences varying K to provide a fuller picture of many-shot ICL .. profiling computation increases

Great suggestion! To show runtime differences, we added Figure 2 in the rebuttal pdf showing per-single generation runtime, averaged across the test set and multiple seeds, for many-shot ICL on summarization (500-shot) and sequential parity prediction (8192-shot).

With KV caching enabled (default for long-context servers), runtime increases linearly with a large number of shots, as opposed to quadratic for self-attention: doubling the number of shots nearly doubles the runtime. However, for a small number of shots, runtime is nearly constant.

Explanation: When computing the next token, we still have to attend to the fixed many-shot prompt, even if KV is cached. When the number of generated tokens is much smaller than many-shot prompts, each new token is still linear, which explains our observed runtime for a large number of shots. We hypothesize that up to a token length of 32K, you can fit the entire KV cache into TPU HBM, which roughly means that you compute next tokens in O(1) memory load.

xsum weird dates after K=50 .. if somehow the model learned the task was to recover the webarchive page title

Our analysis suggests that this hypothesis is likely to be true! Specifically, we extracted the hallucinated years from XSum summaries and plotted their histogram density in Figure 3. Remarkably, more than 95% of these dates indeed lie within the range 2014-2017, suggesting that the model might indeed be retrieving additional information about webarchive last updated time.

Including more distinct examples increases information, but also context length .. separate these effects by extending context length by repeating same examples

We ran this experiment on low-resource MT by repeating 25 examples several times to create many-shot prompts with up to 1000 examples (shuffled ordering) and added the results in Figure 4 in the rebuttal pdf.

The performance with repeated examples stays nearly the same and significantly lags behind many-shot performance with distinct examples. On this task, the benefit of many-shot ICL mainly stems from adding new information as opposed to increasing context length.

Planning logistics: Is there a significant increase for many-shot?

To be certain, we re-evaluated many-shot on Logistics with the latest public version of Gemini 1.5 Pro, and added the result in Figure 5 in the rebuttal pdf. Many-shot accuracy improves uniformly for this version – interestingly, few-shot performance already starts quite high around 40%, and improves to 62.8% with 400-shot and plateaus to 63.8 with 800% shot.

Supervised fine tuning: base model, and how many epochs? And why were these choices made? Estimates on the computation resources?

We performed “full” fine-tuning (no adapters) on the same base model that was used for the many-shot ICL (Gemini 1.5 Pro). We performed 5 epochs of training, picking intermediate checkpoint with lowest validation loss (often from the first few epochs). These choices were made to ensure that we can obtain quite strong results for SFT. We’ll add these details in Sec 4.3.

Since Gemini 1.5 Pro is closed-source, we used Vertex API for SFT and cannot provide estimates of computation resources.

Supervised finetuning comparison only on machine translation task .. limited conclusion

Correct, our results only demonstrate that many-shot ICL can be competitive to fine-tuning on some tasks. The high dollar cost of “fine-tuning” limited our experimentation to 4 runs (2 tasks x 2 data sizes). We also performed comparison to SFT on parity prediction, where we find that many-shot ICL requires 20x less samples compared to fine-tuning GPT-2 to reach the same performance on this synthetic task (Appendix A.13). A more thorough comparison would be interesting for future work.

what error bars indicate .. say it is stdev

Yes, error bars indicate stdev of test performance across multiple random seeds, where K-shot prompts are sampled randomly for each seed (Lines 68-70). We’ll update the text to clarify this.

---

We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment.

We thank the reviewer for their detailed comments and questions. Our detailed response follows:

While authors include some results with GPT-4-Turbo and Claude-3-Opus, a more comprehensive comparison across models would strengthen the generalizability

While we do not fully address this limitation, in the rebuttal pdf, we’ve added results for Gemini 1.5 Flash in Figure 1, a smaller LLM than Gemini 1.5 Pro and. We find that even 1.5 Flash can match or surpass Claude and GPT with many-shot ICL, despite having worse few-shot performance. This demonstrates that even small LLMs with long-contexts might be capable of many-shot ICL.

While the paper provides extensive empirical results, it lacks a theoretical framework to explain why many-shot ICL works so well

Related to this, an ICML'24 paper [1] argues that ICL operates under two possible modes, task recognition and task learning – activating the task learning mode requires a large enough number of shots (however they only empirically studied up to 128-shots), which is highly task dependent. It is possible that the success of many-shot ICL is partially explained by activating this task learning mode. We’ll include this in discussion.

Theoretical Analysis for why performance degrades

A very recent submission under review [2] argues that performance drop in many-shot ICL might be due to more demonstrations diverting the model attention from the query, hindering its understanding of the query.

Another reason might be that really long many-shot prompts might be highly out-of-distribution (OOD) as LLMs might not have seen such prompts during pre-training or post-training (current mixtures might be optimized for needle-in-a-haystack tests). Furthermore, LLM pre-training has a maximum sequence length, followed by continued pre-training for context lengthening, for example, LLaMa 3.1 uses 8k and 128k while Apple FM uses 8k and 32k. We’ll update the discussion to include these hypotheses.

An explicit discussion of drawbacks and risks with many-shot ICL

We’ll update the discussion to include the following drawbacks and risks.

• Computational Cost: Many-shot ICL can be computationally expensive, especially with a large number of examples. This can be mitigated with context caching and KV caching.
• Bias Amplification: If the many-shot examples in the prompt contain biases (e.g., stereotypes, unfair representations), the model can amplify these biases, which raises ethical concerns.
• Lack of Transparency: The inner workings of how ICL works is not well understood. This makes it difficult to pinpoint exactly why a model generates a specific output with many-shot ICL. This can be problematic to assure safety and alignment of LLMs [3].
• Jailbreaking: Many-shot ICL can be used for overriding safety-training biases, manipulating LLMs to behave in unintended or harmful ways.

Is improving ethical alignment a potential application of many-shot?

We agree that inference time steering of LLMs with many-shot ICL has more flexibility compared to fine-tuning (e.g. different alignment criteria for different use cases), could also allow for faster adaptation to changing criteria, and fast "patching" for newly discovered legal or ethical issues. As such, many-shot ICL for ethical alignment seems to be a very promising application.

Since sometimes the performance can degrade with too many examples, and the ordering of examples can affect results, applying many-shots ICL to ethical alignment could be challenging.

In our work, sensitivity to example ordering seems to be highly task dependent – we do not see much impact on ordering on tasks like low-resource MT or summarization but larger impact on other tasks. Nevertheless, a simple approach might be to tune the ordering itself based on a held-out validation set and off-the-shelf libraries such as DSPy can be easily used for such purposes.

Regarding the optimal number of examples, this is likely a more critical parameter and would also matter for ethical alignment. As a rule of thumb, we swept across the number of shots on a logarithmic scale. Interestingly, our results on overriding pretraining biases show that performance only plateaued rather than degrading when adding too many shots.

---

[1] Dual Operating Modes of In-Context Learning. Lin and Lee, 2024.

[2] Focused Large Language Models are Stable Many-Shot Learners. Anonymous, 2024.

[3] Foundational Challenges in Assuring Alignment and Safety of LLMs. Anwar et al, 2024.

We thank the reviewer for their detailed comments, which we try to address below. We have added a many-shot ICL comparison for Gemini 1.5 Flash with frontier LLMs to understand the role of model size in the rebuttal pdf.

unsure of what role the model size plays here .. would a smaller LLM with a larger context also benefit from many-shot ICL? Gains from increase in the model size versus many-shot ICL

To understand the role of model size, we’ve evaluated the many-shot performance of Gemini 1.5 Flash, a smaller long-context LLM than Gemini 1.5 Pro. We used the low-resource MT to compare against LLMs at the larger-size of the spectrum, namely 1.5 Pro, Claude-3-Opus and GPT-4. Our results suggest that even smaller LLMs can benefit from many-shot ICL and outperform LLMs with stronger few-shot performance with enough shots. We report these results in Figure 1 in the rebuttal pdf.

On the English → Bemba task, we find that 1.5 Flash matches Claude-3-Opus and outperforms GPT-4 with 997-shots, despite having much worse few-shot performance than Claude and GPT. On English → Tamil MT, 1.5 Flash performs comparably to 1.5 Pro and Claude in terms of few-shot performance. However, 1.5 Flash outperforms Claude-3 in terms of many-shot performance, while lags behind 1.5 Pro.

Were the test splits in line with community standards? If there are any exceptions, these should be listed .. XSum/XLSum for only 150 test articles seems small. Were the articles randomly sampled?

Yes, for almost all of the tasks, we used the standard test sets used for evaluation, e.g., MATH500 and GPQA Diamond split). The only exceptions were summarization and low-resource MT, we randomly subsampled 150 examples from the entire test set to reduce the cost of evaluating many-shot ICL across multiple seeds. We'll update the text to specify this clearly.

Note that for XSum, we actually used the GEM-XSUM [1], which is a cleaner version of XSum with 1.2K test articles.

Do the authors have any intuition for why at times performance reduces as more exemplars are added?

A very recent submission under review [2] argues that performance drop in many-shot ICL might be due to more demonstrations diverting the model attention from the query, hindering its understanding of the query.

Another reason might be that really long many-shot prompts might be highly out-of-distribution (OOD) as LLMs might not have seen such prompts during pre-training or post-training (current mixtures might be optimized for needle-in-a-haystack tests). Furthermore, LLM pre-training has a maximum sequence length, followed by continued pre-training for context lengthening, for example, LLaMa 3.1 uses 8k and 128k while Apple FM uses 8k and 32k. We’ll update the discussion to include these hypotheses.

[1] The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. Gehrmann et. al, 2021.

[2] Focused Large Language Models are Stable Many-Shot Learners. Anonymous, 2024.

---

We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment

We thank the reviewer for their comments and questions, which we address below. We have added new results on 1.5 Flash and inference time of many-shot in the rebuttal pdf.

Experiments are mostly limited to 1.5 Pro .. different models have varying pre-training data and biases.

Indeed, our work serves as an existence proof for the huge potential of many-shot ICL. That said, we provided preliminary results for GPT-4-Turbo and Claude-3-Opus in Figure A.2, indicating that they can also benefit from many-shot ICL to varying degrees.

We added Figure 1 in rebuttal pdf to report many-shot performance for Gemini 1.5 Flash, a smaller LLM than 1.5 Pro, and show that it can match or surpass Claude-3-Opus and GPT-4-Turbo with enough shots, despite having worse few-shot performance. We’ll move this to the main paper.

Recently, follow-ups [1, 2, 3] have exhibited many-shot ICL with open-weights and closed-source models on different tasks. Our work contributes to this growing body of evidence, and contributes several analyses of the phenomenon.

ICL shows significant variation on GPQA (Fig 6) .. more seeds or better metrics

Our paper reports standard deviation across seeds on most of the tasks. We’d emphasize that GPQA is an exception, rather than the norm, even noted by the Anthropic report due to its extreme difficulty and small size (198 questions only). To show this variability, we directly report the individual performance on 5 seeds on both GPQA and MATH.

As the number of shots increases, so does tokens, making computation budget a bottleneck

Indeed, many-shot ICL does increase inference computation time, but can allow for quick prototyping and experimentation using just an inference API. That said, it can be sped-up with KV caching and context caching [7], which is default for long-context servers.

To empirically measure this inference time, we’ve added Figure 2 in the rebuttal pdf showing per-output generation runtime for many-shot ICL, averaged across test set and multiple seeds, on summarization (500-shot) and sequential parity prediction (8192-shot). With caching enabled, runtime increases linearly with a large number of shots, as opposed to quadratic for self-attention: doubling the number of shots nearly doubles the runtime. However, for a small number of shots (less than 32k tokens), we see that runtime is nearly constant.

It's surprising that Reinforced and Unsupervised ICL outperform ground truth. Do you have any explanations?

• Reinforced ICL: The Reinforced Self-Training work [4] showed that fine-tuning using model-generated rationales can be more effective than human-generated ground-truth outputs. We show that a similar finding holds true for many-shot ICL. Since model-generated outputs utilize only the skills / knowledge possessed by the LLM, it might make such outputs easier to learn from.

• Unsupervised ICL: This is harder to explain as unsupervised ICL does not always work well: it outperforms ground truth for MATH, but is substantially worse for low-resource MT (Appendix A.10). Our hypothesis is that it works well when the LLM already has all the required knowledge to solve a task. As such, ground-truth outputs might bias the model and the model prefers to utilize its underlying knowledge by just relying on inputs.

The performance shows significant variations when different random seeds are used to select in-context examples. Could you explain why this happens?

Impact of random seed for ICL example selection seems to be highly task dependent – on low-resource MT and summarization, it has a minimal effect, as can be noticed from small error bars that show standard deviation of mean performance across seeds. However, on other tasks, such as MATH and GPQA, it has a higher impact. Prior work has found the following factors impact few-shot performance when selecting examples:

• If chosen examples are semantically similar to test examples, it leads to improved performance [5]
• Increased diversity in chosen examples leads to improved performance [6]

When we select different example subsets by setting different random seeds, we perturb these factors, leading to differences in downstream performance. Our general trends hold even taking into consideration this variation in performance (shown by standard deviation bars).

Flipped and abstract labels .. achieve worst performance when K=8 / K=16. Is this universal across different model sizes?

We do not know if this is a universal result across different model sizes. That said, we also evaluated Gemini 1.5 Flash on flipped labels and observed similar accuracy trends: 56% for K=4, 34% at K=8, 40% for K=16, 76% for K=32, and 86.5% for K=64.

Did you supervise fine-tune the same model and report on the dataset?

Yes, we performed “full” fine-tuning on the same model (Gemini 1.5 Pro) on the same examples that were used for many-shot ICL. We performed 5 epochs of training, picking intermediate checkpoint with lowest validation loss to ensure strong SFT results.

Future work could focus on how to select better ICL examples that capture the task's nature while being concise

Agreed, would add to future work.

---

[1] Many-Shot In-Context Learning in Multimodal Foundation Models. Jiang et al, 2024.

[2] Many-Shot In-Context Learning for Molecular Inverse Design. Moayedpour et al, 2024.

[3] In-Context Learning with Long-Context Models: An In-Depth Exploration. Bertsch et al, 2024.

[4] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. Singh et al, 2024.

[5] What Makes Good In-Context Examples for GPT. Liu et al, 2021.

[6] Selective Annotation Makes LMs Better Few-Shot Learners. Su et al, 2022.

[7] Context caching. ai.google.dev/gemini-api/docs/caching

---

We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment.