From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation

We thank the reviewer for their insightful and positive comments! Please see below for our detailed response, and we hope the reviewer could consider increasing the rating if they feel their concerns have been sufficiently addressed.

The analysis is limited to reinforced ICL setting that only considers reasoning-heavy tasks. It's not clear what many-shot ICL would perform on simpler tasks that do not heavily relies on reasoning. Also, could BRIDGE anyway be helpful on simpler tasks?

We primarily focused on the reasoning-heavy tasks as they are broadly deemed to be a class of problems that is demanding even for the frontier models (e.g., the SoTA performance of BIRD dataset that we study in Table 4 is around 70% at the time of writing, even with complicated agentic designs and state-of-the-art LLMs). That the crucial intermediate outputs have to be model-generated has also made the problem challenging, but at the same time these facts also mean that 1) the potential room of improvement is ample and 2) any performance bottlenecks can simply be overcome by switching to a stronger model. In fact, while [1], the seminal work in modern many-shot ICL has covered a wide range of tasks, one of their primary contributions is to exactly address the same setup in this paper, hence the proposal of reinforced ICL and unsupervised ICL. Several recent works and popular prompt optimization frameworks [2-4] have also largely focused on this setup, and modern LLMs largely report performances on reasoning-heavy benchmarks (BBH and MATH datasets we considered are common choices) as a demonstration of their general caliber, and we believe the emphasis by these prominent prior works has underscored the importance of this class of problems, which motivated our focus in the paper.

On the other hand, we do agree that studying the utility of many-shot ICL for other problems can be an important direction. While we acknowledge that the present work has primarily focused on the reinforced ICL setting (Line 535 in conclusion), we have also included a study on extending BRIDGE to information-heavy tasks like low-resource machine translation (MT) in Appendix C.4 where we keep the overall BRIDGE pipeline of iterative “optimize” and “generate” intact but modify, where appropriate, the exact implementation of these steps. While the section is a preliminary study, we hope that it shows the potential promise of BRIDGE in an even wider range of applications, which will be a key next step following up to this work.

The experiments are only evaluated with Gemini. What would be the performance of proposed method on open-source methods like Llama-3.1?

We thank the reviewer for their suggestions and we agree that it is important to ensure the findings are not model-specific. As such, we have added results on Mistral large (mistral-large-2407), which is an open-sourced model as the reviewer suggested, and also Claude 3.5 Sonnet. We refer the reviewer to the App. C5 of the updated manuscript.

More importantly, for open-source model, would there exists a better way of importance attribution?

We agree that there should exist at least more efficient methods for importance attribution when we lift the restrictions on model access (although we have some clarifications regarding this part – please see our response below). For example, currently we assume black-box, textual output-only access which is highly restrictive but is also the most generally applicable and model-agnostic – we rely only on textual outputs which are provided by all models, instead of any model-specific outputs or features. Currently the importance is currently estimated via repeated sampling, but for a white-box model where we can access the internal parameters and/or gradients, importance can be more efficiently done with, for example, input gradient / saliency attribution or some variant of Shapley value attribution [5] (which only requires output log-probability rather than gradients). We agree that this can be an important next step for this work, but we believe those are standalone improvements separate from the contributions in this paper.

We have incorporated a discussion of this in the Conclusion of the updated manuscript.

The description of "optimzie" step is not clear (Line 261). The text in page 6 is generally unclear to me. For example, what does "TCM" stand for in algorithm 2 step 6? How does the problem become a "bi-objective" problem and how the optimize step is connected to "Bayes optmization"?

The “Optimize” step, as described in Line 265, is motivated by the findings in Sec 2: given that “using all examples as demonstrations is sub-optimal”, the natural question is how do we actually pick the demonstrations that is indeed optimal, and “optimize” step is designed to achieve this. Using the binary vector representation of example sets mentioned in Line 162 onward, the objective is to find some $\mathbf{e}^* \in [0, 1]^m$ to be used as demonstrations for the next “generate” step. Practically, we use a Bayesian optimization for this optimization problem – we opted for Bayesian optimization because it is sample-efficient and zeroth-order (i.e., no gradient information required), although it is also possible to use an alternative optimizer – for example, in App C.1 we also used random search (BRIDGE-RS) where instead of using Bayesian optimization (BO), we randomly sample m sets of examples and pick the best based on validation performance. Besides explicit iterative optimization, it is also possible to use retrieval based on embedding similarity or heuristics like diversity (App. C1) for this purpose; practically, however, we find using BO performed the best, in consistency with prior works like [4, 6] which also used some formulation of BO.

The “bi-objective” is a practical design choice, and means that instead of only optimizing for $\mathbf{e}^* \in \{0, 1\}^m$ that maximizes the validation performance, we also encourage some kind of sparsity as regularization. One straightforward approach is to simply treat sparsity as some penalty term, but the problem is that we do not know how much weight this term should be. The bi-objective formulation is to bypass this and instead of setting the penalty weight as a hyperparameter, we essentially aim to both 1) maximizes validation performance and 2) maximizes sparsity (as measured by the number of examples in the selected set) – it is “bi-objective” since there are two objectives here. This part has also been explained in Line 300 to 310, and we’ve modified this part of the paper to make it even clearer.

“TCM” is a typo and we thank the reviewer for pointing it out. It should be “TCH”, which, as explained in Line 316, is a canonical way to scalarize the two optimization objectives above. This has been corrected in the updated version of the manuscript.

References

[1] Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., ... & Larochelle, H. (2024). Many-shot in-context learning. NeurIPS.

[2] Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., ... & Potts, C. (2024). Dspy: Compiling declarative language model calls into self-improving pipelines. ICLR.

[3] Wan, X., Sun, R., Nakhost, H., & Arik, S. O. (2024). Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. NeurIPS.

[4] Opsahl-Ong, K., Ryan, M. J., Purtell, J., Broman, D., Potts, C., Zaharia, M., & Khattab, O. (2024). Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. arXiv preprint arXiv:2406.11695.

[5] Enouen, J., Nakhost, H., Ebrahimi, S., Arik, S. O., Liu, Y., & Pfister, T. (2023). TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents. arXiv preprint arXiv:2312.01279.

[6] Chen, L., Chen, J., Goldstein, T., Huang, H., & Zhou, T. (2023). Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082.

[7] Wan, X., Sun, R., Dai, H., Arik, S. O., & Pfister, T. (2023). Better zero-shot reasoning with self-adaptive prompting. ACL.

for base model where the reasoning ability is limited, would the proposed method also be helpful?

Similar to Reinforced ICL and other bootstrapping methods (e.g. [2,4,7]), BRIDGE by default initializes with zero-shot CoT to obtain the initial example sets, so the model indeed needs some reasoning ability to begin with (this is also true for the other methods mentioned above). However, even for a weaker model, as long as there is some in-context learning ability, we believe there are various straightforward ways to reduce the reliance on the models’ initial reasoning capabilities. For example, instead of starting with zero-shot CoT, we can initialize with few-shot CoT with a few examples either annotated by humans (e.g., this has been used for highly challenging tasks like BIRD in Table 4) or a stronger model (i.e., similar to distillation in DSPy).

Lack of the analysis on the computation cost. What's the computational cost of optimize and generate step of BRIDGE, respectively?

The cost of the “optimize” step depends on the budget allocated ($n_{\mathrm{eval}}$ in Line 5 of Algorithm 2), which is user-configurable. If we opt for iterative optimization (such as using Bayesian optimization in the main section of the paper, or random search in App. C.1), each “optimize” step thus entails $n_{\mathrm{eval}}$ LLM inferences on the validation set. As shown in the App. C1 (added materials during rebuttal), it is also possible to use non-iterative method based on retrieval or embedding diversity, in which case each “optimize” step entails a single round of LLM inferences on the validation set (or the train set, if we use the dataset for both training and validation). In general, as expected with any optimization method, there is a cost-performance tradeoff; running an iterative algorithm multiple times incurs a higher one-off cost but generally performs better – this has also been shown in related works like [3].

The “generate” step always involves a single round of LLM inferences on the train set where we simply use the optimized examples from the “optimize” step above as demonstrations and run inference again on the train set.

We agree with the reviewer that understanding the computational cost is an important aspect and we’ve added the discussions above to the paper (App. D).

The definition of importance score function is well-justified. In section 2 Line 153, the importance of each example is estimated by the averate performance of randomly sampled demonstrations on the validation set. The variance would be huge considering different choices of subsets and orders. Have the author performed analysis on the varaince of importance score?

We thank the reviewer for their feedback. While sampling-based importance approximation inevitably involves some kind of randomness and variance, one key point we’d like to note is that it is meant to be a motivating experiment proving the point that “It does not always take “many shots” to achieve many-shot performance” (caption of Fig 1) and that “many-shot performance can be driven by few high-performing examples” (Page 3) only – we do not have to perform this step every time we run BRIDGE. Instead, the key insight we’d like to show is that 1) there exists some way such that we can approach the many-shot results with much fewer examples, whereas on the other hand 2) there also exists another way such that even including many examples does not improve performance – these motivate the proposition that we should not simply scale up the number of examples and should instead pay attention to what examples we include if different selection strategies lead to drastically different outcomes. The importance estimation and ranking procedure is simply an exemplary procedure to practically construct both selection methods in one go (by traversing the examples in ascending and descending orders) in an interpretable manner. Furthermore, given the fact that this procedure leads to well-separated lines in Fig 1 which correspond to the two aforementioned criteria, we argue that the importance score has fulfilled its practical purpose and that its noise should at least not dominate over the signal (as otherwise we will not be able to obtain well-separated lines).

Importance attribution is not well justified. In section 2 Line 153, it's not clear how importance score is actually estimated and why it would make sense.

We compute the importance by estimating the input gradient of the target metric (e.g., validation accuracy) w.r.t. each example – input gradient attribution is a well-used technique for importance/saliency estimation. Due to page limit, we had to leave the exact details on how to approximate the importance score in Appendix A. We’d like to refer the reviewer to Appendix A for detailed explanations.

We thank the reviewer for their constructive feedback! To our understanding, the main concerns of the reviewer lies in 1) results on additional LLMs and 2) adding an infilling baseline – we address both in our response. We hope that the reviewer could consider increasing the score if they feel the concerns have been sufficiently addressed.

I would also ask for including similar analysis for other LLMs beside close-source ones like Gemini. At least on the subset of the considered tasks, to see whether the conclusions generalize to other models.

We thank the reviewer for their suggestions and we agree that it is important to ensure the findings are not model-specific. As such, we have added results on Mistral large (mistral-large-2407), which is an open-sourced model, and also Claude 3.5 Sonnet. Please see below for the aggregated results on BBH tasks and Appendix C.5 for detailed per-task breakdown.

Aggregated BBH results in 4 different LLMs

The important baseline might be missing to ensure the strength of the claims, please see Questions section.

Can you please provide the baseline when a model conditions on all available examples but with all examples using model generated reasoning? This baseline can decouple the importance of subselecting a subset of examples and influence of having model-generated reasoning, and strengthen your claims if, indeed, you need both rather than only model-generated reasoning. The recent works [1, 2] can be used to ensure that a model is constrained to generate reasoning given the input and the ground truth answer.

We thank the reviewer for their insightful suggestion. First, we’d like to note that we already provide some baselines that 1) use all (correctly predicted) examples and 2) use model-generated reasoning (“reinf ICL” uses all examples (input, intermediate output and targets) that the model predicted correctly whereas “All CoT” use all examples, regardless of the prediction correctness), although they differ from the reviewer’s suggestions (which we refer to as “infill” hereafter) that the final labels are not revealed to the model but are used for post-hoc filtering (for reinforced ICL). Having said that, we agree that the “infilling” can indeed be a sensible baseline and thank the reviewer for bringing it up. We have added the suggested baseline as “All - infill” in Tables 1 and 3 using the the prompt adapted from Appendix C of [1] the reviewer suggested, and we found that on Gemini 1.5 Pro, it performs roughly on par with Reinforced ICL (i.e., {input, intermediate output, target} of all correctly predicted train samples, but the targets are not revealed to the model). On Gemini 1.5 Flash, it performs slightly worse than Reinforced ICL but in both models they performed stronger than the “All - Direct” and “All - CoT” baselines, but BRIDGE outperformed all these baselines by a large margin. We hope the additional results address the reviewer’s concerns.

We thank the reviewer for their prompt response, and we will shortly respond to the comment fully, but before that, we'd like a quick clarification regarding the proxy experiment requested:

How is the requested experiment different from the "iterative reinf ICL" baseline?

For the benefit of the reviewer, we paste the description from the caption of Table 1 (lines 383-385) below (with emphasis added):

"Iterative Reinf.” refers to the iterative variant of reinforced many-shot ICL where we directly use all the generated correct examples from the previous round as demonstrations for the next round without the optimize step.

Note that the "optimize" step performs example selection. Iterative reinf ICL dispenses of this step and only performs expansion iteratively with the full train set from Reinf ICL. This baseline is present in all experiments we have performed (e.g., Table 1-3).

We thank the reviewer again for providing actionable items, and we look forward to their clarification.

We thank the reviewer for their time and patience while we run their requested experiments.

I acknowledge the wider applicability of BRIDGE, however to see if it's the only benefit of BRIDGE I requested the baseline. Thats why in my initial review I suggested using open-source models to allow running meaningful baseline allowing validating the efficacy of BRIDGE at combining refining rationales and sub selecting demonstrations.

We thank the reviewer for acknowledging the value of BRIDGE. Firstly, we would like to emphasize that "wider applicability" is an important advantage over tuning methods given that at least as of now, many SoTA models are proprietary (including those shown with strong many-shot abilities, e.g., Gemini & Claude) where there is no white-box access, and in our opinion the ability to run on these models is a crucial advantage. Having said that, we did add experiments in open-sourced models (Mistral large and Mistral Nemo (12B)) in the rebuttal.

Prompt consistency.

There was an initial misunderstanding regarding the meaning of the term, and we thank the reviewer for clarifying what they meant.

Additional baseline.

We highly appreciate the reviewer for clarifying the request and providing an actionable item and a critical baseline. We run experiments on gemini-1.5-pro-001 using the suggested setting (we termed these experiments "iterative reinf. (restricted)" and "BRIDGE (restricted)", which refer to Iterative Reinf ICL and BRIDGE with the train support set restricted to the subset of train set where the model predicted correctly initially. This can be practically done by filtering out the train samples where the model predicted incorrectly initially to ensure we only iterate and refine on the subset where the model predicted correctly after the initial round (i.e., the candidate set generated by running the vanilla Reinf ICL). We refer the reviewer to the results below (we also added a row showing Average rank to ensure the average is not unduly biased by outlier subtasks).

On a high level, this provides further evidence on the importance of selection: Iterative Reinf ICL (restricted) without the "optimize" step actually did not improve over standard Reinf ICL (avg acc = 79.6%). BRIDGE (restricted), however, still meaningfully improves with the subsequent optimize and generate steps, although the gain is less than the original BRIDGE which utilizes more examples via the larger train set support.

We hope the additional requested baseline should at least provide some evidence demonstrating the value of the proposed algorithm and we plan to will add the results to the paper. We thank the reviewer again for their valuable suggestions and we hope that in light of the updated results the reviewer could consider adjust their rating.

We'd like to thank the reviewer for their constructive feedback and engaging with us during the discussion period -- we believe that the reviewer's suggestions have strengthened our paper, and we are happy that our responses have addressed the reviewer's concerns.

We agree that the tuning-based baselines [1-2] suggested by the reviewer are highly relevant -- we will definitely incorporate the discussions of them alongside the new experimental results obtained in the final version of the paper when we get the chance to revise it again. We will endeavor to investigate these methods more in-depth both for a comparison and for possible combinations of tuning/training-based and inference-time-based methods, which, as we previously mentioned, an important future direction in our opinion.

We thank the reviewer for their positive and insightful comments. We’d like to refer the reviewer both to the overall response and the detailed, point-by-point reply below, which we believe has thoroughly addressed their concerns. We hope that the reviewer could consider increasing the rating in light of our response.

The experiments are done only with Gemini models on text tasks. It would be interesting to see the performance on GPT-4o / o1 models and multimodal tasks.

We thank the reviewer for their suggestions and we agree that it is important to ensure the findings are not model-specific. As such, we have added results on Mistral large (mistral-large-2407) and Claude 3.5 Sonnet. Please see below for the aggregated results on BBH tasks and Appendix C.5 for detailed per-task breakdown.

Aggregated BBH results in 4 different LLMs

In the future, we will endeavor to include experimental results on more models and more tasks (such as the GPT-4 family of models and/or multimodal tasks as the reviewer suggested). We hope the reviewer could sympathize with us that running more models and multimodal tasks require significant computational resources which is not feasible during the rebuttal period, that the existing results (along with the new results added in the rebuttal), which we believe are already more expansive than many existing and contemporary works, are already sufficient in establishing the value of this work.

Variance is not reported for MATH, GSM8K, and BIRD. Given the relatively small improvement on these datasets, it's unclear if the gain is significant.

We’d like to refer the reviewer to the updated manuscript where we updated the results with standard deviation for MATH and GSM-Hard. On BIRD, the finetuning baseline requires more expensive resources for repeated experiments beyond the time and computational constraints during the rebuttal period. However, to show that the improvement is unlikely to be a case of random fluctuations, we note that the CHASE dev set can be divided into 11 distinct databases with very different accuracy levels, and we can compare ”CHASE+BRIDGE, Round 2” (62.0%) and “CHASE PROMPT” (60.1%) head-to-head in each of the databases – we can see that BRIDGE performs better or equal in 9/11 databases (Wilcoxon signed-rank test p-value = 0.066).

We thank the reviewer for their detailed feedback, and we’d like to refer them to both our revised manuscript which contains details of additional experiments requested and our detailed, point-by-point response below. We hope that the reviewer could consider increasing the rating if they feel their concerns have been sufficiently addressed.

The validation of the findings appears overly simplistic. For instance, as model capabilities scale, the order and number of demonstration examples may have diminishing importance in affecting in-context learning (ICL) performance. The authors should reconsider their conclusions by re-evaluating their results on less advanced LLMs, providing a more comprehensive understanding of the interplay between long-context utilization and demonstration selection. Without this analysis, the conclusion may be over-claimed to specific models.

We thank the reviewer for their comment. While we acknowledge the value of improving performance on less advanced LLMs, in this work we mainly experimented on the SoTA models as we are more interested in pushing the frontier of model capabilities (i.e., focusing on the issues where there headroom for improvement is ample even for the most advanced models) rather than focusing on problems that can be easily addressed by switching to a stronger model – specifically to the reviewer’s comment, we believe that while SoTA models are expected to be considerably more robust to the ordering of the examples (this can be seen by the relatively small standard deviation in Tables 1 & 2), selecting and iteratively improving demonstrations still have a significant performance impact judging from the margin of improvements even on the SoTA models. Furthermore, Many-shot ICL, especially on more challenging task categories (e.g., reasoning) typically demands stronger models, and seminal works [1-2] have also focused on frontier models like Gemini 1.5, Claude 3.5 and models of the GPT-4 family.

We do value the reviewer’s suggestion on ensuring that the findings are not model-specific, and as such we have added experiments on Mistral (mistral-large-2407) and Claude (claude-3.5-sonnet) models. Please see below for the aggregated results on BBH tasks and Appendix C.5 for detailed per-task breakdown.

Aggregated BBH results in 4 different LLMs

Lack of baselines. In the experiments, the authors did not compare their approach with existing learning-based strategies and learn-free ones for demonstration selection.

We appreciate the reviewer’s comment and following their suggestions, we have added two baselines based on diversity and retrieval based on embedding similarity in App. C1 of the updated draft. In a nutshell, if we simply would like to compare the demonstration selection component of BRIDGE outperforms the heuristic selection with the best tested hyperparameter (see table below for a summary). Since we BRIDGE is not only about demo selection, we can also combine these heuristics with the BRIDGE pipeline orthogonally, and the best performance achieved is 84.14% (Table 8), but before we elaborate on the new results, there are a few clarifications we’d like to make.

Average test accuracy on BBH using gemini-1.5-pro-001

First, the setup of BRIDGE (i.e., many-shot) significantly differs from the typical premises of demonstration selection (i.e., few-shot). Demonstration selection has often been assumed to be indispensable because the context window has been historically a limiting factor where it was impossible or extremely expensive to scale the number of demonstrations; even when such scaling was possible, prior works on earlier models even on simpler, discriminative tasks have shown mixed results [3], and these facts have necessitated some sort of selection. However, as mentioned in the introduction, frontier models are much less subjected to these issues. For example, with context limit well into the millions, previous works showed that Gemini is both compatible with and benefits from more examples, which suggests that demonstration selection is no longer a necessity – this is reflected in the fact that several prior works in many-shot ICL [1-2] do not employ any advanced selection techniques beyond random sampling. Unlike these previous observations, Our first insight precisely shows that demonstration selection is still beneficial even if it is no longer necessary (Page 3) and only removes information from the context (comparing to using all demonstrations, demonstration selection picks a subset) – we believe this is already a significant contribution in the many-shot setup and is contrary to the intuition that “more is better”.

Second, BRIDGE is not just about demonstration selection: while we proposed a Bayesian optimization framework for efficient demonstration selection, it is only one component of the BRIDGE pipeline – the “generate” step and the iterative procedure, both of which are absent in typical demonstration selection algorithm, are crucial in ensuring we can still take advantage of the long-context capabilities of frontier models (a selection-only algorithm would reduce the problem back to few-shot and won’t be able to take advantage of the long context window available) and continually improve performance, as we’ve shown in Tables 1-3. In fact, the baselines mentioned by the reviewer (e.g., diversity and similarity) are orthogonal to the BRIDGE framework by swapping the “optimize” component to these methods, which we will describe below.

With these points in mind, we’d like to refer the reviewer to the added results in the App. C.1 where we replace the demonstration selection component with retrieval based on cosine similarity (similar to [4]) and diversity (similar to [5]) – we can observe that these variants underperform the version proposed in the paper: one key reason is that as we’ve shown in Sec 2, not all samples, even those with correct reasoning, are necessarily useful as demonstrations. Whereas concretely evaluating on the validation set would allow us to distinguish and prune out unuseful demos, heuristics like similarity and diversity cannot distinguish that since the selection is purely based on similarity / embedding diversity but is not linked to validation performance. Another learning/cross-validation-based baseline which uses random search that is similar to the selection procedure in the popular DSPy [6] are already included in Table 5 of the appendix. It can be seen that BRIDGE-BO outperformed all these learning-based & learning-free variants. Finally, while it is certainly possible to use an alternative demonstration selection method beyond what we experimented on, as we mentioned above, any such method is orthogonal to the overall BRIDGE framework, and a stronger selection method will only enhance, not diminish, the overall performance of BRIDGE.

The writing of the paper requires further refinement. Sections 3 and 4 are challenging to read and follow, despite the method itself being relatively straightforward. For instance, the description of the different variants could be more appropriately placed in the Experimental Setting section for better clarity.

We thank the reviewer for their suggestions and we have added some additional descriptions to “Experimental Setting” to clarify this further, as suggested. However, we note that there are no “different variants” of the algorithm; Tables 1 and 3 show the progression of one single run of BRIDGE at different points in time as we mentioned in the caption of Table 1. We also used different color codings for the “optimize” and “generate” step to make it more clear, and made miscellaneous edits in Sec 3 and 4 to improve clarity.

Q1 In reference to Figure 1, does bottom_k refer to selecting examples that are less similar to the test case? Why does bottom_k achieve performance comparable to top_k as the number of demonstrations increases? Additionally, why does bottom_k outperform the all-examples approach?

Top/bottom-K does not refer to similarity to test cases (we indeed added a baseline based on embedding similarity between demos and test cases following the reviewer’s suggestion, but that is different and distinct from Fig 1). The meaning of top_k and bottom_k is explained in the paragraph beginning on Page 4. Specifically, we ran an attribution method to learn the importance of each example. Then, we sorted the examples based on their imputed importance score; top-k means if we select the top-k examples based on their importance score, whereas bottom-k means if we select the bottom-k examples based on the score. Top-k and bottom-k are bound to converge when we set k -> N (where N is the total number of examples) because when k=N, top-k and bottom-k refer to the identical set of examples. We’ve updated the caption of Fig 1 to make it even clearer.

Regarding the second part of the question, there are two aspects that we’d like to clarify: firstly, there is no guarantee that using all examples will lead to optimal performance, which is one of our key findings and, in fact, is a key motivation for the “optimize” step. A key argument in the paper is that even for long-context models where providing many examples in the context is possible, it is often suboptimal, and a judicious selection of examples can often outperform all examples. Secondly, we find “all examples,” which we assume is what the reviewer refers to, only using {input, output} pairs in the context is a rather weak baseline (e.g., ~10% worse from Table 1) as it does not contain the intermediate outputs, i.e., no reasoning (the same baseline is also included as the first column (“All - Direct”) in Table 1 and 3, and the fact that even bottom-K outperforms it suggests that incorporating intermediate, model-generated rationales is key for strong performance. We also provide baselines utilizing examples with reasoning (“All - CoT”, “All - Infill” and “Reinf. ICL”) in Tables 1 and 3; they are noticeably stronger than “All -Direct”, but BRIDGE still outperforms them.

References:

[1] Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., ... & Larochelle, H. (2024). Many-shot in-context learning. NeurIPS.

[2] Y. Jiang, J. Irvin, J. H. Wang, M. A. Chaudhry, J. H. Chen, and A. Y. Ng (2024), “Many-Shot In-Context Learning in Multimodal Foundation Models,” arXiv preprint: arXiv:2405.09798.

[3] Hao, Y., Sun, Y., Dong, L., Han, Z., Gu, Y., & Wei, F. (2022). Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713.

[4] Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021). What Makes Good In-Context Examples for GPT-$3$?. arXiv preprint arXiv:2101.06804.

[5] Zhang, Z., Zhang, A., Li, M., & Smola, A. (2023). Automatic chain of thought prompting in large language models. ICLR.

[6] Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., ... & Potts, C. (2024). Dspy: Compiling declarative language model calls into self-improving pipelines. ICLR.

As we mentioned originally in the reply, we did not say “SoTA models matter more” — we said improving smaller models is valuable, but we focused on SoTA models in this paper (although we do add a new set of experiments on mistral-nemo (12B), which is a small model by modern standards). We believe it is a consensus from the community that both are important.

Second, we are unsure why the desire to “not fully finetune the model” leads to the conclusion that “many-shot is more practical to less powerful models.” On the contrary, besides the reasons we stated above (e.g., the benefits in scaling from many-shot are often more obvious in long-context SoTA models), finetuning large frontier models can also be extremely costly or impossible for end-users (proprietary models), which means many-shot can be a more desirable alternative than in less powerful models where finetuning is easier and cheaper. In fact, the paper the reviewer cited itself [Agarwal, NeurIP’24] only considers the SoTA models (mostly Gemini 1.5, some Claude 3.5 and GPT-4-turbo). We’d be grateful if the reviewer could clarify further.

We thank the reviewer again for their time and effort. Complementary to our previous comment, while the work primarily focused on SoTA models (we provided a justification why in previous comments), we value the reviewer's suggestion on the applicability of BRIDGE on weaker and/or smaller models. We add a new experiment where we run BRIDGE on Mistral-Nemo 12B, along with Mistral-Large results we've already added in the updated paper. We refer the reviewer to the results below (and we'll add the details to the paper as well). We chose mistral-nemo because 1) it is a small and accessible model, yet has state-of-the-art performance amongst models of similar sizes; and 2) it supports a reasonably long context (128k), which makes it suitable in our setup.

Overall, we find that BRIDGE continues to lead to large improvement margins (+10% on average compared to the base Reinf ICL), although similar to mistral-large, the model seems to benefit less from more examples compared to proprietary models with strong many-shot abilities (e.g., Gemini and Claude), but this is more related to how the models are trained to take advantage of the examples. Nonetheless, the generate step leads to better candidates that act as a pool for the subsequent "optimize" step, which is highly effective in this case. We believe this convincingly show that BRIDGE can still be highly effective in the context of smaller models / models that are weaker in many-shot ICL, which we believe was the crux of the reviewer concern -- we hope the reviewer could consider adjusting their score if they feel their concerns have been addressed adequately.

thanks very much for the addtional results by using Mistral-Nemo (12B). Most of my concerns have been addressed, I am happy to increase the score, but I still expect to see the performance of SLM (e.g., 7B) (maybe with some discussion?), even the results are not desired.

Dear Reviewer cswN,

We have provided additional experiments and explanations to address your valuable feedback.

Could you please kindly take a look, and let us know if you have further concerns so that we can provide further clarifications if needed?

Thank you so much!

Best,

Authors

We thank the reviewer for engaging with the discussions and we are glad that most of the concerns have been addressed and they are happy with increasing the score. We also greatly value the reviewer's suggestions on SLMs and will definitely incorporate some discussion (such as the one towards the end of our last reply) into the paper when we get the chance to revise it again.

In response to the request of running experiments on even smaller models, we have conducted a preliminary set of experiments a 7B model (specifically, Mistral-7B-Instruct-v0.3):

We hope the reviewer could sympathize with us that given the time constraint (the last chance to reply is by AoE time today (12/3); we were only aware of the reviewer's request a few hours ago since OpenReview does not actively notify participants of edits to existing comments), it was infeasible to complete a full set of experiments on all tasks on time and we have to use a smaller search budget for the optimize step (we used $n_{\mathrm{eval}} = 8$ in the table above), although we endeavor to have results that are compatible with the rest of the paper by the camera-ready deadline. Having said that, we believe that the preliminary results still show promise of the proposed method. Additionally, we believe that the since the 12B model should behave more similarly to typical SLMs than to the SoTA models, the results in our previous response should also at least imply some evidence on the effectiveness of our method on even smaller models.