Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

We thank the reviewer for their insightful questions. Please see responses to individual comments below. We hope to address your concerns and we look forward to a constructive discussion.

Is the RL update illustrated in Example 2.1 adopted in experiments?

In the revised submission, we added the RL update result in revised Fig. 4 (now highlighted in blue). The RL update is labeled as “7B-RL” with a star marker on the plot. We observe that RL update has very similar dynamics as rejection sampling. We appreciate the question and we believe the RL result will improve the completeness of our paper.

Does the High-fidelity model update condition mentioned in Section 2.1 examined / verified in experiments？What's the practical benefit for having this condition (principle)?

The high-fidelity model update condition can be examined implicitly in Section 5 on the left of figure 4 by comparing the gap and performance difference in each iteration. To make this examination more explicit, below we provide the Pearson correlation coefficient between the gap and actual performance different across all models and verification method in our GSM8K experiments:

We believe this analysis exactly captures the motivation of using gap to measure the self-improvement capabilities of LLMs - while gap and final performance always have strong positive correlation, it is lower in the initial iterations because confounders are introduced during early model update.

High-fidelity model update is important in the iterative self-improvement setting, because we need a distillation of the filtered dataset (which has higher accuracy than the original model) so that we will have an improved generator for the next iteration.

Will different decoding algorithms (e.g., search algorithms) for generation help LLMs to (self-)improve? And will the choices of different generation methods alter the conclusions drawn in this paper?

We believe using search algorithms such as MCTS can indeed improve as shown in the inference-time scaling papers [1,2] (although during search they rely on a stronger reward model which is absent in the self-improvement framework). However, it does not strictly fit into our framework as searching algorithms are using a different generation distribution other than the base model, and performing such experiments requires another order of magnitude of compute so we will leave it as future work. Previous works on scaling laws in downstream tasks also only samples from the models without any searching algorithms [3].

That said, in Appendix B.1, we record the results for Qwen-1.5 model family in GSM8K, with different sampling temperatures. We observe the similar results as long as the temperature is within a reasonable range. We hope this result can help demonstrate the robustness of our result under different sampling conditions.

I like the experiments with Sudoku (Section 4.4), as this is a setting with answer generation strictly harder than verification (NP-hard > P; assuming NP!=P). However, according to the experimental results, non-trivial gaps are observed only with models of > 70B parameters. I wonder if the authors could conduct additional experiments by sweeping the settings of the Sudoku problems: currently it's of 4x4 matrices; how about 3x3, or even 2x2 sized matrices, where the problems should be much easier? With these experiments, can we have the observation that LLMs with different sizes can self-improve?

Thank you for your appreciation of our sudoku result! We agree with the reviewer that trying an easier P vs. NP problem is of great interest. However, sudoku can only be of size of $n^2 \times n^2$, so a smaller sudoku puzzle is 1x1 and the problem becomes trivial. We remark that we also tried the canonical 9x9 sudoku but it is even challenging for capable closed-sourced models such as Sonnet 3.5 or ChatGPT 4(o). That said, we are excited to try other tasks in the future.

We thank the reviewer for their insightful feedback. Please see responses to individual questions below. We hope to address your concerns and we look forward to a constructive discussion.

My main concern is about the experiment setup. The main experiments are limited to one benchmark, GSM8k, in the math task. Despite GSM8k is a classical benchmark for mathematics, it is relatively easy for current cutting-edge LLMs and its benchmarking function may decrease due to potential data contamination. Additionally, the self-improvement performance as well as the findings may differ across different datasets or scenarios. Therefore, I think it would be beneficial for the authors to complement experiments on more datasets and scenarios.

Thank you for pointing this out. We want to first remark that we indeed examine self-improvement on information retrieval tasks in Section 4.3 and Sudoku in Section 4.4. We hope these results can provide more completeness to our overall message.

On the other hand, we are excited to share our new results on the more challenging MATH dataset [1], which consists of competition level math questions. We repeat the same experiments as in GSM8K on all model families, and an additional Qwen-2.5 model family that consists of 7 models. We observed similar results in both scaling experiments and iterative self-improvement experiments. We present the results in Appendix A in the revised submission.

The defined metric reveals the capability gap between generation and verification. Empirically the metric is useful in predicting potential self-improvement. But it would be more solid if the authors could analyze the correlations between the metric and the actual performance changes after iterations of self-improvement. Experimental evidences and analyses would provide more understanding of the metric and its potential use.

Thank you for the suggestion. Below we present the Pearson correlation coefficient between the gap and actual performance different across all models and verification method in our GSM8K experiments:

We believe this analysis exactly captures the motivation of using gap to capture the self-improvement capabilities of LLMs - while gap and final performance always have strong positive correlation, it is lower in the initial iterations because confounders are introduced during early model update. For example, as we discussed in the paper, models might be just better at following format instead of better at solving the problems, and thus the final performance gap might be higher than the gap in the smaller models. Similar observations are present in other concurrent papers [2], and we argue that directly measuring performance difference is not the ideal metric. We hope the results can better motivate that gap is the more rigorous metric.

The findings on "effective diversity of generations" are interesting. Have you conducted further experiments to clarify the underlying causes? For instance, could it be attributed to the training process itself? (Perhaps an experiment involving training on golden labels could provide insights.) Alternatively, could it stem from noise introduced by self-verification? (An experiment to filter out incorrect samples using golden labels or employing stronger evaluators might help determine if this mitigates the issue.)

Thank you for the suggestion. We conduct additional experiments as suggested by the reviewer, with iterative improvement on Qwen-1.5-7B model on GSM8K in the following two settings: 1) using gold labels and 2) use Qwen-1.5-72B model as the verifier. We observe that there is no decrease in effective diversity if we use the gold label as verifier, and the decrease in effective diversity still happens in the second setting. This suggests that one cause of this phenomenon is that the model concentrates on some wrong answers that the imperfect verifier labels as correct. We refer the reviewer to Appendix B.3 for the concrete results.

From the main results on scaling law (Figure 1), the scaling laws seem to be severely influenced by the verification methods. Why do some methods (like MC, To) not exhibit such scaling laws? And whether the other methods share similar findings (like CoT-B in the paper and some others)?

We believe one reason is that MC and To have higher variance in terms of gap across prompts as we show in section 6.1, which prevents a consistent trend due to the noise. For CoT-B, we present the results in Table 5 in the appendix, and we can see it has monotone scaling except the Qwen-1.5-14B model. For a more positive result, we observe a more consistent pattern in our new results under the MATH dataset (Appendix A), where CoT-B, CoT-S, and even MC demonstrate the scaling property.

We thank the reviewer for their insightful comments. Please see responses to individual questions below. We hope to address your concerns and we look forward to a constructive discussion.

While the definition of the gv-gap is pretty strict, the real world choice of the utility function making the practical measurements very noisy as authors pointed out in the text (which is great!). They have proposed some ways on improving the verification signal, however, that is a weak point in the entire framework that does not present itself when we just measure the model performance before and after self-improvement steps.

Thank you for acknowledging our remark on the noisiness of the verification procedure. We agree very much that the current approach for verification is not ideal, so we try to understand it in both theoretical and empirical ways. In theory, one can reduce the noise of the estimation of the gap by either sampling more verifications or by simply generating more candidate responses. If given a fixed compute budget, there will be a tradeoff between making more generations or making more verification, depending on the variance of each process. We think this is actually an interesting future direction with good practical implications.

To have a better understanding on the variance of the verification, we perform the following experiment (detailed in Appendix B.4): with the Qwen-1.5B-72B model on GSM8K, we sample 8 generations for each question, and for each generation we sample 32 verification (CoT-B), and measure the empirical variance of the 32 verifications for each generation. We observe that the variance of the verification is near 0 for most of the generations, which indicates that in practice a handful of samples of verifications might suffice for an accurate estimation of the gap. We hope this result will make the discussion we had on the noisiness of the verification more complete.

While I agree that studying base models makes things much easier to handle in paper-size experimental setup, the absence of any signal about the gap changes with instruct based model weakens the provided takeaways. For instance, recent work showed that smaller (8B) instruct models are self-improving very well while in this work there is an opposite signal from the base smaller models that are seemingly lacking reasoning capabilities even under few shot settings.

adding more about the limitations coming from only using the base models would make overall takeaways not weaker, but clearer w.r.t. other recent research in this area.

Thank you for the suggestion! We agree that analyzing instruct models is more aligned with the recent research in the area. As the reviewer mentions, finetuned models have more confounders that may not lead to a clean conclusion. Another reason that we were hesitant to perform the instructed model experiment is that some latest models such as llama 3.1 already have a self-improvement component in the finetune process so it is unclear if we will see any signal with such models and may confound various analyses. We will make sure to add more discussion in the paper.

Towards a better understanding, we conduct the following experiments on instruct models: Qwen-1.5-Chat on GSM8K and Qwen-2.5-Instruct on MATH. We observe that Qwen-1.5-Chat models do not have the scaling property. In fact, the accuracy does not even increase monotonically with respect to the model size (and our accuracy result matches the reported results in the technical report). Also, some instruct model has lower accuracy than the base model. On the other hand, Qwen-2.5-Instruct models indeed have the scaling property. In this sense, we think the new result provides completeness as the reviewer suggests but also motivates our focus on the base models as well, since the instruct models introduce more confounders.

overall I think this will be a very useful analysis work for future self-improvement research direction.

Thank you for acknowledging our contributions and we hope our work can benefit the community as you suggested.

We thank the reviewer for their careful review. Please see responses to individual questions below. We hope to address your comments and we look forward to a constructive discussion.

The paper lacks a clear definition of the Generation-Verification Gap, with no illustrative diagram and only complex, confusing formulas cluttered with symbols.

We appreciate your suggestions on improving the presentation of the core concepts. In the revised pdf, we provide an illustration of the generation-verification gap (definition 2.1) in Figure.7, on top of page 17. Figure.7 uses rejection sampling and CoT-Binary verification as an example. In this case, the generation-verification gap can be understood as the accuracy difference between the generator and the filtered dataset, where the filtering criteria is determined by the verifier model. We keep it in the appendix for the revised submission to minimally impact the current main text - we will make sure to reorganize and include the diagram in the main text in the final version. We hope the illustrative diagram can help with understanding our key concept and definitions.

The authors fail to clearly outline the overall methodological framework or how the experimental process operates.

Thank you for pointing this out. We hope our improvement on the presentation on the generation-verification gap in the last response can help clarify how the experiments operate. In addition, we add the following description in Section 3 (highlighted in blue): For each model $f$ and prompt $x$, we sample $128$ responses $y \sim f(x)$, and sample $1$ verification for each response, which defines the proxy utility score $\hat u_f(x,y)$. We mainly consider the rejection sampling setting (for completeness we investigate the RL setting in Section 5), and thus the weight function $w$ is the indicator function with either quantile or global threshold (c.r. Example 2.2). Then we calculate $\mathsf{gap}$ or $\mathsf{gap}_{\mathsf{rel}}$ according to Definitions 2.1 and 2.2, which is the accuracy difference between the filtered generations and the original generations.

The contributions to the community are relatively weak, as most of the conclusions are already known, such as the "verification gap scales monotonically with the model pre-training flops" and the "Saturation Limit and Cause of Saturation."

We assume by "verification gap scales monotonically with the model pre-training flops" the reviewer refers to the result in Fig. 1, the scaling property of the relative generation-verification gap. Please let us know if this is not the case. We agree that many previous self-improvement papers have shown that a bigger model may have larger improvement, but there are a few reasons why the previous result is not sufficient.

1. Our paper defines the metric of generation-verification gap and we propose to decouple the gap and model update, while previous papers only measure the performance difference before and after the update (which combines the gap and model update), introducing confounders in the results.
2. The previous papers only measured two models or one model family at best, but our paper studies across multiple families of models. This enables us to discover a trend in terms of pretraining flops which has real practical implications and can serve as a guideline for future training.
3. Previous papers measure the absolute performance difference. While the performance differences might be monotone if only two or three models are measured, our large-scale study shows that this is not true - absolute gap does not scale monotonically with model size (c.r., Fig. 9 and Fig. 12 in the appendix). Our paper introduces the definition of relative gap which instead enjoys the scaling property and to our best knowledge, the concept of relative gap is not present in the previous literature. That said, we greatly appreciate any pointers on the relevant works to better contextualize our result!

Regarding "Saturation Limit and Cause of Saturation", we agree that previous works have observed similar saturation phenomena in self-improvement [1,2], which we discussed in Section 5. The difference is, our study argues that one should disentangle the gap and the model update while studying iterative self-improvement. We highlight the discussion and add an additional remark in blue in the section paragraph of Section 5 (page 8). As for the “cause of saturation”, to our best knowledge we are not aware of any previous similar results at least in the context of self-improvement. However, we might have missed something as the field is rapidly growing and we are happy to add a discussion of any relevant work that the reviewer deems necessary.

We hope our explanation highlights the contribution of our results and we will make sure to add more discussion in the paper.

We thank the reviewer for their additional comments. We are glad to see that our rebuttal addresses all but one concern, which is the novelty regarding the "Saturation Limit and Cause of Saturation" takeaway.

Saturation Limit and Cause of Saturation

For the saturation limit result, as we have acknowledged in both orginal paper and in the previous rebuttal, similar phenomena have been observed in the preivous literature. We never intend to claim that we are the first to observe this phenomena, but we have argued in both the original paper and the rebuttal that previous studies have been using the wrong metric to study the saturation limit, and we propose to use the generation-verification gap to avoid confounders from the policy distrillation step. We refer the reviewer to the highlighted discussion in section 5, and we will make sure to make this point more clear by revising the takeaway in section 5.

Regarding the cause of saturation, we would like to repeat our argument in the rebuttal that, to our best knowledge we are not aware of any previous similar results on examining potential cause of saturation, at least in the context of self-improvement. However, we might have missed something as the field is rapidly growing and we are happy to add a discussion of any relevant work that the reviewer deems necessary. To relate to the common knowledge about "no new knowledge is introduced during self-improvement", such a heuristic explaination might be plausible, but it does not offer insights on how to solve the problem. Our result on the effective diversity, on the other hand, offers a rigirous metric and practical implications on mitigating the saturation issue. We believe progresses in our field are often made through rigirous scientific studies.

Computation

Regarding the comment on the computation, we remark that our task is rather different from the reviewer's example. 1) for each question, we need to generate 128 responses, not 1 response. 2) we need to generate verifications for each generation. 3) tasks like MMLU only considers the logits of 4 tokens, while GSM8K and MATH requires generating hundreds or thousands of tokens for each generation. Thus the computation cost of our experiment is orders of magnitudes larger than the examples provided by the reviewer.

Practical Guidance

As a final remark, we want to remind the reviewer our results in addition to the iterative self-improvement results, and their practical implications:

• the scaling result can help predict the self-improvement capability of bigger models and potentially lead to a way to find the compute optimal model with self-improvement.
• our results on multiple tasks suggest the possibility of self-improvement on different types of downstreaning tasks.
• the generalization of verification result suggests that the filtering threshold can transfer between models within the same task and verification mechanism.
• the emsemble result suggests that in practice one can use multiple verification together to enhance self-improvement.

We hope our response can clarify the reviewers concerns and change the reviewer's pessimistic perspective on the practical guidance our paper can provide, and we are happy to answer any additional questions.

Thanks for your response! The clarifications on Generation-Verification Gap diagrams, the details of the experimental procedures, and the additional resutl on the MATH dataset enrich the paper’s completeness. Most importantly, you clarified the concept of the relative discrepancy between the validator's verification accuracy and the generative model's inference accuracy—a point indeed rarely mentioned in previous studies.

The concern regarding whether "Saturation Limit and Cause of Saturation" represents a new insight stems from the “Takeaway on iterative self-improvement” in original text. In fact, LLMs lose their ability to self-improve after 2 to 3 iterations due to repeated learning of the same data (oversampling 2 to 3 times), which prevents new knowledge from being introduced, or because new data is overridden by the original training data. This is conclusion widely known among LLM researchers.

As a side note, the authors only used off-the-shelf offline LLMs for local inference. With the aid of vllm tools, even a 70B model takes less than an hour for most tasks in the lm-evaluation-harness, and only a few hours for the MMLU tasks. For a purely analytical paper, this isn't considered a high computational cost.

Finally, I am pessimistic that this work will offer substantial help or guidance on how existing LLMs can self-improve their performance. Therefore, I am maintaining my current opinion and score.

We posted a revision of the paper that includes more experiment results, and we hope these results can further improve the completeness of our paper:

• (Appendix A) For the result on MATH, we updated Figure 10 to include more rounds of self-improvement for the bigger models.
• (Appendix B.2) For the result on instruct models, we added the results on the Llama-2-Chat model family.
• (Appendix B.3) For the result on fixed verifier, we added one more round of finetuning and the result is updated in Figure 13.