Self-Improvement in Language Models: The Sharpening Mechanism

We thank the reviewer for their careful attention to our work.

My main concern is the lack of implications of the theoretical results for practical use. While the theory covers many popular methods, such as BoN and DPO, there is not much connection to what this theory implies in practice. For example, what predictions does it make about BoN, can we choose “N” based on your theory?

The main role of our theory is to propose sharpening as a simple, yet plausible mechanism for self-improvement in the absence of external feedback; while we analyze specific algorithms, our focus is not to prescribe a specific way to perform sharpening, and we view the algorithms and analyses themselves as supporting results.

Nonetheless, in the rebuttal, we provide additional experiments that show that sharpening can indeed be amortized via SFT, which we hope resolves the disconnect between the presented theory and what is feasible in practice. Please refer to the discussion of additional experiments in the joint response to all reviewers above for more details.

What is the minimal experimental setup to cover both SFT and RLHF?

Please refer to the discussion of additional experiments in the joint response to all reviewers above. Briefly, our new experiments show that it is indeed possible to amortize sharpening via SFT on tasks for which we report improvements from inference-time sharpening.

How should we interpret Figure-1 based on your theory? Such as, would convergence rate of BoN be explained by your method?

We view Figure 1 primarily as a validation of the claim that maximum likelihood sharpening is a reasonable desideratum. Figure 1 demonstrates that if we are given sufficient inference-time compute, we can improve our performance on a number of benchmarks using inference-time BoN-Sharpening for a sufficiently large $N$. This is somewhat complementary to our main theoretical results in Section 4, which attempt to amortize inference time compute into training but do not say anything about the quality of the generations given sufficient inference-time compute (with BoN being a particular example of how one might use additional inference-time computation). In this regard, Figure 1 can be viewed as a sort of “skyline” on the performance improvement we may expect by running our proposed training-time sharpening algorithms.

Let us mention in passing that we do give a theoretical guarantee for inference-time BoN-Sharpening in Proposition B.1, which shows that this algorithm succeeds whenever $N$ is chosen to be sufficiently large as a function of the coverage $\pi_{base}(y^{*}(x) | x)$ for the base model. Our results in Figure 1 suggest that the tasks we consider do indeed exhibit favorable coverage.

(In the above, we refer to the old Figure 1; the updated figure includes many more models and datasets.)

While I understand that using the same context “x” for both policy and reward function is meaningful, in practice it is generally different. Can you discuss if an extension of your theory explains the sample complexity when using different contexts that are related by a function? In euclidean space, this could simply be defined as a bound on the distance between two context.

We would appreciate it if the reviewer can expand upon this question with a bit more context, as we are a little confused about what the question is asking. We can interpret the question in two distinct ways and address each below, but are happy to elaborate if we have misunderstood.

Interpretation #1: The reviewer is asking whether our theory covers the setting where there are possibly different prompts (contexts) at training time and inference time. If the training-time prompts and inference-time prompts are different, but come from the same distribution, then this is precisely what our theory covers. In greater detail, our theory considers the setting where prompts/contexts are drawn iid from some distribution and our performance is measured on new prompts drawn from the same distribution. An interesting direction for future research would be to provide guarantees for a more general setting in which the prompts/context at training time come from a different distribution than at test time; note however that this problem is not yet well understood even for simpler supervised learning settings, in spite of extensive research.

Interpretation #2: The reviewer is considering a scenario where the reward is obtained by prompting a language model (i.e., LLM-as-a-judge). Further, the prompt consists of a “problem statement” as well as “other text” providing instructions, etc, and where the “other text” for the policy and the reward function are different. This setting is precisely captured by our framework: we should view “x” as just the “problem statement” while the “other text” is abstracted into the policy and reward.

We thank the reviewer for their careful attention to our work.

While the authors empirically show there is useful signal in Best-of-N Sharpening (by sampling at inference), they do not empirically verify that SFT or RLHF training with this signal works.

Please refer to the discussion of additional experiments in the joint response to all reviewers above. Briefly, our new experiments show that it is indeed possible to amortize sharpening via SFT on tasks for which we report improvements from inference-time sharpening.

The authors focus their analysis on the maximum likelihood self-reward. While this objective is simple, it has not been empirically explored a lot in self-improvement literature (as the authors themselves note).

We do not view this as a weakness, but rather a contribution/novelty. In particular, the sharpening (maximum likelihood self-reward) framework permits strong theoretical guarantees, and our inference-time results show that empirically, it is a powerful self-improvement signal. Unlike reward-based methods, this self-reward comes “for free”, as it does not require external feedback or preference data. For future work, we hope to build on these theoretical techniques to analyze more complex self-reward heuristics such as self-judging, with the present work serving as a starting point.

The sentence in the abstract "Motivated by the observation that language models are often better at verifying response quality than they are at generating correct responses" is misleading in the context of this paper. Here, the reward is not assigned by asking the model to verify or critique its own solutions but by having access the log-probabilities of generated solutions.

We view the log-probabilities as a stylized self-reward function, which offers perhaps the simplest objective for self-improvement in the absence of external feedback, and which permits strong theoretical guarantees; other self-rewards derived from asking the model to verify or critique its own responses also fall into our general formulation in Eq. (1), but may not admit such strong guarantees in general (though this is an interesting topic to explore in future research). In particular, not all models, e.g., smaller or less general ones, are accurate at judging themselves, and this is where the simplicity of our framework is beneficial.

Recent evidence contradicting that LLMs can self-improve without external feedback

We are not claiming that LLMs can always self-improve; we identify specific conditions (boundedness of the coverage coefficient in Eq. (6), and expressivity of the model class (Assumption 4.1 or 4.3)) under which the sharpening methods we consider are guaranteed to do so in theory, but make no claims outside of the regime where these assumptions hold. That said, we agree it would be helpful to cite these works and include the above discussion of how they pertain to our results.

Can your work provide any guidelines for the practical deployment of sharpening techniques?

Yes, we show empirically that inference-time maximum-likelihood sharpening can lead to improvement across a variety of standard benchmarks and models; note that this improvement comes “for free”, in the sense that it only uses the base model, and does not require a reward model or external feedback. We also show that it is indeed possible to amortize this form of sharpening via SFT.

Please refer to the joint response to all reviewers above for further discussion.

Thank you for the responses to my questions. Additional experiments strengthen the contribution of this paper, so I am increasing my score.

While the paper does mention the prior literature on self-distillation, it would benefit from a more complete summary of this prior line of work, including key findings and making it very clear how this work is different/novel.

Thank you for the natural suggestion! Most theoretical work on self-distillation (c.f Appendix A) has focused on supervised learning settings (e.g., regression or binary classification) instead of language modeling. For these settings, we typically have explicit supervision and a clear choice of metric (e.g., MSE or classification accuracy) through which to evaluate the algorithm. This is somewhat complementary to the question we consider, which asks how to evaluate self-training algorithms for generative language modeling in the absence of such supervision. Whether some form of self-distillation can be analyzed using the sharpening objective we consider is a very natural question, and we would be curious to explore this in future work.

While empirical results are shown for inference-time sharpening, there are no experiments for self-training methods (SFT-sharpening, RLHF-sharpening) which are the main focus of the theoretical analysis. The paper would greatly benefit from including experimental results for SFT-sharpening and/or RLHF-sharpening, with a focus on whether the theoretical results can be used to correctly predict something about the empirical results. For example, can the theoretical results be used to predict the difference between running an RLHF-sharpening setup with and without online exploration? Or between SFT- and RLHF-sharpening? I would be curious to hear the authors' views on this. Please refer to the discussion of additional experiments in the joint response to all reviewers above. Briefly, we show empirically that it is indeed possible to amortize sharpening via SFT, as predicted by our theoretical results.

On the subject of whether the theory can predict more-detailed empirical phenomena like tradeoffs between algorithms, we agree that this is a good question. However, due to the scope and depth of an investigation into the empirical benefits and drawbacks, we leave this fascinating question open to further inquiry.

We thank the reviewer for their careful attention to our work.

What should experimentalists take away from this work? For example, researchers working on inference-time computation scaling, or those doing self-training. Can this work help them in some way?

Please refer to the discussion of additional experiments in the joint response to all reviewers above. Briefly, some key takeaways include:

• We show empirically that inference-time maximum-likelihood sharpening can lead to improvement across a variety of standard benchmarks and models; note that this improvement comes “for free”, in the sense that only uses the base model, and does not require a reward model or external feedback.
• We show that it is indeed possible to amortize this form of sharpening via SFT.

Do the authors see this work as being helpful for investigations into scaling laws for self-improvement? If so, how?

If sharpening in the sense of Eq. (2) is the desideratum, then our work suggests that one should observe favorable scaling laws for sharpening-based self-improvement, both with respect to inference-time computation (e.g., through inference-time Best-of-N Sharpening) or through training-time computation (SFT-Sharpening or RLHF-Sharpening). However, we acknowledge that this is somewhat speculative, as sharpening is almost certainly not the only objective one might care about for self-training, and may not always correlate with performance at the task of interest. We hope to investigate this more deeply in future work.

In addition, we have not deeply investigated the effect of model capacity/architecture etc on the ability to sharpen. In this direction, we noticed a concurrent paper, also in submission to ICLR, that considers scaling laws for self-improvement but does not use our sharpening setup (https://openreview.net/forum?id=mtJSMcF3ek). Developing analogous scaling laws for sharpening is a great direction for future work.

In the RLHF-sharpening formulation, my understanding is that the KL-term ensures that the finetuned policy stays close to the base-policy, which is required since self-reward is defined using the base-policy (i.e., to keep generations in-distribution for the reward function). Is this correct?

Yes, this is correct; note however that the regularization parameter \beta is chosen to be relatively small for our theoretical guarantees, which ensures that the fine-tuned policy can become sufficiently sharpened.

Furthermore, I am wondering whether the self-reward can be defined using the non-stationary policy that is being fine-tuned. What would be the effect of this change?

This is a very natural question! In fact, we can show that an iterative version of the RLHF-Sharpening algorithm which repeatedly updates the self-reward function based on policy being fine-tuned does indeed converge, and enjoys similar guarantees to Theorem 4.2. We do not currently know whether this approach enjoys provable benefits over Theorem 4.2 (one interesting difference is that it does permit a larger setting for the regularization parameter \beta discussed above), but we are very keen to explore this in future work.

Can you provide a more detailed explanation on the meaning of the coverage coefficient and how it can be problem dependent?

Briefly, the coverage coefficient captures the probability mass that the base policy \pi_base places on the optimal sequence-level response $\pi*$. If the coverage coefficient is large, it means that we are very unlikely to observe the sequence-level response by sampling y ~ pi_base, which means that algorithms like SFT-Sharpening and RLHF-Sharpening will be slow to converge. This quantity is problem-dependent in the sense that it depends on both $\pi_base$ itself and $y*(x)$.

On lines 187 and 199, the paper states that the corresponding algorithm scheme converges to a sharpened model / sharpening objective in the limit. However, it does not seem to provide a justification or proof for these two statements (please let me know if I have missed it).

This is proven in our main theoretical guarantees, Theorem 4.1 (SFT-Sharpening) and Theorem 4.2 (RLHF-Sharpening), provided the regularity assumptions (e.g., realizability for \Pi) required for the theorems to hold. We are happy to clarify this in the final version of the paper.

Although focusing on log-probability as self-reward is justified, the paper would benefit from including a survey of other self-reward functions used in prior self-improvement works. This context would provide valuable guidance for others to build on the sharpening perspective by looking at alternative self-rewards.

Thank you for the helpful suggestion! We are happy to include more extensive discussion of the specific self-rewards used in the prior work we cite in the final version of the paper. If you are aware of any references we missed, please let us know and we’d be happy to include them as well!

We thank the reviewer for their careful attention to our work and for catching the typo in line 194.

Line 256: It's appreciated and essential that the paragraph "Empirical validation of maximum-likelihood sharpening" verifies that sharpening provides downstream task improvement; however, in my view, this is a fairly known fact which could be presented in a more concise way, or acknowledged as a general fact.

We are not familiar with prior works that explicitly validate the utility of log-probabilities as a reward (rather than some external reward or more complex self-reward); an exception is that various decoding strategies like beam search (Meister et al., 2020) or chain-of-thought decoding (Wang & Zhou, 2024) admit an interpretation as sharpening w.r.t. log-probabilities, which we discuss in Appendix A. However, if the reviewer has specific references in mind we are more than happy to add them.

The results seem limited to the specific algorithms chosen IN RLHF-sharpening mainly REBEL and XPO.

We emphasize that our general framework is not restricted to these algorithms and we could likely extend our results to most post-training algorithms that attempt to optimize some notion of reward. While we choose several (of the most ubiquitous) algorithms for concreteness, guarantees similar to Theorem 4.2 can likely be proven for most common RLHF algorithms. Theorem 4.3 is tailored to XPO because it makes use of exploration, and to our knowledge, XPO is currently the most well-known/realistic RLHF algorithm that provides provable exploration. We also remark that Theorem 4.1 is about an SFT-style algorithm (SFT-Sharpening), not REBEL/XPO.

It can be misunderstood from the paper that self-improvement methods should exclusively be seen as sharpening methods, from the strong questions raised in the abstract and the introduction. I suggest rephrasing those as motivation to see self-improvement as sharpening.

Thank you for the suggestion, we will be sure to emphasize in the revision that sharpening is a type of self-improvement.

Proposition 3.1 introduces an important result, which is then not discussed. The transition to the next section is too abrupt.

Thank you for pointing this out; we will include a more extensive discussion in the revision. The main finding here is that for autoregressive models, Proposition 3.1 shows that one only needs to sharpen to constant precision $\delta$ if the arg-max sequence is unique. That is, it suffices to take $\delta < 1 / 2$ for greedy decoding on the sharpened model to return the arg-max sequence. This is an important takeaway, as the complexity guarantees for SFT-Sharpening and RLHF-Sharpening depend polynomially on $1/\delta$, so setting delta to be constant can improve sample complexity.

I thank the authors for the complementary experiments. Can they also please add references to the figures that show the results of these experiments in this general response?