Thank you for your feedback! We will address each of your points below.

W1: Practicality of True Reward Oracle Requirement

You've highlighted a key practical challenge: the high cost of true reward data. HedgeTune is designed for this exact scenario. Instead of a costly brute-force search, it leverages the unimodality guaranteed by Theorem 1 to apply an efficient root-finding algorithm. This finds the optimal parameter with significantly fewer expensive validation samples. Our sample complexity analysis, detailed in our response to Reviewer pDYx, provides a formal treatment of this efficiency. We will add a discussion on the practical limitation to the conclusion.

Q1: Working with noisy, approximated, or preference-based true rewards

This is an excellent question about future directions, which we will add to our conclusion.

Noisy/Approximated Rewards: We hypothesize that the true reward's unimodal structure acts as a natural regularizer, helping HedgeTune find the right peak. However, more work is needed for a formal theoretical treatment.

Preference-Based Rewards: This is the most exciting and challenging future direction. One promising method would be to design a surrogate objective based on preference data akin to DPO. A central theoretical question would be to prove that this new objective also exhibits a unimodal or single-crossing property. We believe this is a rich area for future work. We will add a detailed discussion of these potential extensions to our conclusion.

W2/Q2: Proxy Uniformity Assumption

Thank you for these points. The proxy reward scores do not need to be uniform themselves.

We first map the proxy rewards to a standardized discrete space by transforming them into their quantiles using their own empirical CDF $F_p$. This process, defined by the relation $u = F_p(r_p)$ ensures that the new reward $u$ is uniform on $[0, 1]$ by construction. This transformation preserves the rank-ordering of the scores: an output with a higher proxy reward will also have a higher transformed score. Of course -- the resulting random variable will not be a continuous uniform random variable (which we assume), and instead will be a discrete uniform random variable. Recent work [1] has shown for a dense enough set of examples, the error incurred by this continuous model is negligible in the context of LLMs. We recognize this point was not sufficiently clear in the manuscript, and we will revise it to precisely explain our construction.

Nevertheless, your question correctly highlights if this approximation step could be entirely removed. Theorem 1 can in fact apply directly for the case where rewards have discrete support with small changes to the current proof (swapping integrals with sums). This is particularly exciting, as it confirms that reward hacking follows a predictable behavior and this is a fundamental property of these inference methods.

Instead of the continuous reward $u \in [0, 1]$, we consider a discrete set of $M$ possible outputs, ranked from worst to best according to their proxy reward scores. Let the true reward for the output at rank $k$ be $w_k$. The probability matrix $P(\theta, k) = p_k(\theta)$ gives the probability of selecting the output at rank $k$ given an inference-time parameter $\theta$. The expected true reward is now expressed as a discrete sum:

We note that the Variation-Diminishing Theorem we employ is a general result that applies to discrete sums just as it does to continuous integrals. The Variation-Diminishing Theorem guarantees that if our probability matrix $P$ is $TP_2$, then expected true reward $\mathbb{E}[R_t \mid \theta]$ remains simple and unimodal. The core result—and thus our ability to find a single optimal HedgeTune parameter—holds! This restated proof solidifies our findings and will be incorporated into the final paper. We thank you for prompting this important extension.

W3: Experimental Scope and Real-World Applicability

Thank you for this great question. To address this, we ran complementary experiments showing our method mitigates reward hacking on complex reasoning benchmarks (MMLU Pro, MATH, GPQA) even with large 8B proxy reward models!

1. Datasets: MMLU Pro (complex reasoning), MATH (mathematical problem solving), and GPQA (questions in natural sciences). A correct response gets a true reward of 1, while an incorrect response gets a true reward of 0.
2. LLMs: We leverage an open-source dataset with multiple responses generated per benchmark question from models such as GPT-4o-mini and Claude Haiku 3 [2].
3. Proxy Rewards: Each response is scored using various reward models, including InterLM-2 1.8B, Llama-3-Offset-Bias 8B, and Skywork-Llama-3.1 8B [2].

The authors of [2] note reward hacking with Best-of-$n$ on a suite of reward models and datasets. Due to time constraints during the rebuttal period, we only test Soft Best-of-$n$; however, we intend to show the results for all the methods in the final paper. We present the results in the table below:

The observed reward hacking curve exactly matches our theoretical prediction in Theorem 1. For example, Best-of-$n$ clearly demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (ranked # 10 out of all non-generative reward models on RewardBench [3]). While performance initially climbs from 40.70% at $n = 1$ to a peak of 45.36% at $n = 12$, it ultimately decreases to 43.35% as over-optimization sets in.

Hedging mitigates reward hacking in all cases. Soft BoN delivers more stable and often superior performance than BoN. On GPQA, it reduces the performance drop from over 2 points to less than 0.4 points. On MATH, where hacking is most severe, it effectively mitigates BoN’s 2.5-point collapse. This stability also enables Soft BoN to achieve a higher peak score on MMLU (Llama-3 8B).

The hedging benefit holds for learning from human preferences (RLHF), complex reasoning (MMLU), math (MATH), and science questions (GPQA), and is demonstrated with multiple large reward models (8B parameters). Please let us know if you have any additional questions about our setup or our results!

Q3: Scaling to larger LLMs and complex, open-ended tasks

We hope that our supplementary experimental results on complex reasoning demonstrate the effectiveness of hedging, even with larger reward models. The advantage of hedging is that it does not change across setups. Given access to "true" reward measurements, you would tune inference-time parameters to operate in a regime where you do not over-optimize to a proxy reward.

What does change is our ability to observe differences between true and proxy rewards, or even obtain such measurements. This might be feasible in narrow tasks, but more difficult in LLMs that are used across tasks like summarizing financial documents or refactoring a legacy codebase. Obtaining a consistent and reliable "true reward" signal in such domains is a significant research challenge in itself. If a form of hacking is undetectable by our evaluation process (human or otherwise), HedgeTune cannot guard against it. As the research community develops better methods for measuring hacking, we view HedgeTune as an immediate, lightweight, and principled tool to translate that into a more robust inference-time policy.

Summary:

We thank you for the thoughtful questions that have significantly clarified and strengthened our work. In response to your feedback, we will reframe HedgeTune's practicality for real-world tasks with expensive but limited ground-truth data. We have also extended our core theory to the discrete case and demonstrated our method's scalability with extensive new experiments on reasoning benchmarks. We hope these additions directly address your concerns and we appreciate your help in improving the paper.

References:

1. Gui et al. BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling. NeurIPS 2024.
2. Frick et al. How to Evaluate Reward Models for RLHF. ICLR 2025.
3. Lambert et al. "RewardBench: Evaluating Reward Models for Language Modeling". NAACL 2025.

Thank you for the thoughtful review. We address your questions and comments below and would be happy to answer any additional questions during the discussion period.

Q1: Additional intuition on technical assumptions like $TP_2$ and strictly MLR

We will add the following discussion about the technical conditions to the paper.

What the Technical Conditions Mean Intuitively: The conditions guarantee that the inference-time policies behave in a well-ordered manner: increasing the tuning parameter $\theta$ consistently makes the policy "greedier" and more likely to select outputs with high proxy rewards. The clearest example is the $n$ in Best-of-$n$: the larger the $n$ (which is generalized by our $\theta$ parameter), the more aggressively we optimize for the proxy.

This provides the most natural setting to study reward hacking. The monotone likelihood ratio (MLR) property induces a simple pairwise ordering: for any two policies with different parameters, the policy with the larger parameter is greedier. This property arises as a consequence of the stronger total positivity of order 2 ($TP_2$) condition. $TP_2$ is a structural property of the entire policy family, ensuring that such monotonic behavior holds uniformly across all parameter values and outcomes. It is a standard tool in the study of order statistics, utility theory, and stochastic processes.

Why They Help Us Understand Hacking: The core challenge of reward hacking is that the true reward function can be complex. This is where our theoretical contribution yields a significant practical payoff. We leverage the Variation-Diminishing Theorem to show that for any bounded reward, the expected true reward must follow a simple, predictable path that we characterize exactly. This result makes the problem of reward hacking tractable and allows us to search for the single, optimal hacking threshold.

W2/Q2: Empirical Setup

Thank you for this great question. To address this, we ran complementary experiments showing our method mitigates reward hacking on complex reasoning benchmarks (MMLU Pro, MATH, GPQA) even with large 8B proxy reward models.

1. Datasets: MMLU Pro (complex reasoning), MATH (mathematical problem solving), and GPQA (questions in natural sciences). A correct response gets a true reward of 1, while an incorrect response gets a true reward of 0.
2. LLMs: We leverage an open-source dataset with multiple responses generated per benchmark question from models such as GPT-4o-mini and Claude Haiku 3 [1].
3. Proxy Rewards: Each response is scored using various reward models, including InterLM-2 1.8B, Llama-3-Offset-Bias 8B, and Skywork-Llama-3.1 8B [1].

The authors of [1] note reward hacking with Best-of-$n$ on a suite of reward models and datasets. Due to time constraints during the rebuttal period, we only test Soft Best-of-$n$; however, we intend to show the results for all the methods in the final paper. We present the results in the table below:

The observed reward hacking curve exactly matches our theoretical prediction in Theorem 1. For example, Best-of-$n$ clearly demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (ranked #10 out of all non-generative reward models on RewardBench [2]). While performance initially climbs from $40.70\%$ at $n = 1$ to a peak of $45.36\%$ at $n = 12$, it ultimately decreases to $43.35\%$ as over-optimization sets in.

Hedging mitigates reward hacking in all cases. Soft BoN delivers more stable and often superior performance than BoN. On GPQA, it reduces the performance drop from over 2 points to less than 0.4 points. On MATH, where hacking is most severe, it effectively mitigates BoN’s 2.5-point collapse. This stability also enables Soft BoN to achieve a higher peak score on MMLU (Llama-3 8B).

The hedging benefit holds for learning from human preferences (RLHF), complex reasoning (MMLU), math (MATH), and science questions (GPQA), and is demonstrated with multiple large reward models (8B parameters). Please let us know if you have any additional questions about our setup or our results.

References:

1. Frick et al. How to Evaluate Reward Models for RLHF. ICLR 2025.
2. Lambert et al. RewardBench: Evaluating Reward Models for Language Modeling. NAACL 2025.

Q3: How does hedging compare with other approaches for mitigating reward hacking

Thank you for this excellent question. We provide the following conceptual analysis. The research community has developed several important strategies to combat reward hacking, which (broadly speaking) fall into the following categories:

Training-Time Regularization. This is the most common approach, typified by adding a KL-divergence penalty to the reward maximization objective as in RLHF. The goal is to prevent the policy from deviating too far from a trusted reference model. While a vital first line of defense, this approach requires expensive full model retraining and risks under-optimization if the penalty is too strong.

Improving the Proxy Reward Model. A second category aims to build a more robust reward signal that is inherently harder to hack. Ensembling involves averaging scores from multiple, independently trained reward models. However, this method multiplies inference costs and cannot fix systemic biases shared by all models in the ensemble. Meanwhile, robust training methods (e.g., RRM, InfoRM) modify the reward model's training process itself to disentangle response quality from spurious cues, introducing significant complexity to the training pipeline.

Inference-Time Methods. The closest analogues to our work include:

• Regularized Best-of-$n$ (RBoN): A powerful heuristic that balances the proxy reward against a penalty term: $R(x) - \beta \cdot \text{Penalty}(x)$.
• Inference-Time Pessimism: This method uses a principled rejection sampling step, which requires estimating a normalization constant.

Our Framework: Hedging introduces a distinct approach that operates purely at inference time. It offers two key advantages:

1. Zero Retraining Cost and Maximum Flexibility: Hedging is applied to an existing, trained policy using existing proxy reward models. A practitioner can take a single policy and calibrate it against different proxy rewards or for different tasks without incurring any retraining costs.
2. Principled and Efficient: We prove (Theorem 1) that the expected true reward function follows a predictable "rise-and-fall" path with a single optimal peak. HedgeTune therefore replaces heuristic balancing or more involved sampling schemes with a principled search for this unique optimum—a problem we prove is tractable.

However, one key limitation is that hedging requires a tuning dataset with "ground truth" labels. Ultimately, we see hedging not as a replacement for some of the previous methods. Instead, HedgeTune offers a computationally lightweight, theoretically-grounded, and highly flexible method to mitigate hacking at deployment.

W1: The presentation of the paper could be improved

Answer: We have gone through the paper again to fix typos and improve figure formatting. The major edits include enhancing the figure legends and style, streamlining the discussion of technical assumptions in Section 2 (e.g., merging Lines 153-160 with Lines 177-185), providing more direct proofs in the appendix, and making other minor edits throughout the paper. We believe these changes significantly improve the paper's quality and appreciate your helpful feedback.

Summary:

We thank you for providing a clear path to improve the paper. In response, we have demonstrated our method’s practical generality with new experiments on complex reasoning benchmarks (MMLU, MATH, GPQA) using large reward models. We have also clarified the intuition behind our core theoretical assumptions to improve accessibility, situated our work with a new conceptual analysis of alternative mitigation methods, and improved the presentation of the paper. We believe the paper is significantly stronger as a result of your review, so thank you!

We sincerely thank you for your detailed and thoughtful feedback. We address next the questions and weaknesses raised in your review.

Q1: Sample efficiency of HedgeTune

We agree sample efficiency is an important addition. We provide a new formal analysis, which we will add to the paper. As you noted, HedgeTune finds the root of an empirical gradient constructed from a dataset of $M$ true-reward evaluations. We leverage our paper's key theoretical contribution: under Theorem 1, the true gradient $R(\theta)$ will have exactly one-root, moving from positive to negative. Under boundedness assumptions, we can apply standard concentration inequalities like Hoeffding's inequality.

We remind the reviewer of the following expression for the gradient of the true reward (see Eq. 8 in Appendix A for the derivation):

$R(\theta) = E_{X \sim p_\theta}[r_t(X) \cdot \varphi(X, \theta)] = E_{U \sim \text{Unif}[0,1]}[r_t(U) \cdot \varphi(U, \theta) \cdot p_\theta(U)] = E_{U \sim \text{Unif}[0,1]}[h(U, \theta)]$

where $r_t(\cdot)$ is the true reward function, $\varphi(\cdot, \cdot)$ is the score function, and $p_\theta$ is the density of the inference-time policy parameterized by $\theta$. We consider the following assumptions on a compact parameter space $\Theta = [\theta_{\min}, \theta_{\max}]$ (equivalent to the search range of our algorithm):

Assumption 1 (Unimodality) The true performance function $f(\theta) = E_{X \sim p_\theta}[r_t(X)]$ is unimodal with a unique maximizer $\theta^* \in \Theta$, so its derivative, $R(\theta)$, is monotonically decreasing. This holds under Theorem 1.

Assumption 2 (Bounded Gradient Summands) There exists a constant $H < \infty$ such that:

Assumption 3 (Strong Slope) The true gradient $R(\theta)$ has a non-zero slope at its root $\theta^\ast$. That is, there exists a constant $m>0$ such that for all $\theta \in \Theta$:

Theorem

To guarantee that the estimated optimum $\theta^\dagger$ is within a desired accuracy $\epsilon$ of the true optimum $\theta^\ast$ with probability at least $1 - \alpha$, the required number of true-reward evaluations $M$ must satisfy:

Proof

The proof relies on showing that for a sufficiently large $M$, the empirical gradient will, with high probability, have opposite signs at the boundaries of the interval $[\theta^\ast - \epsilon, \theta^\ast + \epsilon]$. By the Strong Slope assumption (A3), the true gradient $R(\theta)$ is bounded away from zero at the points $\theta_{lo} = \theta^\ast - \epsilon$ and $\theta_{hi} = \theta^\ast + \epsilon$:

• At $\theta_{hi}$: $R(\theta_{hi}) \le -m|\theta_{hi} - \theta^*| = -m\epsilon$.
• At $\theta_{lo}$: $R(\theta_{lo}) \ge m|\theta_{lo} - \theta^*| = m\epsilon$.

Let $\mathcal{F}$ be the event that the empirical gradient ${\hat{R}}(\theta)$ does not respect these signs, then by Hoeffding's inequality, we have that:

M \ge \frac{2H^2}{m^2\epsilon^2}\log\left(\frac{2}{\alpha}\right)

Thank you so much for your helpful feedback and thoughtful reading of our work. We address below the questions and points raised in your review and would be excited to answer any additional questions.

W1: Does DPO optimize a reward function?

Thanks for raising this point. BoN, RLHF, and DPO make use of reward functions/signals. These reward functions can be implicit or explicit. For instance, BoN uses an explicit reward function (e.g., a model that assigns scores to responses). In contrast, DPO optimizes policy under an implicit reward function by recognizing that the optimal policy in RLHF (defined in Eq. 1 in our work) is an exponential tilt of the reference policy. DPO simply inverts this relationship and defines the loss directly on the policy (hence a language model being "secretly" a reward model). We will clarify this in the paper.

W2: The $10^{-3}$ KL gap as an empirical result

The $10^{-3}$ gap presented in Appendix B is indeed based on a numerical estimate under our assumptions. We first derive a closed-form expression of the KL gap (see Eq. 84) in terms of the Poisson parameter $\mu$. Next, we numerically find an upper bound of the expression (see Figure 2 for a plot) to demonstrate how negligible the gap is. For completeness, we will add a proof to the paper that shows explicitly why the bound holds. We added the proof to the end of our response so we would not disrupt the flow of our conversation. Please let us know if you have any questions about it.

W3/W4: Algorithmic nature and access to true rewards

You are correct that the need for true reward scores can be a practical barrier. Without a "true reward" to serve as a comparison (ground-truth) baseline, it is challenging to measure hacking. By definition, hacking occurs when a model over-optimizes to a proxy reward, and, consequently, precisely evaluating this phenomenon requires a reference ground-truth. We recognize this in Line 234 and plan to re-iterate it in a revised conclusion section, along with the point you raised on the online/offline nature of the method.

Q1: ‘Soft best-of-Poisson’ seems like a natural extension

We love this idea! After reading your review, we looked into this method and found that its properties can potentially extend our existing results, albeit at the cost of a more cumbersome mathematical analysis. We will include this as a blueprint for future research direction in the last section. We will add to the appendix: (i) the SBoP algorithm, (ii) its expected reward, and (iii) the KL divergence with respect to the reference policy. As one might expect, these correspond exactly to the expressions for Soft Best-of-$n$, but averaged over a Poisson distribution.

We find your question particularly interesting since it highlights our intention behind studying Best-of-Poisson and Soft Best-of-$n$. We wanted to leverage two distinct control mechanisms that inference-time alignment offers us: control over the generation through $n$ (be it deterministic or randomized) and control over the selection through the temperature $\lambda$. Extensions such as SBoP show that these methods can be combined, offering richer control over the alignment process. Thank you for prompting this interesting discussion!

Q2: Paper specified area

We do recognize that our work is largely technical. AI safety concerns are, at their heart, socio-technical [1]. The impact of model failures due to hacking (and beyond) will depend on the application and stakeholders at hand. One clear example is an AI agent generating controversial and toxic content on X (previously Twitter) to optimize its engagement metric at the direct expense of content quality and safety [2]. Other dangerous patterns of failure include models that learn to be sycophantic instead of truthful [3] or actively exploit loopholes to circumvent oversight [4]. The goal of our work was to study the phenomenon of hacking from a rigorous mathematical perspective -- which can help with the design of hacking mitigation (and hence AI safety methods) with provable performance guarantees. We find technical approaches such as ours important for AI safety in practice, although recognize that they must exist within a larger safety ecosystem where rigorous technical foundations inform responsible deployment practices and policy decisions, and vice-versa. We will add this discussion to the last section of the paper.

References:

1. Amodei et al. "Concrete Problems in AI Safety." 2016.
2. Pan et al. "Feedback loops with language models drive in-context reward hacking." ICML 2024.
3. Sharma et al. "Towards Understanding Sycophancy in Language Models." ICLR 2024.
4. OpenAI. "Detecting misbehavior in frontier reasoning models." Blog post 2025.

Limitations

Thanks -- we will improve this. We will rename our conclusion section to "Future Work and Limitations" and further highlight the points raised by the reviewer:

1. Need for true reward model to measure and control hacking as stated in Line 235;
2. Algorithmic advantages and limitations in terms of the online/offline nature;
3. Richer hedging policies (e.g., "Soft Best-of-Poisson");
4. How this work can be used in AI safety practices and deployment Thanks again for this suggestion.

Majority of the references are arXiV preprints:

This was an oversight on our part during the preparation of the manuscript, and we will fix it promptly!

Summary:

In summary, your feedback has prompted us to strengthen the paper on several key fronts. We have replaced a numerical claim with a formal proof, suggested new inference-time methods for future work, and elaborated on the limitations of our work and its impact to the AI safety community. We believe these additions make a much stronger case for our contributions, and we thank you for your guidance and time!

Proof of the KL gap:

Let the following two functions

denote, respectively, the BoP density with the Poisson rate $\mu$ and the exponential‑tilt density with parameter $\lambda$. For any $\mu > 0$, the value $\lambda^\ast=\lambda^\ast(\mu)$ is chosen so that the two laws have the same first moment (i.e., same expected reward). We first note the expected reward of the BoP policy lies in the interval $(\frac{1}{2}, 1)$. Additionally, the tilted expected reward is a strictly increasing function of $\lambda$ over the same range since its derivative is the (positive) variance of the reward under the tilted distribution. By the Intermediate Value Theorem, there exists a unique $\lambda^* = \mu + \delta(\mu)$, where $\delta(\mu) = \lambda^*(\mu) - \mu$, such that the two rewards are equal.

Because $g_{\lambda(\mu)}$ is the information projection of $q_{\mu}$ onto the exponential family $\{g_{\lambda}\}_{\lambda>0}$ under the linear reward-matching constraint, the Csiszár–Pythagoras identity gives

$D_{\text{KL}}(q_\mu || \pi_{\text{ref}}) - D_{\text{KL}}(g_{\lambda(\mu)} || \pi_{\text{ref}}) = D_{\text{KL}}(q_\mu || g_{\lambda(\mu)})$

To bound this KL divergence, we use a general inequality which we derive below from the properties of the likelihood ratio function. Let $L(x) = \frac{p(x)}{q(x)}$ denote the likelihood ratio between two distributions $p$ and $q$. We first define the KL divergence in terms of the convex function $\varphi(t) = t \log t - t + 1$ and then note that the expectation of any function is upper bounded by its essential supremum. We can retrieve the following upper bound:

$D_{\mathrm{KL}}(p || q) = E_q[\varphi(L(x))] \le \sup_{x} \varphi(L(x)) = \sup_{t \in \mathrm{Im}(L)} \varphi(t)$

In our problem, the likelihood ratio is given by

$L_\mu(x) = \frac{q_\mu(x)}{g_{\lambda^*}(x)} = \frac{(\mu x + 1)(e^{\lambda^\ast} - 1)}{\lambda^\ast e^{\mu}} \cdot e^{-\delta \mu x}$

To obtain a uniform bound on $L_\mu(x)$, define

$\alpha = \sup_{\mu > 0, x \in [0,1]} \left| \log L_\mu(x) \right|$ This is a well-posed, nested optimization problem requiring numerical solution. The procedure is as follows:

1. For a given $\mu$, numerically solve for the unique $\lambda^*$ satisfying the reward-matching equation.
2. For the resulting pair $(\mu, \lambda^*)$, compute $\max_{x \in [0,1]} |\log L_\mu(x)|$ by evaluating the boundary values and any interior critical points.
3. Maximize the result of step (2) over all $\mu > 0$.

This procedure is deterministic, and its solution yields a uniform bound:

Since the function $\varphi(t)$ is convex, we verify that its supremum over $[e^{-\alpha}, e^{\alpha}]$ is attained at an endpoint:

This bound is independent of $\mu$, and therefore holds uniformly.