Thank you very much for the time and effort you spent reviewing our work and your helpful comments. We are especially appreciative of your comments on the importance of strategyproof RLHF without monetary incentives, the conceptual clarity of the tradeoff between incentive and policy alignment, and the novelty of the proposed algorithm.

Below we respond to your comments and questions, where we first jointly respond to your questions and suggestions about adding an experimental evaluation.

Adding an Experimental Evaluation:

While the paper’s main contributions are theoretical, we agree that experiments can help interpret and better understand the guarantees and the dynamics of strategic manipulation in RLHF. We therefore added synthetic experiments to the paper, an excerpt of which can be found further below. Following your suggestions and questions, we specifically analyzed the following two aspects:

1. “Dependence on the number of samples?”
   We provide results for sample sizes $n=20, 50, 100, 200$ with $d=16$ below. In the low sample regime $n=20$, the pessimistic approaches (Pessimistic Median of MLEs and Pessimistic Social Welfare) achieve only about 60% of the optimal policy’s social welfare (1.6 is the optimal social welfare in the experiments), but quickly improve performance with increasing sample size. We find that eventually the main performance limitation of Pessimistic Median of MLEs becomes not the sample size, but instead the inherent cost of being strategyproof (by using a median rule), as indicated by diminishing performance gains from more samples. That said, the influence of strategic manipulation also grows with more data, making strategyproofness increasingly valuable. By $n = 200$, Pessimistic Median of MLEs outperforms all baselines under strategic behavior.
2. “Performance in average-case scenarios (not worst-case)?”
   You are correct in that the proofs and guarantees are worst-case over possible problem instances. In the newly added experiments, we sample the true reward parameters $\theta_i^\star$ independently from a multivariate Gaussian distribution. From the experiments we observe that even in such average-case scenarios, labelers can have significant incentives to manipulate (see Difference column).

“More intuition about Assumption 2; What if it’s violated?”
Assumption 2 models the set of achievable feature expectations as a hyperrectangle rather than, for example, a sphere or general convex set. This structure simplifies the analysis and our intuition is that this should act as a reasonable approximation when features are bounded and vary independently across dimensions. In practice, the set $\{ \mathbb{E}_{s \sim \rho}[\phi(s, \pi(s))] : \pi \in \Pi \}$ often occupies a bounded convex subset of $\mathbb{R}^d$, and the hyperrectangle assumption can be seen as an idealized version of such sets. As noted in Footnote 4, our results extend naturally to mild violations of this assumption, with only small additive error terms in the guarantees.

Please let us know if any of your concerns or questions remain.

Excerpt from Newly Added Experiments

Experimental Setup: We simulate strategic labeling behavior by performing approximate gradient ascent (using SPSA) on each labeler's utility $J_i(\hat \pi)$ w.r.t. the labelers' internal reward parameters $\tilde \theta_i$, which govern their preference distributions $P_{\tilde \theta_i}$. Each labeler is initialized at their ground-truth reward vector $\theta_i^*$, sampled from a multivariate Gaussian. Labeler strategies are optimized for 200 steps. Since this process requires repeatedly relabeling data and rerunning each algorithm, we focus on small settings. All results are averaged over 5 random seeds, and we report standard errors. The results below are for embedding dimension $d = 16$, numer of labelers $k = 5$, and offline samples $n = 20, 50, 100, 200$, and additional runs will be included in the appendix of the paper. We report the suboptimality under both truthful and strategic labeling, along with the difference.

Overall, we observe that while the Naive MLEs and its pessimistic version, Pessimistic Social Welfare, perform well when labelers are truthful, the performance degrades substantially under strategic labeler behavior. In contrast, Pessimistic Median of MLEs exhibits almost no degradation, consistent with its approximate strategyproofness. Median of MLEs also shows slighty more robustness, though not to the same degree.

n = 20:

$n=50$

n=100

n = 200:

Thank you very much for taking the time to review our paper and your helpful suggestions. We are pleased to hear that you found our research topic timely and promising (S1) and our algorithm design natural (S2). We respond to your questions below.

W1. (Space of deviations is restricted) & Question 3 (What if labelers deviate from the BT model if they wanted to?):
In fact, our approximate strategyproofness result for the Pessimistic Median of MLEs does not rely on the assumption that deviating labelers must follow a BT model. The analysis extends directly to arbitrary deviation strategies, that is, labelers can follow any preference distribution if they wanted to. To briefly outline why, in the proof of Theorem 4.1, we show that for any fixed data $D_{-i}$ from the other labelers, the optimal outcome for labeler $i$ in terms of expected utility would be to directly report their true reward parameter $\theta_i^\star$. Since labelers only influence the mechanism through the data they provide, we then show that providing labels according to a BT model with parameter $\theta_i^\star$ yields expected utility within $O(\sqrt{d/n})$ of this best case. This argument does not rely on any assumptions about the structure of potential deviations. As for the specific case where a labeler deterministically selects the higher-reward option, this behavior corresponds to a BT model with high-magnitude parameters (rewards) and is therefore already included within our analysis. Thank you very much for highlighting this in your review, as this will be helpful to clarify in the paper. We have added a remark in the main text to make this generalization explicit.

W2. (Empirical validation; the paper could benefit from an empirical evaluation):
As you already pointed out, our main contributions are theoretical and conceptual. Still, we appreciate your suggestion, and added synthetic experiments to the paper (you can find an excerpt of the results with brief explanations further below).

Responses to your other Questions:

1. Coverage constants $\kappa_i$ and $\nu_i$:
   The constants $\kappa_i$ and $\nu_i$ importantly do not depend on the other parameters. This means that there are no other parameter dependencies hidden in these constants.
2. Source of randomness in Theorem 4.1:
   Thank you for catching this. The source of randomness is meant to come from sampling from $\smash{P_{\tilde \theta_i}}$ and $\smash{P_{\theta_i^\star}}$ and we shouldn’t have written the expectation over these distributions but instead that the labels in $\smash{\tilde D_i}$ are sampled from $P_{\tilde \theta_i}$ and the labels in $D_i^\star$ are sampled from $P_{\theta_i^\star}$. We fixed this.
3. "What if the labelers are allowed to deviate from the BT model?"
   (Please see the response to W1 above.)
4. “How difficult is it in practice to achieve $\varepsilon$-strategyproofness? What about non-pessimistic median MLE?”
   This is an interesting question, which we don’t have a fully definitive answer to. What we do know is that in terms of social welfare (i.e., policy suboptimality) the median approach without pessimism will be theoretically much worse since pessimism is generally needed for provable sample efficiency: Theorem 1.2 in [35] tells us that the non-pessimistic “naive” MLE approach to RLHF suffers from constant suboptimality. Regarding the strategyproofness, our intuition is that the basic median approach should be approximately strategyproof as well (partially confirmed by the newly added experiments). However, a theoretical analysis is quite challenging as it becomes fully probabilistic due to the lack of pessimistic confidence bounds.
5. “Are Pessimistic Social Welfare and MaxMin-RLHF epsilon-strategyproof?”
   No. We can adapt the examples in the proof of Proposition 3.3 so that for any value of $\varepsilon$ the incentive to manipulate exceeds $\varepsilon$. Note that the current counterexample in the proof already implies $\varepsilon > 1/4$. We included a short sentence mentioning this below the proposition.

Please let us know if any of your concerns or questions remain. We are happy to further discuss.

Excerpt from Newly Added Experiments

Experimental Setup: We simulate strategic labeling behavior by performing approximate gradient ascent (using SPSA) on each labeler's utility $J_i(\hat \pi)$ w.r.t. the labelers' internal reward parameters $\tilde \theta_i$, which govern their preference distributions $P_{\tilde \theta_i}$. Each labeler is initialized at their ground-truth reward vector $\theta_i^*$, sampled from a multivariate Gaussian. Strategies are optimized for 200 steps. Since this process requires repeatedly relabeling data and rerunning each algorithm, we focus on small settings. All results are averaged over 5 random seeds, and we report standard errors. The results below are for embedding dimension $d = 16$, numer of labelers $k = 5$, and offline samples $n = 50$. We report the suboptimality under both truthful and strategic labeling, along with the difference.

We observe that while the Naive MLEs and its pessimistic version, Pessimistic Social Welfare, perform well when labelers are truthful, the performance degrades substantially under strategic labeler behavior. In contrast, Pessimistic Median of MLEs exhibits almost no degradation, consistent with its approximate strategyproofness. Median of MLEs also shows slighty more robustness, though not to the same degree.

[35] Zhu et al. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.

Thank you for taking the time and effort to review our work and your helpful comments. We are glad to read that you found our results interesting and to be a good addition to the literature. We respond to your concerns and questions below.

1. “Is strategic manipulation of RLHF a practical issue?”
   We believe that strategic manipulation is a serious concern in practice, which is becoming increasingly important. We want to highlight previous work [29, 19] that showed ChatGPT models exhibiting more politically biased behavior after RLHF fine-tuning, which suggests that the human evaluators’ personal preferences influenced the model. One major take-away from this is that such real-world biases in who provides feedback have had measurable effects on model behavior. Other recent work also shows that even small-scale strategic manipulation of human feedback can have disproportionately large effects on LLM fine-tuning in actual RLHF pipelines [Baumgärtner et al. 2024, Haider et al. 2025], which our Proposition 3.4 also supports theoretically. There is also other recent evidence that fine-tuning on a narrow task with a strategic / malicious annotator can cause general misalignment [Betley et al. 2025].

2. “The paper is highly theoretical and no experiments are given.”
   While the paper’s main contributions are theoretical, we now added synthetic experiments to the paper to corroborate our theoretical results, following your and the other reviewers' suggestions. You can find selected results of the newly added experiments further below.
   We now also respond to your additional questions concerning "How realistic are the results?":

   1. “Typical values for the dimension d?”
      Yes, $d$ is the embedding dimension of the state-action pairs and its value really depends on the application. If these embeddings are derived from a neural network or LLM the dimension can be quite large, e.g., 512 to 4096 for LLMs. Compared to this, most open-source preference datasets include anywhere between 5-50k comparisons. We agree with your intuition that our bounds should be interpreted in relation to $d$ and $n$ and if $d \gg n$ offline RL(HF) bounds can become vacuous. However, this is expected and the $O(\sqrt{d/n})$ bound in offline RLHF is tight as discussed in Remark 4.4 and Corollary 4.5.
   2. ”How realistic is Assumption 2? Is it reasonable to expect that the set partially covers the space instead of spanning $\mathbb{R}^d$?”
      Assumption 2 models the set of achievable feature expectations as a hyperrectangle rather than, for example, a sphere or general convex set. This structure simplifies the analysis and should act as a reasonable approximation when features are bounded and vary independently across dimensions. In practice, the set $\{ \mathbb{E}_{s \sim \rho}[\phi(s, \pi(s))] : \pi \in \Pi \}$ often occupies a bounded convex subset of $\mathbb{R}^d$, and the hyperrectangle assumption can be seen as an idealized version of such sets. As noted in Footnote 4, our results extend naturally to mild violations of this assumption, with only small additive error terms in the guarantees.
   3. “It's not clear [to me] that the upper bound of Proposition 4.3 converges to 0.”
      In the mentioned paragraph we are referring to the asymptotic behavior of the algorithm and the corresponding bounds, which means we are looking at the case where the labelers can label an arbitrarily large amount of points even though of course this amount will always remain finite in practice. We wanted to express with this statement that if we could keep asking an increasing amount of labelers and each labeler for an arbitrarily large amount of data points, then the algorithm converges. We agree that the paragraph can be confusing and we rephrased the sentence in the paper to clarify this.
   4. “Aren’t financial incentives already involved? Why incentive design without payments?”
      While labelers are typically paid per annotation, implementing financial mechanisms that ensure strategyproofness in RLHF is often impractical. For instance, existing methods [24, 32] require conditioning payments or penalties on downstream effects, such as the performance of a fine-tuned policy, which are only observed well after labeling is complete. This makes enforcement difficult in real-world RLHF pipelines. Moreover, some RLHF settings, such as community-driven or crowdsourced alignment, do not involve financial compensation at all. In these cases, strategyproofness without payments is necessary by design. We also note that achieving strategyproofness without payments is significantly harder than when utility can be transferred directly through financial incentives.
   5. “Can labelers deviate to preference distributions other than the BT model?”
      While we define $P_{\tilde \theta_i}$ via the BT model of Eq. (1), our approximate strategyproofness results in fact do not rely on this specific parameterization. In the proof of Theorem 4.1, we show that for any fixed data from other labelers, the best possible outcome for labeler $i$ would be to directly report their true reward vector ${\theta_i^\star}$. Since labelers can only influence the mechanism through data, we argue that reporting according to $P_{\theta_i^\star}$ is nearly as good, within $O(\sqrt{d/n})$, regardless of what other preference distributions are allowed. This argument extends to arbitrary preference distributions, not just the BT model. We have added a remark in the main text to clarify this generality. Thank you for drawing our attention to this point.

Please let us know if any concerns or questions remain.

Example from Newly Added Experiments

Experimental Setup: We simulate strategic labeling behavior by performing approximate gradient ascent (using SPSA) on each labeler's utility $J_i(\hat \pi)$ w.r.t. the labelers' internal reward parameters $\tilde \theta_i$, which govern their preference distributions $P_{\tilde \theta_i}$. Each labeler is initialized at their ground-truth reward vector $\theta_i^*$, sampled from a multivariate Gaussian. Strategies are optimized for 200 steps. Since this process requires repeatedly relabeling data and rerunning each algorithm, we focus on small settings. All results are averaged over 5 random seeds, and we report standard errors. The results below are for embedding dimension $d = 16$, numer of labelers $k = 5$, and offline samples $n = 50$. We report the suboptimality under both truthful and strategic labeling, along with the difference.

We observe that while the Naive MLEs and its pessimistic version, Pessimistic Social Welfare, perform well when labelers are truthful, the performance degrades substantially under strategic labeler behavior. In contrast, Pessimistic Median of MLEs exhibits almost no degradation, consistent with its approximate strategyproofness. Median of MLEs also shows slighty more robustness, though not to the same degree.

[19] Hartmann et al., 2023. The political ideology of conversational AI.

[24] Park et al., 2024. RLHF from heterogeneous feedback via personalization and preference aggregation.

[29] Santurkar et al., 2023. Whose opinions do language models reflect?

[32] Sun et al., 2024. Mechanism design for LLM fine-tuning with multiple reward models.

Baumgärtner et al., 2024, Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data.

Haider et al., 2025. A framework for mitigating malicious RLHF feedback in LLM training using consensus based reward.

Betley et al., 2025. Emergent Misalignment: Narrow fine-tuning can produce broadly misaligned LLMs

Thank you for taking the time to review our work. We are happy to hear that you found the research topic interesting and timely with solid theoretical results, and our proposed method interesting.

Below, we address your concerns and questions about the motivation (what about large numbers of labelers), assuming linearly realizable rewards, and experimental support (we now added synthetic experiments):

1. Motivation ("What if the number of labelers is very large?"):
   When the number of individual labelers in RLHF is large, it is often more appropriate to think in terms of groups of labelers whose preferences closely align, for example along political, ideological, or stylistic lines. Each such group can be treated as a single effective labeler in our framework. Notably, such group effects have been observed in practice, where ChatGPT models became more politically biased after RLHF fine-tuning, which suggests that the human labelers’ personal preferences influenced the model [19, 29]. If a group is aware of this potential impact, it may have an incentive to strategize to steer the model toward its values. Moreover, single labelers (small groups) can have substantial influence even in large labeler populations. Proposition 3.4 shows this theoretically, and recent work by Baumgärtner et al. (2024) demonstrates empirically that injecting as little as 1-5% manipulated data is sufficient to shift a model's behavior toward targeted sentiment generation. Additional evidence from Betley et al. (2025) shows that fine-tuning on a narrow task with a strategic or malicious annotator can cause general misalignment. We now highlight these points more clearly in the introduction to strengthen the motivation.
2. Assuming Linearly Realizable Rewards:
   We agree that assuming linearly realizable rewards can be a restrictive assumption that may be violated in practice. For more complex reward models, our ideas could likely be extended, as MLE concentration results exist for more general models (Zhan et al. 2023). The pessimistic median based algorithm might not be strategyproof but we believe some variant of the algorithm might work for general reward models and this would be interesting to explore in the future.
3. Pessimistic Estimates and Trade-Offs:
   We agree with your intuition that pessimism may lead to conservative policy choices, however, it is necessary to some extent in offline RL(HF) to obtain sample efficient algorithms: Theorem 1.2 in [35] shows that without pessimism the “naive” MLE approach suffers from constant suboptimality in offline RLHF. We don’t have a definitive answer to potential trade-offs, but our newly added experiments compare also with non-pessimistic versions of the median and social welfare algorithms (see further below).

We would be happy to further discuss if any concerns or questions remain.

Excerpt from Additional Experiments

Experimental Setup: We simulate strategic labeling behavior by performing approximate gradient ascent (using SPSA) on each labeler's utility $J_i(\hat \pi)$ w.r.t. the labelers' internal reward parameters $\tilde \theta_i$, which govern their preference distributions $P_{\tilde \theta_i}$. Each labeler is initialized at their ground-truth reward vector $\theta_i^*$, sampled from a multivariate Gaussian. Strategies are optimized for 200 steps. Since this process requires repeatedly relabeling data and rerunning each algorithm, we focus on small settings. All results are averaged over 5 random seeds, and we report standard errors. The results below are for embedding dimension $d = 16$, numer of labelers $k = 5$, and offline samples $n = 50$. We report the suboptimality under both truthful and strategic labeling, along with the difference.

We observe that while the Naive MLEs and its pessimistic version, Pessimistic Social Welfare, perform well when labelers are truthful, the performance degrades substantially under strategic labeler behavior. In contrast, Pessimistic Median of MLEs exhibits almost no degradation, consistent with its approximate strategyproofness. Median of MLEs also shows slighty more robustness, though not to the same degree.

[35] Zhu et al., 2023. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.

Zhan et al., 2023. Provable Offline Preference-Based Reinforcement Learning

Baumgärtner et al. 2024, Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data.

Betley et al., 2025. Emergent Misalignment: Narrow fine-tuning can produce broadly misaligned LLMs