ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning

Thank you for your review and constructive feedback. Due to inclusion of the two new experiments, we unfortunately failed to finalize all responses before the original rebuttal deadline. We apologize for the delay and hope you will consider our reply nonetheless.

Weakness 1: It would be great to see real-world evaluations, though it is understandable that the paper presents a first proof-of-concept.

We agree real-world evaluations would strengthen the paper. We are constrained by limited publicly available response time datasets: Existing datasets from prior work are typically small with single evaluators, unsuited for demonstrating our method's strengths. As an approximation, we report MultiPref results using weighted inter-annotator agreement for strength rankings.

MultiPref experiments. We trained language reward models on MultiPref and evaluated on RewardBench v1 and v2, with v2 showing strong correlation with downstream performance [1]. More details in our response to jccL. Results across training sizes (10 seeds, differences to BT baseline in parentheses):

RL experiments. We conducted experiments on classical RL tasks (HalfCheetah, Swimmer, Walker2d, Highway merge), with RtRank commonly outperforming the BT baseline, particularly in low-noise scenarios. Detailed results in response to reviewer jccL.

Weakness 2: The method’s performance depends on strata quality; deeper analysis of stratification choices would strengthen the work (see also my question below).

We agree, but unfortunately this is limited by real-world data availability. Both new experiments do not rely on data-based stratification, as this is unnecessary for agreement-based or oracle-based preference strength rankings. We encourage future work on datasets supporting more realistic experiments.

Question 1: Can you comment on the feasibility and challenges of applying RtRank to real-world datasets? Are you planning or conducting such evaluations? What pitfalls do you expect when scaling RtRank to real-world data?

We believe real-world application is highly feasible. Computationally, RtRank could scale to large-scale LLM posttraining pipelines. Key challenges include stratification techniques and implementation constraints: entire rankings must be co-located in single on-device batches, limiting maximum stratum size and requiring non-iid batches. We will include a discussion of this in the paper.

As demonstrated by our new experiments, these technical limitations are manageable: short rankings convey substantial strength information, and non-iid batches are already common for grouping similar-length texts. The primary requirement is access to high-quality response time datasets.

Question 2: How sensitive is the method to poor stratification or suboptimal strata sizes? Could automated or adaptive stratification be a viable extension? The backbone of the work is an assumption that correlation between RT and preference-strength is high in “optimal” stratas; I believe it would strengthen the work if such discussion is included.

The RtRank-Perm baseline addresses this by shuffling response times, simulating poor stratification with no strength signal. Results show that while performance degrades, the model still learns ordinal preferences reasonably well. Suboptimal stratification would likely provide more signal than -Perm baseline. We will add discussion of stratum quality dependence to the paper.

Stratification robustness: Our method includes built-in protections: (1) single comparisons reduce to Bradley-Terry (Theorem 1), preserving preference learning and (2) Plackett-Luce loss handles ranking noise robustly.

Adaptive stratification could be viable! Future work in this direction may prove promising.

Question 3: Beyond RTs, have you thought about other implicit indicators of preference strength? Would it make sense to extend RtRank to take into account multiple indicators (assuming they exist in the dataset)?

Certainly! RtRank is flexible and can learn from any relative preference strength information. Response time is natural but could be extended by other factors such as mouse movement or gaze patterns, or replaced by other decision difficulty estimates.

[1] Malik et al., "RewardBench 2: Advancing Reward Model Evaluation", arXiv preprint, 2025.

Thank you for recognizing the strength of our contribution and highlighting areas for improvement! Due to inclusion of the two new experiments, we failed to finalize all responses before the original rebuttal deadline. We apologize for the delay and hope you will consider our reply nonetheless.

Weaknesses

This paper focuses on human response times but contains no actual human data.

We acknowledge this limitation. To partially address this, we conducted a new experiment using the MultiPref dataset - initially chosen as the only publicly available large-scale preference dataset with response time information. However, the reported response times are conflated with extensive metadata annotation (graded preferences across helpful/truthful/harmless categories plus justifications). We therefore generated strength rankings from multi-annotator agreement weighted by self-reported preference strength. While not using response times directly, this experiment is a proof of concept of our method's applicability to real language modeling data. We encourage future research with cleanly collected response times.

Language experiments. We trained language reward models on MultiPref and evaluated on RewardBench v1 and v2, with v2 showing strong correlation with downstream performance [1]. We use inter-annotator agreement for strength rankings. Results across training sizes (10 seeds, differences to BT baseline in parentheses):

Experimental details: Training on MultiPref subsets after withholding test sets, filtering texts >1024 tokens (~1400 test samples). Fine-tuned from Llama-3.1-8B-Instruct with optimal hyperparameters: 3 epochs, batch size 64, learning rate 15e-6, linear LR scheduler (0.05 warmup), gradient clipping 1.0, weight decay 0.1. For RtRank, strength rankings computed via annotator agreement weighted by reported strength, constructing balanced strata with no ties (mostly size 4).

No justification for the synthetic data creation or why it is a good model of human data

Our method makes minimal assumptions about human behavior. It requires only that (1) preferences approximately follow some utility function (standard Bradley-Terry assumption) and (2) we can approximately rank preference groups by strength. The only assumption about response times is an inverse monotonic relationship with preference strength.

Our synthetic data reflects these properties: preferences generated from utility functions and response times following inverse monotonic relationships (supported by psychology literature [2]). One of our settings follows the drift-diffusion model, which has extensive psychological validation [3], providing a model of RT that goes beyond the monotonic relationship. We choose base utility functions randomly, which while not aligned with specific human preferences, is reasonable since the model should learn any utility function matching our two key assumptions.

We will clarify this in the paper.

No results showing down-stream performance using the learned reward models.

We agree such results would strengthen the paper. The MultiPref experiments partially address this, as RewardBench v2 performance is strongly correlated with down-stream tasks [1]. For a more direct evaluation, we added RL experiments probing downstream performance on control tasks.

RL experiments. We conducted control domain experiments using established ground-truth reward functions to synthesize preferences and strength rankings. We tested three Mujoco environments (HalfCheetah, Swimmer, Walker2d) and Highway merge-v0, comparing Bradley-Terry baseline with RtRank using 5000 queries per training run (5 seeds each). RtRank often outperforms BT models, achieving higher accuracy by utilizing additional information. While performance degrades slightly under preference and response time noise, RtRank generally demonstrates robust performance, showing that reward models trained with RtRank often translate to better downstream performance. Detailed results are in our response to reviewer jccL.

[1] Malik et al., "RewardBench 2: Advancing Reward Model Evaluation", arXiv preprint, 2025.

[2] Milosavljevic et al., "The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure." Judgment and Decision Making, 2010.

[3] Ratcliff and McKoon, "The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks." Neural computation, 2008.

Thank you for your review and your insightful comments, which will help us improve the paper. We appreciate that you recognize the strength of our method, but also appreciate the weaknesses identified which we will discuss in the following.

1. First, the reason for the effectiveness of leveraging the response time to quantify the preference strength is not clear. Why the response time can indicate the preference strength? Is there any toy experiments on showing the effectiveness of use response?

Foundational support for RT. Psychology research demonstrates that response time is inversely correlated with preference strength [1]. When options are similar in utility, humans require more time to analyze differences and estimate each option's worth accurately. This principle underlies the drift-diffusion model, which has extensive empirical support. We discuss this thoroughly in Appendix A.1 (third paragraph) and Section 5 ("Learning from RTs"), with additional details in Appendix G.1.

Caveats of RT. Response times require clean data and careful stratification. We initially planned experiments using MultiPref dataset response times but found them unusable due to confounding time taken for metadata annotation. Unfortunately, we lack access to clean real-world RT datasets. However, our MultiPref experiments using agreement-based strength rankings provide strong support for real-world applicability when preference strength rankings are available.

RtRank beyond RT. Crucially, our core contribution, the rtrank loss, is not restricted to response times. It can incorporate any secondary information about preference strength, requiring only ordinal and local comparisons (no globally comparable values). Beyond response times, we could use mouse movement, gaze information, domain knowledge, or directly stated strength information commonly collected but underutilized in preference datasets.

2. There is no reinforcement learning experiment using the proposed method. Since the high accuracy or PDC do not directly reflect the better performance of RL agent, it would be better to show the performance of RL agent using the proposed reward model.

RL experiments. We conducted additional control domain experiments using established ground-truth reward functions to synthesize preferences and strength rankings. Following recent methodology [2], we tested three Mujoco environments (HalfCheetah, Swimmer, Walker2d) and Highway merge-v0, comparing Bradley-Terry baseline with RtRank using 5000 queries per training run (5 seeds each). Here, we report both max. and final scores (in brackets the comparison with the BT baseline):

RtRank generally outperforms BT models. For the Mujoco environments, we find that RtRank reward models also achieve higher accuracy, in line with improved downstream performance. For merge-v0, RtRank consistently outperforms despite similar reward model accuracy, suggesting better reward shaping.

Noise robustness. The results above were measured with perfect environment rewards. To measure robustness to noise, we also benchmarked with different noise levels (according to the methodology outlined in [2]), which perturbs the underlying ground-truth reward and in turn extracted response times and preferences. We find that RtRank outperforms the BT-model in low-noise scenarios, and only underperforms when overall RL performance deteriorates significantly. In environments that are generally robust to noise (such as merge-v0 or HalfCheetah), RtRank also stays stable.

MultiPref experiments. To demonstrate real-world applicability, we trained language reward models on MultiPref using inter-annotator agreement for strength rankings (lacking reliable RT data). While not a perfect test for RtRank with response times, this serves as proof of concept for our method's applicability to language modeling settings. Results across training sizes:

We fine-tuned from Llama-3.1-8B-Instruct with optimal hyperparameters found for both variants. Strength rankings used weighted annotator agreement (clear preference: weight 1.0, slight: 0.5). Results demonstrate consistent improvements across most settings.

All sizes are subsets of the total MultiPref dataset after withholding a test set. We filter both sets for texts exceeding 1024 tokens, resulting in ~1400 test samples. We shuffle the dataset with different seeds and report mean performance across 10 seeds. Columns show performance on in-distribution (MultiPref) test set and two out-of-distribution sets: RewardBench v1 and v2 [4,5] (measured per official methodology). Both RtRank and BT use the same optimal hyperparameters: 3 epochs, effective batch size 64 (16 on-device, 4 accumulation steps), learning rate 15e-6, linear LR scheduler with 0.05 warmup ratio, gradient clipping at 1.0, weight decay 0.1, Adam optimizer (0.9/0.999/1e-08). For RtRank, we compute strength rank by weighted annotator agreement, then construct balanced strata with no ties (most strata size 4).

We will include detailed descriptions of these experiments in the paper.

These preliminary results highlight RtRank's potential for downstream tasks. Particularly RewardBench v2 has been shown to correlate strongly with downstream performance [4], further validating our approach.

3. More comparison should be performed with other state-of-the-art baselines such as sequential pair-wise comparison methods [1] or list-wise comparison methods [2]. The authors compared the proposed method only with standard models.

Thank you for highlighting these relevant works! Both Choi et al. and Hwang et al. use transitivity to construct item rankings from pairwise comparisons, then apply Bradley-Terry loss to generated pairs. Their rankings do communicate strength information similar to ours.

However, they differ in two crucial aspects: (1) They construct rankings over items while we rank comparisons. (2) They require specialized sampling procedures while rtrank works with any preference data given local strength judgments (e.g., response times).

This makes direct comparison difficult since their contribution lies primarily in querying procedures and data augmentation, while ours focuses on novel loss formulation. We cannot apply their methods unchanged to our dataset, and implementing their querying strategy would make approaches quite similar. Importantly, the methods are inherently complementary.

We will add discussion of these approaches, their impact on preference strength learning, and potential complementary usage to our related work section.

---

[1] Milosavljevic et al., "The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure." Judgment and Decision Making, 2010.

[2] Metz et al., Reward Learning from Multiple Feedback Types, ICLR2025

[3] Lambert et al., "RewardBench: Evaluating Reward Models for Language Modeling", NAACL findings, 2025.

[4] Malik et al., "RewardBench 2: Advancing Reward Model Evaluation", arXiv preprint, 2025.

Thank you for your thoughtful review and for recognizing both contributions as strengths. Due to including two new experiments, we unfortunately failed to finalize this response for the original rebuttal deadline. We apologize for the delay and hope you will consider our response addressing each of your points below.

Response to weaknesses

1. Lacks training and testing results in real-world scenarios. The experimental section of the paper is based on synthetic data, which inherently involves strong assumptions. The actual effectiveness in real preference learning scenarios remains unknown.

Thank you for raising this point, which has also been echoed by other reviewers. In response, we have conducted two new experiments: (1) Training a reward model on the MultiPref dataset and (2) training a reinforcement learning agent on control environments.

Multipref. This dataset provides text preferences with rich metadata and four annotations per comparison. We initially chose this dataset as the only publicly available large-scale preference dataset with response time information. Unfortunately, total response time measures not only preference selection time, but also time for fine-grained graded preferences across multiple categories (helpful, truthful, harmless) and justifications. As this conflates the strength signal, we instead generated strength rankings from multi-annotator agreement rates weighed by self-reported preference strength. While this evaluation does not use RTs, it is grounded in real data and serves as proof of concept for our method's applicability to language modeling settings. We strongly encourage future research with response times collected in more controlled settings.

We trained reward models on MultiPref and evaluated on a held-out test set as well as RewardBench v1 and v2, with v2 correlating strongly with downstream performance [1]. Hyperparameters were tuned for RewardBench v2. Further details in response to reviewer jccL. We observed that optimizing BT involves a tradeoff between RewardBench and test-set accuracy. RtRank seemed more robust, potentially due to regularizing effects of strength. Results across training sizes (10 seeds for different random subsets, mean performance with differences to BT baseline in parentheses):

RL experiments. We conducted control domain experiments using ground-truth reward functions to synthesize preferences and strength rankings. We tested three Mujoco environments (HalfCheetah, Swimmer, Walker2d) and Highway merge-v0, comparing Bradley-Terry baseline with RtRank using 5000 queries per training run (5 seeds each). We find RtRank often outperforms BT models, particularly in low-noise scenarios. For the Mujoco environments, we find that RtRank reward models also achieve higher accuracy, in line with improved downstream performance. For merge-v0, RtRank consistently outperforms the baseline despite similar reward model accuracy, suggesting better reward shaping. Detailed results are in our response to reviewer jccL.

2. The PDC metric is difficult to utilize. The use of the PDC metric in real-world scenarios seems to conflict with the method itself. Since we use BT as an approximation for the golden truth in real scenarios, this implies the assumption that BT performs best on large-scale data. However, the paper's method shows better performance than the BT model on synthetic data, which contradicts this assumption.

Using BT models on large datasets as PDC ground-truth does not conflict with our method. The Bradley-Terry model theoretically identifies utilities up to shift [2]. With (infinitely) large datasets, it learns “correct” utilities up to shift, which suffices for PDC. RtRank's goal is not greater accuracy with infinite data, but extracting more information from limited data. This motivates training BT on large datasets, then testing sample-efficiency of methods like RtRank on small subsets evaluated against this ground-truth. We will clarify this in the paper.

We nonetheless agree that this can be difficult to use in practice. It suits method development rather than production model evaluation. For final training, you would use all available data. Nonetheless, PDC provides valuable development insights.

[1] Malik et al., "RewardBench 2: Advancing Reward Model Evaluation", arXiv preprint, 2025.

[2] Train, Discrete Choice Methods with Simulation, Cambridge University Press, 2009.