Preference Learning with Response Time: Robust Losses and Guarantees

We thank the reviewer for highlighting the important work of Shvartsman et al. While our contributions are distinctly different, we agree that a direct comparison belongs in our work, and we will update our paper with a dedicated comparison with their work. In light of this prior work, we will update the title of the paper to "Robust Losses for Preference Learning with Response Time" in the final version.

While both papers leverage closed‑form DDM moments, our contributions and intended applications are complementary. Specifically:

1. Scalability to high-dimensional inputs.
   Our setting is motivated by large-scale “learning from human feedback” pipelines—e.g., fine‑tuning large language or vision models—where inputs live in very high dimensions and downstream tasks require a pointwise reward estimate. Shvartsman et al. use a GP prior on rewards and develop a variational approximate Bayesian inference approach using moment matching to refine the posterior so as to incorporate response time information. In particular, the paper suffers from the standard issues with GP regression and does not scale to high-dimensional (curse of dimensionality in GP regression [1,2,3]) and is not suitable for large-scale machine learning models.
   We completely bypass the use of variational approximations and moment-matching techniques as our method avoids modeling the full likelihood altogether. We develop a statistical learning theory approach with a Neyman orthogonal (aka locally robust) loss function that allows for flexible function approximation that can scale large input dimensions and function spaces.

2. Theoretical guarantees for reward learning.
   We are the first to show that incorporating response times can provably reduce estimation error in reward models in a supervised learning setting. In doing so, we also bridge a gap in prior DDM parameter‑estimation literature and modern learning from human feedback settings by deriving a loss function that admits finite‑sample error bounds for a wide range of function classes.

3. Function‑class‑agnostic method.
   Our orthogonal loss wraps around any reward learner—neural nets, kernels, or even Bayesian models—allowing practitioners to plug in any approximator and optimize via standard gradient methods. This agnosticism also paves the way to online learning (e.g., logistic bandits) or implicit reward‑learning (direct policy optimization as described in Appendix A) as future directions.

Regarding adaptability to the response time model in Shvartsman et al.

Aside from an initial‑state offset $x_o$ of the diffusion process, the DDM in Shvartsman et al. is mathematically equivalent to ours. We assume that the queries are symmetric, that is, swapping the queries $X^1$ and $X^2$ does not affect the model. In fact, enforcing symmetry—$\text{Pr}(Y=1│X_1=a, X_2 = b) =1–\text{Pr}(Y=1│X_1= b, X_2 =a)$—forces $x_o=0$, yielding exactly the DDM formulation (EZ- diffusion model) in our work. Although our current work does not require it, our orthogonal-loss construction can be extended to accommodate a nonzero initial-state offset.

Regarding different approaches for nuisance estimation: data-splitting, cross-fitting, data reuse.
In the experiments for linear functions, we use data splitting (highlighted in the linear paragraph section in Appendix D) in the main paper. For the case of linear rewards, our preliminary results using data-reuse and cross-fitting show that data-reuse consistently outperforms both cross-fitting and standard data-splitting. In the paper, to demonstrate the benefit of orthogonal loss over preference only estimator for the linear case, we stuck to the more conservative approach of data splitting. We will include a more detailed discussion in the appendix of the experimental section. We ran our neural network experiments with data reuse and data splitting. We consistently observed that data reuse outperformed the data splitting estimator.

Regarding methodology scope and limitations.
Our method presumes that the drift‐diffusion model (DDM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation—especially at modern, large scale—but it is important to recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.

[1] Giordano, M., Ray, K., & Schmidt-Hieber, J. (2022). On the inability of Gaussian process regression to optimally learn compositional functions. Advances in Neural Information Processing Systems.
[2] Binois, M., & Wycoff, N. (2022). A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization.
[3] Krauth, K., Bonilla, E. V., Cutajar, K., & Filippone, M. (2016). AutoGP: Exploring the capabilities and limitations of Gaussian process models.
[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.
[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.
[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology.

We thank the reviewer for the feedback. We address the comments and questions below:

1. Regarding estimation of nuisance function and robustness of preliminary reward error.
   Note that, since our Neyman‑orthogonal loss is first‑order insensitive to errors in both nuisance functions (the preliminary reward estimate and the response‑time model), one can tolerate significantly slower convergence in the preliminary estimator. For example, an $n^{-1/4}$ rate suffices to achieve $n^{-1/2}$ error rate in reward estimation for various function classes (such as RKHS or parametric models).
   Moreover, as the reviewer points out, the loss is in fact robust to estimation of the preliminary reward function. In particular, because the preliminary‑reward error only enters multiplied by the error in $t_o$, one may have arbitrarily slow convergence for the preliminary reward estimator so long as the response‑time estimator achieves an $n^{−1/2}$ (or the oracle convergence rate). For example, see the case of linear reward (line 219 (8)).

2. Regarding performance in downstream tasks.
   In our experiments, in addition to MSE loss, we also measure reward‐estimation quality in terms of policy value (see lines 288–289). Although this policy simply selects the option with the higher estimated reward, it serves as a proxy for how a large‑scale policy (e.g., a generative model) would behave after preference tuning. Figures 2 and 3 present these results, showing a clear improvement in policy value.

We thank the reviewer for the feedback. We address the comments and questions below:

1. Additional experiments based on [LZR+24].
   We tackle a more general—and inherently harder—supervised learning problem without adaptive query selection, yet even in the simpler adaptive best-arm identification setting, our method outperforms the estimator from [LZR⁺24]. We evaluate our orthogonal-loss estimator on bandit tasks derived from the real-world food-preference dataset [1], which contains choice and response-time data from 42 participants. Following their sequential-elimination protocol (Algorithm 1 in [LZR⁺24]), we substitute our orthogonal estimator for their response time estimator, run 70 independent trials per participant, and then summarize the probability of correctly identifying the best arm across participants (Figure 4 in [LZR+24]) via the 25th percentile (Q1), median, and 75th percentile (Q3) for each total sample budget. Results at budgets 500, 1000, and 1500 are shown below. We observe that our estimator clearly outperforms their proposed estimator in [LZR+24] for smaller budget values and the improvement reduces as the budget increases.

---

Budget = 500

---

Budget = 1000

---

Budget = 1500

---

2. Limitations of our work.
   Our method presumes that the drift‐diffusion model (EZ-DM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation. We recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.

3. Regarding asymptotic guarantees.
   Our non-asymptotic bounds in Section 5 already imply finite-sample convergence rates for linear reward models. We further derive asymptotic results in the linear setting to emphasize the exponential pointwise improvement over the preference-only estimator. We will clarify this distinction in the updated version.

4. "Does $Y=1$ signify a choice for $X^1$ or $X^2$?"
   Thank you for highlighting this. We will revise the text to state explicitly that $Y=1$ indicates that $X^1$ was chosen.

[1] S. M. Smith and I. Krajbich. Attention and choice across domains. Journal of Experimental Psychology.
[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.
[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.
[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology.

We thank the reviewer for increasing the score and judging the paper favorably.

Yes, the numbers in the table represent the probability of error in identifying the best arm, as in [LZR+24]. Our earlier reference to it as identifying the probability of the best arm was indeed a typo.

We thank the reviewer for the feedback. We address the comments and questions below:

1. Regarding accurate estimation of Nuisance functions.
   One of the core advantages of our Neyman‑orthogonal loss is that it is first‑order insensitive to errors in both nuisance functions (the preliminary reward estimate and the response‑time model). In fact, the estimate of the initial reward function can be arbitrarily poor if the estimate of $t_o(X)$ is $\sqrt{n}$ consistent. Furthermore, Theorem 5.1, implies that we only need slow rates for the nuisance functions to achieve the oracle rates; nuisance estimators can converge at rates orders of magnitude slower than the oracle rate. For example, for certain RKHS settings, one still obtains a $\frac{1}{\sqrt{n}}$ convergence rate even when the preliminary reward estimate and the response‑time estimate both converge at a much slower rate of $\frac{1}{n^{1/4}}$. Similarly, in equation (8) a $\frac{1}{n^{1/4}}$ rate suffices and can be easily achieved [7] (see the discussion on lines 220-224).

2. Regarding fair comparison of log-loss estimator and ortho-estimator.
   Our goal is not to argue that the log‑loss estimator is inferior—when only binary choices are available, log loss is indeed the minimax‐optimal approach. Rather, our focus is on incorporating response‑time data, which the log‑loss objective cannot accommodate. We include the log‑loss estimator purely as a baseline to quantify the gain from response‑time augmentation. Initializing our orthogonal estimator with the log‑loss solution speeds up convergence but confers no statistical advantage: all performance improvements stem from the additional information in T.

3. Regarding scope of the methodology.
   Our work targets large‑scale machine learning —characterized by high dimensional input and large datasets. To emulate real‑world conditions at this scale, we use semi‑synthetic data targeting this regime and work with image and text datasets. Fully real data experiments in this regime would be prohibitively expensive.
   At its core, our work is driven by the practical challenge of integrating drift‑diffusion models into modern, large‑scale machine‑learning pipelines. Whereas classical preference learners (e.g. Bradley–Terry) rely on a tractable likelihood and plug directly into gradient‑based training via maximum‑likelihood estimation, the DDM’s full likelihood is intractable—making it unclear how to exploit response time at scale. Our paper addresses this important gap.
   Our method presumes that the drift‐diffusion model (EZ-DM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation. We recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.

[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.
[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.
[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology.
[7] Ji, Ziwei, and Matus Telgarsky. "Risk and parameter convergence of logistic regression." arXiv preprint arXiv:1803.07300 (2018).

We thank the reviewer for the feedback. We address the comments and questions below:

1. Regarding intuition and connections to the Bradley-Terry model and example application.
   Note that the EZ diffusion model naturally incorporates the Bradley Terry model for preferences (see equation 1 in the paper). Crucially, the EZ diffusion model allows to exploit the response time information in learning the reward model. In practical settings, the response time can be cheaply recorded along with preferences and are indicative of preference strength (see the discussion on lines 43-54 for a detailed discussion). A key application is fine-tuning diffusion models with human preference data [c], which motivates our text- and image-based experiments in Figure 3.

2. Regarding the distribution of response time.
   One of the core contributions of our paper is handling the intractable likelihood of the response time. The likelihood of the response time is reduced to an infinite sum [a], and prior works approximate with either a parametrized heavy-tailed distribution [b] or series truncation [a]. In contrast, our Neyman-orthogonal loss entirely obviates likelihood computation, relying only on the closed-form conditional expectation.

3. Regarding bandit regret bounds.
   Our framework operates in a standard supervised-learning setting rather than an adaptive bandit environment. In our experiments, we evaluate our method by the MSE of reward estimation, and—as a proxy for downstream tasks—compare the value of the naïve policy that always picks the action with the highest estimated reward.
   This crucially differentiates us from prior work in [LZR+24] and addresses a major drawback of the prior approach. We tackle a more general—and inherently harder—supervised learning problem without adaptive query selection. The queries are assumed to be coming from an unknown distribution. While one could extend our approach to active query selection (e.g., optimal experiment design) and recover a bandit‐style analysis in the linear case—yielding guarantees similar to [LZR+24]—this is beyond the paper’s current scope. In practice, even for the case of the best arm selection problem, we observe that our method generally has a superior performance to [LZR+24]. For a more detailed discussion, please see our response to reviewer rfdo.

4. Regarding computational complexity.
   Our meta‐algorithm supports any function class and gradient‐based optimizer. Its runtime scales with the chosen optimizer, the model class, and the dataset size, but remains asymptotically identical to the preference‐only estimator. In particular, when instantiated with a linear reward model, the optimization is convex and thus solvable in polynomial time. Empirically, for neural network experiments, nuisance estimation and orthogonal-loss minimization require similar runtimes.

[a] Navarro, D. J., & Fuss, I. G. (2009). Fast and accurate calculations for first-passage times in Wiener diffusion models. Journal of mathematical psychology.
[b] Shvartsman, M., Letham, B., & Keeley, S. (2023). Response Time Improves Choice Prediction and Function Estimation for Gaussian Process Models of Perception and Preferences.
[c] Wu, X., Sun, K., Zhu, F., Zhao, R., & Li, H. (2023). Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision.