Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift

Thank you for your review and recognising the strengths of our work, these include the simplicity and clarity of our work, the rigorous theoretical analysis ensuring our methods reliability, and the robust experimental design. We provide clarifications and additional context regarding the points you raised.

Baseline Considerations

As clearly addressed in line 245-251 (Assumption 3 of Section 4), our setting assumes no prior knowledge of when or how preference drift occurs during the data collection process. Since we cannot identify the onset of "new" preferences in the dataset, ensuring the consistency of preferences is infeasible. Iterative updating of our model regularized to prior versions of the model via $\beta$ is also unfeasible as we consider the offline setting and as such we can only work with the data available and do not collect more data in streaming fashion.

We also disagree with the necessity of RAG as a baseline, for a fair comparison the data within the RAG retrieval database would also vary with time, it is unclear to us then how RAG would address the problem of LLM alignment under temporal preference drift, without a novel method to explicitly do so. As we work in the offline setting it is not an option to collect ‘new’ data for the RAG memory.

Win Rate Experiments

Thank you for pointing out the limitations of reward accuracy as an evaluation metric. We agree that the win rate, which measures the proportion of instances where the model generations outperform those of a baseline, offers a more actionable metric for assessing model generations. However, it is not trivial in our setting to properly evaluate the win rate. Existing works [5] use external LLMs to evaluate the generated responses and it is difficult to prompt LLMs to align with the preferences at the current time step $T$. We are encouraged that the paper the reviewer cited notes that the reward accuracy is a correlated metric with win rate when the trained model remains close to the reference models. Our experiments set $\tau = 0.1$ to ensure this property. If the reviewer has further advice on how to evaluate this metric we would be keen to engage further on this point.

Fine-Tuning Strategies

Thank you again for the suggestion. Based on your suggestion, we have trained additional language models with full fine-tuning for additional experiments. As Figure 6 shows, NS-DPO successfully fine-tunes the entire parameters of the language models, showing that NS-DPO continues to outperform DPO when all the model parameters are fine tuned. Please note that the primary distinction between DPO and NS-DPO lies in introducing the discount parameter γ in the loss function. This design choice should not lead to significant behavioral differences between full model fine-tuning and LoRA fine-tuning. This approach is well-supported in the literature for models of comparable size [1, 2, 3]. We report full fine-tuning results with smaller language models (llama-3.2-1b) due to compute limitations [4].

Additional Revisions

To address concerns regarding the clarity of our paper, we have revised the abstract and the caption for Figure 3. These updates explicitly describe NS-DPO’s behavior under stationary preferences and its relationship to ρdiff. We also modified the experiment section to report full fine-tuning results in Figure 6. All changes have been marked in blue in the updated manuscript. We have addressed all the questions the reviewer has raised and updated our manuscript to reflect this. We also reinforced the paper based on the reviewer’s suggestion. We ask that they increase their score or provide further arguments for rejecting our work.

References

[1] Muldrew, William, et al. "Active Preference Learning for Large Language Models." arXiv preprint arXiv:2402.08114 (2024).

[2] Ji, Kaixuan, Jiafan He, and Quanquan Gu. "Reinforcement learning from human feedback with active queries." arXiv preprint arXiv:2402.09401 (2024).

[3] Mehta, Viraj, et al. "Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration." (2023).

[4] Muldrew, William, et al. "Active Preference Learning for Large Language Models." arXiv preprint arXiv:2402.08114 (2024).

[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36, 2024.

Thank you for your review and for highlighting the strengths of our work, including the clarity of presentation and the comprehensiveness of the theoretical analysis. However, we disagree with the weaknesses identified by the reviewer.

Motivation for Using Older Data

“The study lacks strong motivation” - The offline time-varying problem that we solve in this paper has a strong basis and motivation in the related literature and has been studied in bandit settings [1,2], Bayesian optimisation [3,4], and reinforcement learning [5]. We encourage the reviewer to check out papers in our related work which clearly highlight this.

The relevance of older data and collecting new data

In this paper we focus on the offline learning scenario, where the dataset is pre-collected, and additional data collection is not possible. This makes the effective use of all available data critical. The theoretical guarantees we have obtained in Section 4 are based on no assumptions about when or how preference drift occurs (line 245-251, page 5). This complicates the identification of "fresh" or "updated" preferences.

NS-DPO employs an exponential weighting strategy, which downweights but does not discard older data. This approach allows the algorithm to leverage information from potentially valuable older datapoints while prioritizing more recent data. The advantage of this can be seen in Figure 2, where we compare NS-DPO to SW-DPO, an alternative that considers only the most recent data within a sliding window. Even when SW-DPO uses the optimal window size to capture only data with post-drift preferences, NS-DPO achieves faster convergence. This result highlights the benefit of incorporating older datapoints rather than discarding them outright.

Cost of Collecting Preference Data

While it is true that preference feedback can be less expensive to collect than scalar rewards, the cost of human annotation remains significant, particularly for RLHF applications. Recent work on active learning techniques to minimize the need for human queries [6, 7, 8], further illustrating the importance of efficient selection of training data. This reinforces the relevance of our setting, where we aim to maximize the utility of pre-collected datasets under non-stationary conditions.

Handling Stable Preferences

We analyze how the algorithm handles a mixture of drifting and fixed preference data in Figures 4 and 5. We test NS-DPO for different values of $\rho_\mathrm{diff}$ which controls the degree of fixed and drifting preferences within the data (see Section 5.1.2 lines 398-400 and Appendix D.1). Figure 3 shows that NS-DPO is robust in settings where no preference shift is observed. We are confused by this point as it is clearly addressed within our paper. We have added further analysis of fixed preferences in Appendix C where we analyze the gradient of the NS-DPO loss.

Planned Revisions

We have updated the abstract, the caption for Figure 3 to explicitly explain the behavior of NS-DPO under stationary preferences and its relationship to $\rho_\mathrm{diff}$, and highlight the offline nature of our work. All modifications are marked in blue. We have addressed all the questions the reviewer has raised and updated our manuscript to reflect this. We ask that the reviewer increase their score or provide further strong arguments for rejecting our work.

References

[1] Yoan Russac, Claire Vernade, and Olivier Cappé. Weighted linear bandits for non-stationary environments. Advances in neural information processing systems, 32, 2019.

[2] Louis Faury, Yoan Russac, Marc Abeille, and Clément Calauzenes. Regret bounds for generalized linear bandits under parameter drift. arXiv preprint arXiv:2103.05750, 2021.

[3] Bogunovic, Ilija, Jonathan Scarlett, and Volkan Cevher. "Time-varying Gaussian process bandit optimization." Artificial Intelligence and Statistics. PMLR, 2016.

[4] Brunzema, Paul, Alexander Von Rohr, and Sebastian Trimpe. "On controller tuning with time-varying Bayesian optimization." 2022 IEEE 61st Conference on Decision and Control (CDC). IEEE, 2022.

[5] Qichao Zhang, Dongbin Zhao. Data-based reinforcement learning for nonzero-sum games with unknown drift dynamics. IEEE Transactions on Cybernetics 49.8 (2018): 2874-2885.

[6] Ji, Kaixuan, Jiafan He, and Quanquan Gu. "Reinforcement learning from human feedback with active queries." arXiv preprint arXiv:2402.09401 (2024).

[7] Muldrew, William, et al. "Active Preference Learning for Large Language Models." arXiv preprint arXiv:2402.08114 (2024).

[8] Das, Nirjhar, et al. "Provably sample efficient rlhf via active preference optimization." arXiv preprint arXiv:2402.10500 (2024).

Dear Reviewer Cn55,

As the discussion deadline approaches, we kindly ask for your engagement in the discussion. We would greatly appreciate it if you could let us know whether you are satisfied with our responses and are open to reconsidering your score, or if you have additional questions or require further clarification.

Thank you for your time and feedback!

We thank the reviewer for recognising the strengths of our work including the importance of the problem and theoretical and empirical contributions our paper makes. However, we disagree with the weaknesses identified by the reviewer.

Assumptions on Preference Drift

The reviewer’s claim on our assumption is incorrect. We do not assume older data points have a higher probability of changed preferences labels. In our setting, we do not assume any knowledge of which points change preferences and which points do not. Without this information we must discount all points from the past as they are less reliable than more contemporary points. This is key to achieving our strong regret bounds in Theorem 2. Similar assumptions are widely accepted in various settings including bandits, Bayesian optimization, and reinforcement learning [1–6], we direct the reviewer to our related work for a detailed summary of papers that address this setting.

When there is no preference drift in the dataset, Theorem 2 and Appendix E.3 state that we should set $\gamma = 1$, weighting all points equally regardless of their timestamps. As shown in Figure 3, NS-DPO matches the performance of vanilla DPO in stationary environments even when $\gamma < 1$. As such NS-DPO is robust in settings with stationary factual data. Figure 4 and 5 address how NS-DPO performs given different ratios of fixed and changing preferences. The parameter $\rho_\mathrm{diff}$ controls this ratio as clearly stated in line 398 - 400.

Experimental Design

We disagree with the claim that our experiments are entirely synthetic. We incorporated real-world datasets with modifications to reflect preference changes. For example:

1. GlobalOpinionsQA: We constructed a dataset exhibiting gradual preference shifts between the preferences of citizens from different countries.

2. Helpful-Harmless and Ultrafeedback datasets: We applied preference drift patterns derived from widely used reward models, such as PairRM and ArmoRM.

This semi-synthetic approach with real prompts and responses allowed us to control the timing and intensity of preference shifts. This approach enabled detailed comparisons between NS-DPO and baseline methods under various interesting scenarios.

Suggested Alternative Approach

The approach suggested by the reviewer has limitations that NS-DPO addresses more effectively:

1. Compatibility with the Setting: The suggested approach by the reviewer requires knowledge of drift timing, which is incompatible with the setting of our paper. NS-DPO does not require explicit knowledge of when drift occurs or ends. This makes it well-suited for handling both sudden and gradual drift without additional steps for labeling or retraining.

2. Inaccuracy in Labeling Older Data: Using a model trained solely on the newest data can introduce inaccuracies when labeling older data, especially if the amount of recent data is limited.

3. Simplicity and Robustness: As shown in Figure 3, NS-DPO achieves comparable performance to vanilla DPO in stationary settings, demonstrating its robustness without the added complexity of iterative labeling.

Request for Reevaluation

In response to the reviewer’s comments, we have revised the caption of Figure 3 (page 9) to explicitly highlight that this experiment is conducted without preference drift. We have also added further commentary on stationary preferences in Appendix C. All modifications in the revised manuscript are marked in blue for convenience.

We request that the reviewer reconsider their evaluation, as we address all the highlighted weaknesses within the paper, and we have further revised the paper to clarify specific points. We ask that they increase their score or provide further strong arguments for rejecting our work.

References

[1] Yoan Russac, Claire Vernade, and Olivier Cappé. Weighted linear bandits for non-stationary environments. Advances in neural information processing systems, 32, 2019.

[2] Louis Faury, Yoan Russac, Marc Abeille, and Clément Calauzenes. Regret bounds for generalized linear bandits under parameter drift. arXiv preprint arXiv:2103.05750, 2021.

[3] Brunzema, Paul, Alexander Von Rohr, and Sebastian Trimpe. "On controller tuning with time-varying Bayesian optimization." 2022 IEEE 61st Conference on Decision and Control (CDC). IEEE, 2022.

[4] Zhou, Xingyu, and Ness Shroff. "No-regret algorithms for time-varying bayesian optimization." 2021 55th Annual Conference on Information Sciences and Systems (CISS). IEEE, 2021.

[5] Qichao Zhang, Dongbin Zhao. Data-based reinforcement learning for nonzero-sum games with unknown drift dynamics. IEEE Transactions on Cybernetics 49.8 (2018): 2874-2885.

[6] Vrabie, Draguna, and Frank Lewis. "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems." Neural Networks 22.3 (2009): 237-246.

Thank you for the detailed reply and additional experiments.

• Regarding the assumption: as mentioned in the rebuttal, "Without this information we must discount all points from the past as they are less reliable than more contemporary points." This actually assumes that older data is less reliable than recent data. However, this assumption does not hold for factual topics (e.g., "What is the capital of France?"), where older labels are just as reliable as recent ones. How does this assumption affect the algorithm's performance?
• As shown in Section 5.1.2, the non-stationarity, which is the main focus of the paper, is synthetically introduced based on predefined patterns. I believe it would be more convincing to conduct experiments on real-world datasets. Such datasets could be constructed by tracing data with temporal changes, such as president change or knowledge-editing datasets that may exhibit preference changes over time.
• Additionally, the suggested baseline does not require knowledge of temporal changes. You could simply evaluate it on the "new" data you defined. I still believe this is a worthwhile baseline to include.

For these reasons, I prefer to maintain my current score.