Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift

We thank the reviewer for their time and recognising the precision of our theoretical analysis. We address the questions and suggestions of the reviewer as follows.

Response to the questions

(1) “For the gradual drift experiments, were the datasets constructed such that the model sees increasing new data during training? ”

Please note that our setting considers offline learning scenarios, so all the datapoints with time step information are provided before the training starts. Because the reward for determining preference is changing gradually, the preferences between responses are more likely to change at datapoints with later timesteps. It is correct that the evaluation is done with the preferences at a time step that is later than all the training datapoints.

(2) “Does the evaluation occur on a held-out timestep that's later than all the training data?”

Yes, the evaluation is done with the preferences at a time step that is later than all the training datapoints.

(3) “The conclusion mentions ICL but I do not see any mention elsewhere”

In page 7, left column, line 369-372, we explained that tDPO refers to the in-context learning baseline. tDPO is a variant of DPO where the time step information is explicitly mentioned in the prompt. As demonstrated in the experiment result (Figure 2, Figure 9), tDPO did not significantly improve from stationary DPO.

Response to the suggestions

(1) “Could the authors provide more discussion on the implications of the theoretical results?”

We believe the core implications of the theoretical results are sufficiently discussed in the paper. To recap and summarize:

(1-a) Preference drift negatively affects the performance of policies. (in the form of tracking error, Theorem 2 in page 5)

(1-b) When we only know the total amount of preference drift ($B_T$ in the paper) and the time steps in the paper, we can use exponential weighting to address the preference drift. The optimal value of the discounting parameter $\gamma$ can be computed when we know $B_T$. (line 293-294, page 6)

(1-c) As $B_T$ approaches zero, NS-DPO with optimally defined $\gamma$ becomes identical to stationary DPO. (Corollary 4 in page 6)

(2) “The experimental analysis could be strengthened with the use of other evaluation metrics (e.g., win rate)”

Win rate evaluation is in the appendix, where we find it correlates well with reward accuracy. We point the reviewer to the UltraFeedback-LM dataset explained in Appendix C.2 and Appendix C.6 where we presented the results of win rate experiments. We used llama-3.2-1b-it for fully fine-tuning the policy within our compute limit. NS-DPO outperforms stationary DPO in terms of Length-Controlled Win rate (LCWR), similar to the reward accuracy comparisons.

(3) "inclusion of the SW-DPO baseline in the main experiments”

We point the reviewer to the additional experiment we conducted with SW-DPO (link 1), where we added SW-DPO results to Table 1 in page 18 of the paper. We trained policies with SW-DPO using the optimal window sizes to only allow the datapoints after the preference change. Despite using information not allowed in the setting of our work, SW-DPO significantly underperforms NS-DPO in Length-Controlled Win Rate (LCWR) of AlpacaEval. We attribute this to SW-DPO ignoring all the datapoints outside the window. We also present the results for UltraFeedback-RM datasets extended from Figure 9, where we show that SW-DPO shows worse reward accuracy when window size is not optimal (link 2) and similar accuracy when we assign optimal window sizes (link 3).

We hope our responses have addressed all the questions the reviewer had about our work. We respectfully ask the reviewer to reconsider their evaluation of our paper in light of our answers and additional experimental results. We are eager to help clarify any further questions the reviewer might have as we believe in the importance of the problem this paper seeks to address.

We thank the reviewer for their time and recognising the intriguing nature of this problem. We believe this will become a far more prevalent problem as LLM providers gather bespoke datasets over longer time frames and are keen to raise this within the LLM research community. As such, we are glad the reviewer agrees with us that this is aligned with the evolving direction of LLM research.

Response to the Concerns: “Adopting decay factor $\gamma$ may fail to account for sudden changes in user preferences.”

We address this at two points in our manuscript:

(a) Firstly we direct the reviewer to Assumption 1 [Section 4]. This assumption places no constraint on how the preference changes, as such our theoretical analysis addresses the case the reviewer raises where preferences change suddenly. As seen in Theorem 3, the value of $\gamma$ is set by $B_T$ which can be thought of as the ‘size’ of the preference change. A large sudden change will result in a smaller value of $\gamma$ focusing the algorithm’s learning on points closer to the present. We also note that this setting and exponential weighting technique are thoroughly analyzed in the multi-armed bandits literature. We point the reviewer to line 99-105 of page 2, left column where we discuss the references.

(b) Secondly, we explicitly test the setting where preferences change suddenly in our experiment section. We direct the reviewer to Figure 3 a) and Figure 4, in which we test NS-DPO when preferences change suddenly at a specific changepoint for a variety of different strengths of preference change (controlled by $\rho_\textrm{diff}$). We note that whilst DPO and IPO fail as the reviewer predicted, NS-DPO maintains its strong performance, even when we fixed the value of $\gamma$ for all sudden preference drifts!

Response to the Question: “Can NS-DPO be aware of the preference change of users?”

Yes, while the information is limited. The setting we consider in our work only allows the knowledge of the total amount of parameter drift, which is $B_T$. NS-DPO requires the knowledge of $B_T$ to set its exponential weight $\gamma$ for the best possible performance. In practice, we additionally provide experiment results where we test NS-DPO under various strengths of preference drift with a fixed value of $\gamma$, which still showed significant performance improvement over stationary baselines.

We hope our responses have addressed all the questions the reviewer had about our work and ask that they raise their score. We are eager to help clarify any further questions the reviewer might have as we believe in the importance of the problem this paper seeks to address.

We thank the reviewer for their response and recognising that our conclusions are well supported by both theoretical evidence and extensive experiments that show NS-DPO’s effectiveness in handling non-stationary preferences. We think this is a super important problem, unaddressed within the literature, and that it will become more prevalent as companies build proprietary datasets over longer time frames.

Response to Limitations

(1) To the best of our knowledge, we are the first to address the problem of non-stationary preferences within the LLM fine-tuning literature. As such we were unable to find any publicly available datasets that exhibit this property, as most are well curated for specific goals e.g. Helpful Harmless, and not preference drift.

(2) To account for this we created datasets by introducing preference drift into pre-existing, realistic preference datasets. We specifically tested challenging preference drifts in our experiments i.e. the preference change is strong and occurs suddenly. In Figures 3a and 4 we explicitly test NS-DPO on such preference drifts, a sudden shift from one reward model to another over varying times and with increasing strengths, and show that NS-DPO is robust to such changes even in the most challenging settings.

(3) By drawing attention to this problem and providing a simple and effective solution, we hope to attract the attention of the wider research community and collaborate to create more realistic datasets, however that is beyond the scope of this initial work which clearly demonstrates the problems that non-stationary preference drift can have on algorithms designed from a stationary perspective.

Once again we thank the reviewer for their time and ask that they consider raising their score further in light of these arguments. This problem is exciting and underexplored though it will be of great interest in the future, and we believe that our work addressed this in a simple and effective way.