We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.

[Separate effects of Bregman divergences and temporal awareness] The objective in Equation (12) that we propose for LLM alignment takes advantage of the insights we developed on MuJoCo. The first is that for $g=\mathrm{log}$ we rediscover the KL-divergence, so we set $\phi^{-1}(x)= \mathrm{log}(x)$, as in DPO or in SimPO. The second is that it is important to keep optimizing even when $\pi(\tau_w)>\pi(\tau_l)$, which is why set $g$ so that it introduces a constant $\gamma$ as in SimPO. The third is that objectives with an SFT term are more robust to noise, which is why we add an SFT term in Equation (12). The fourth is that a gradual transition from SFT to PO helps performance. In the next paragraph, we show that each of the last three insights leads to improved performance in the LLM setting.

[Additional LLM experiments] We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\gamma = 0$, SimPO with $\gamma>0$, and SFT + SimPO with $\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis. We report the results below.

These new results show that having a large $\gamma$, which encourages large updates even when $\pi(\tau_w)>\pi(\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.

[Performance of different TA-MPO algorithms] The difference in performance of the different TA-MPO algorithms is due to the fact that they induce different learning rate schedules for the SFT and PO terms. As shown by [1], these types of hyperparameters can have a strong influence on the final performance of the algorithm, which is why we explored a few variations for the temporal awareness schedule $\alpha$.

[1] Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." Neural Information Processing Systems, 2024.

We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.

[Objective of Figure 5] Understanding the gradient of an objective function is fundamental when analyzing gradient-descent-based methods. We plot the gradient of the objective functions to provide a visual representation of their behavior and to understand where they induce large or small updates. We plot the absolute value as the gradient w.r.t. the chosen trajectory is always positive and the gradient w.r.t. the rejected trajectory is always negative.

[LLM benchmark metrics] We agree with the reviewer, we will report the length controlled win rates as the main metric and report the win rates in parentheses.

[LLM benchmark] We performed our experiments on datasets with 7k samples as our computational resources do not allow for a comprehensive comparison on a larger dataset like ultrafeedback [1]. We highlight that other works have performed their LLM experiments exclusively on small datasets (e.g. [2]) and that [3] have shown that training on a smaller but higher quality dataset can lead to a superior performance than training on a larger dataset.

[Data generation figure] Unfortunately, this year's guidelines prevent us from sharing a figure in the rebuttal. We will include a figure explaining the data generation procedure in a camera ready version of this paper.

[Additional LLM experiments] We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\gamma = 0$, SimPO with $\gamma>0$, and SFT + SimPO with $\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis, for consistency. We report the results below.

These new results show that having a large $\gamma$, which encourages large updates even when $\pi(\tau_w)>\pi(\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.

[1] HuggingFaceH4/ultrafeedback_binarized

[2] Lu, Chris, et al. "Discovering preference optimization algorithms with and for large language models." Neural Information Processing Systems, 2024.

[3] Deng, Xun, et al. "Less is more: Improving llm alignment via preference data selection." arXiv preprint arXiv:2502.14560, 2025.

We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.

[LLM benchmarks] We remark that our LLM experiments have been performed in three different settings, for different combinations of language model (gemma-7b and mistral-7b) and preference dataset (dpo-mix-7k and capybara-7k). Only the evaluation was done on AlpacaEval, which is a standard benchmark for evaluating LLM performance (see for instance [1, 2]). We also highlight that automatic judges present a high level of agreement with human judges, as reported in the AlpacaEval repository.

[Transfer to LLM alignment of discovered objective] While the meta-learned optimizers can overfit to particular tasks, our paper shows that we can still extract generalizable knowledge from them. We provide additional results regarding the transfer of the insights obtained from MuJoCo below.

[Additional LLM experiments] We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\gamma = 0$, SimPO with $\gamma>0$, and SFT + SimPO with $\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis. We report the results below.

These new results show that having a large $\gamma$, which encourages large updates even when $\pi(\tau_w)>\pi(\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.

[1] Lu, Chris, et al. "Discovering preference optimization algorithms with and for large language models." Neural Information Processing Systems, 2024.

[2] Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." Neural Information Processing Systems, 2024.

We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.

[Additional LLM experiments] We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\gamma = 0$, SimPO with $\gamma>0$, and SFT + SimPO with $\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis. We report the results below.

These new results show that having a large $\gamma$, which encourages large updates even when $\pi(\tau_w)>\pi(\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.

[Related works] We provide discussions on related works in the Introduction, in Section 2.1, and in Appendices A and B. Additionally, we test eight existing preference optimization algorithms on our MuJoCo benchmark. Should the reviewer want us to test more algorithms, we would be glad to test any additional proposed algorithm. As to the proposed SPO method, we found four possible candidates in the literature, which we discuss below. Please let us know which one you would like us to include in our work.

• Self-Play Preference Optimization [1]: the authors propose a method to tackle non-Markovian, intransitive, and stochastic preference models. While their method can be applied to the offline setting, i.e. ours, the authors do not perform any experiment in this setting. Additionally, their method involves estimating a preference function, which does not fall in the DPO-style losses we consider in our work.

• Soft Preference Optimization [2]: the authors propose a DPO-style loss where the KL-divergence penalty between the current and the reference policy does not depend on the preference dataset. The KL-divergence is instead estimated by sampling trajectories from the current policy every $T$ policy updates, for a hyperparameter $T$. While the authors have not released code for their implementation, we can implement this algorithm should it be the one meant by the reviewer.

• Self-supervised Preference Optimization [3]: the authors propose a method to generate synthetic data and to use it alongside the original preference dataset to train an LLM. Although their proposed loss is similar to the DPO loss, methods that involve synthetic data are beyond the scope of our work.

• Step-by-step Preference Optimization [4]: the authors propose a method for post-training diffusion models using preference optimization. The method is tailored to the setting of diffusion models and cannot be straightforwardly applied to our setting.

[Objective of our MuJoCo benchmark] Evaluating PO algorithms on simpler environment has been done before in the literature [1, 5]. However, we are not aware of a recent work that has performed a comprehensive comparison of PO algorithms in a simple benchmark, or that has used the simple benchmark to obtain insights transferrable to the LLM alignment setting.

[Bregman divergences vs $f$-divergences] In our work, we choose to use Bregman divergences because $f$-divergences have already been explored by previous works in the literature [6], while the application of Bregman divergences to PO was missing from the literature.

[1] Swamy, Gokul, et al. "A Minimaximalist Approach to Reinforcement Learning from Human Feedback." International Conference on Machine Learning, 2024.

[2] Sharifnassab, Arsalan, et al. "Soft preference optimization: Aligning language models to expert distributions." arXiv preprint arXiv:2405.00747, 2024.

[3] Li, Jian, et al. "Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness." Findings of the Association for Computational Linguistics: EMNLP 2024.

[4] Liang, Zhanhao, et al. "Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization." Proceedings of the Computer Vision and Pattern Recognition Conference, 2025.

[5] Xu, Shusheng, et al. "Is dpo superior to ppo for llm alignment? a comprehensive study." arXiv preprint arXiv:2404.10719, 2024.

[6] Wang, Chaoqi, et al. "Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints". International Conference on Learning Representations, 2023.

As the reviewer-author discussion period is coming to an end, we would greatly appreciate it if you could kindly let us know if our rebuttal has resolved your concerns, and if you have any additional questions about our work.