Language Model Merging in Iterative Preference Learning

1. The comparison with previous baselines is insufficient. Results are only provided for dpo training baselines on Llama-8B, which weakens the authors' claims of effectiveness.
2. The main results are not compelling. The merged model does not demonstrate a clear performance gain compared to the individual checkpoints used for merging.
3. ExPO is highly relevant to the proposed method, simply mentioning that ExPO is a special case of Eq(2) is not enough. More experimental comparisons between ExPO's model merging method and the proposed method are needed, both on Qwen2 and Llama3.
4. The abstract is overly brief and does not adequately focus on the proposed method or the experimental results. Similarly, the introduction provides only a general overview of iterative preference learning, with little attention to the motivation behind the proposed approach. The only mention of motivation is on Line 56: "such training may deprive the performance of the final reward model and policy." However, this is presented as a hypothesis, without any preliminary experimental evidence to support it. This severely limits the overall contribution of the paper.
5. The figure captions lack detail and clarity. For instance, in Figure 2, "the scaling curve of $\alpha$" could benefit from a more thorough explanation, particularly regarding the y-axis and its significance. Figure 4 suffers from the same issue.
6. Captions for Figures 5 and 7 should include a summary of the key takeaway; otherwise, their purpose is unclear. In addition, the statement that "some data are not shown for illustration purposes" is vague. The authors should explain what data is omitted and why its exclusion does not affect the main point they aim to convey.
7. The abstract claims the proposed method can generate Pareto-superior models, but this is not convincingly supported by the experimental results. It's unclear to what method the Pareto frontier is being compared, and in which aspects the comparison is being made.

• The terms "reward modelling" (L73) and "reward modeling" need to be consistent throughout the paper.
• The approach uses equal weights for all models except the final one, which seems somewhat trivial. Since $T_i$ is fine-tuned from $T_{i-1}$, I am curious about the impact of other weighting strategy.
• In Table 2, while the merged model shows improvement compared to Iter 6, the performance gains relative to the SoTA models are inconsistent.

• The innovation of using model merging in the context of iterative preference learning appears limited, as similar techniques have been applied in the LLaMA 3 series [1]. Combining model merging with SimPO to create more powerful models seems like a more compelling direction to explore.
• The primary experimental results focus on the merging of alignment models rather than reward models, which weakens the evidence supporting the paper’s claims about its effectiveness for reward modeling. The overall objective of the paper needs to be articulated more clearly.
• The link between the claim of achieving more accurate reward models and the performance improvements in iterative optimization is insufficiently developed. Given that the ArmoRM reward model is used offline in iterative alignment, it is essential to conduct experiments using DPO-based reward models for iterative learning to validate the approach [2].

[1] The LLaMA 3 Herd of Models

[2] Lambert, Nathan, et al. “Rewardbench: Evaluating reward models for language modeling.” arXiv preprint arXiv:2403.13787 (2024).

1. The use of a first-order Taylor approximation assumes that $\log \pi_\theta(y \mid x)$ is approximately linear around $\theta_0$. However, this simplification overlooks the highly non-linear nature of deep language models due to complex activation functions like softmax. This means that higher-order terms, which can have significant effects, are ignored, potentially leading to inaccurate approximations. Additionally, the assumption that $\theta_t$ is close to $\theta_0$ may not hold in practice, especially when parameter differences are large, further reducing the reliability of the approximation.

2. The statement: "We hypothesize that such training may deprive the performance of the final reward model and policy," lacks references or empirical evidence. This makes it difficult to confirm that the identified issue exists or that the proposed solution effectively addresses it, thus weakening the rationale behind the study.

3. The paper does not discuss related work that shares conceptual similarities, such as [1]. While the primary difference is that the merged checkpoints in the proposed method are derived intuitively rather than offline.

[1] Rewarded Soups: Towards Pareto-optimal Alignment by Interpolating Weights Fine-Tuned on Diverse Rewards

4. Although the authors emphasize their scenario in an iterative setting rather than offline, the technique of using linear interpolation of parameter is fairly standard. The method seems to have been applied specifically to the iterative scenario without significant innovation in the underlying approach, which reduces the overall novelty of the contribution.

5. The paper does not address potential failures associated with the parameter-merging paradigm, as discussed in [2]. These known failures are likely applicable to your proposed method as well, and a discussion on this would contribute to a more balanced evaluation of the approach's limitations.

[2] Decoding-Time Language Model Alignment with Multiple Objectives

1. My major concern comes from the core motivation introduced in lines 154 - 161, where the Taylor approximation is applied. However, the key assumption for the first-order Taylor series expansion to be a good approximation is that $\Delta \theta$ should be sufficiently small relative to $\theta$, so the higher-order terms in the Taylor expansion can be safely ignored. The authors did not illustrate whether this important assumption holds.

2. For the main results reported in Table 2, it needs to discuss the training costs of the proposed method and the baselines to ensure a fair comparison. It is hard to tell whether the performance gain comes from the training overhead based on the limited implementation details as described in the current version.

3. While discussed in related works, the model merging methods are not compared in experiments. It will provide a more comprehensive understanding of the proposed method by including other model merging methods in experiments.

4. Missing reference. [1] WARM: On the Benefits of Weight Averaged Reward Models.