Weighted-Reward Preference Optimization for Implicit Model Fusion

Thank you for your detailed review and insightful questions on our paper. We greatly value your insightful feedback and appreciation of our work's novelty and effectiveness. Below, we address your concerns in detail.

Q1: Regarding the limited improvements on MT-Bench.

A1: We would like to emphasize that the training dataset used in this work, UltraFeedback, is specifically designed for single-turn conversation tasks. The relatively modest improvement observed on MT-Bench, compared to the significant gains on AlpacaEval-2 and Arena-Hard, can be attributed to MT-Bench's distinct emphasis on multi-turn dialogue capabilities. In contrast, AlpacaEval-2 and Arena-Hard are tailored to assess models' instruction-following performance in single-turn interactions.

Moreover, the smaller dataset size of MT-Bench, coupled with its pointwise scoring methodology, limits its capacity to differentiate between models effectively. This limitation has also been noted in [1]. Despite these factors, the consistent improvements observed across all instruction-following benchmarks, including MT-Bench, underscore the general effectiveness of WRPO across diverse evaluation settings.

[1] Simpo: Simple preference optimization with a reference-free reward (Meng Y. et al., NeurIPS 2024)

Q2: Regarding the exclusion of dispreferred responses from source models.

A2: To address concerns about the exclusion of dispreferred responses from source models, we conducted a comparative analysis of reward scores across four categories: (1) preferred responses from source models, (2) preferred responses from the target model, (3) dispreferred responses from source models, and (4) dispreferred responses from the target model.

Our analysis reveals that target model preferred responses achieve an average reward score of 0.152, whereas source model dispreferred responses demonstrate a higher score of 0.158. This finding indicates that incorporating dispreferred responses from source models into the training objective could potentially lead to an undesirable reduction in the probability of higher-scoring responses. Such results would contradict our optimization objectives and potentially compromise the overall training effectiveness.

Moreover, we also conduct additional experiments to investigate the influence of introducing extra dispreferred responses from the source model. Specifically, we use an extension of WRPO loss in the equation below, where $y_{l_t}$ denotes the dispreferred response from the target model, and $y_{l_s}$ denotes the dispreferred response from source models. The empirical results are presented in the table below.

$

\mathcal{L}_{\text{WRPO}_{w/y_{l_s}}}=- \log \sigma \Biggl(\alpha \cdot \Bigl( \beta\log \frac{\pi_{\theta}(y_{w_s} \mid x)}{\pi_{\text{ref}}(y_{w_s} \mid x)} - \beta\log \frac{\pi_{\theta}(y_{l_s} \mid x)}{\pi_{\text{ref}}(y_{l_s} \mid x)}\Bigr) + (1-\alpha) \cdot \Bigl( \beta\log \frac{\pi_{\theta}(y_{w_t} \mid x)}{\pi_{\text{ref}}(y_{w_t} \mid x)} - \beta\log \frac{\pi_{\theta}(y_{l_t} \mid x)}{\pi_{\text{ref}}(y_{l_t} \mid x)}\Bigr)\Biggr)

$

Our analysis demonstrates that incorporating dispreferred responses from the source model results in a decreased length-controlled win rate on AlpacaEval-2. Furthermore, this inclusion leads to increased computational costs due to the necessity of additional forward passes for these responses during training. We will include these additional experiments and analysis in the revised version of the paper.

Q3: Regarding the in-depth analysis of WRPO's efficiency-effectiveness balance.

A3: The efficiency-effectiveness balance of WRPO is particularly evident when compared to collective LLM approaches. WRPO represents a significant advancement over traditional explicit model fusion (EMF) techniques, such as FuseLLM and FuseChat, by eliminating the need for complex probabilistic distribution fusion and vocabulary alignment procedures. Empirical evidence in Table 3 of our paper demonstrates that WRPO consistently outperforms when tested on identical target and source LLMs, highlighting its advantages through implicit model fusion.

Moreover, unlike ensemble methods (PackLLM and LLM-Blender) that require maintaining multiple large models simultaneously during inference, WRPO operates with a single model that is 106 times smaller, leading to substantial reductions in computational costs and improved inference efficiency. While WRPO may not exceed the absolute performance of larger ensemble systems across all evaluation metrics, its ability to achieve comparable results with far lower computational demands presents an elegant solution to the ongoing efficiency-effectiveness trade-off in language model deployment. These findings will be incorporated into the revised manuscript.

Q4: Regarding the discussion of limitations and future work.

A4: The limitations of this paper, along with directions for future work, can be summarized in three key aspects. First, the WRPO training objective currently incorporates only the highest-scoring response from source models as the preferred output for each prompt. This selective approach may overlook other valuable responses, potentially underutilizing the full range of capabilities offered by the source models. Future work could explore more inclusive methods that incorporate multiple responses from source models into the training objective.

Second, while WRPO demonstrates strong empirical performance, it relies heavily on existing preference optimization frameworks. A more rigorous theoretical analysis is needed to provide deeper insights into the internal fusion dynamics of WRPO and to further strengthen its theoretical foundation.

Finally, while WRPO significantly improves performance on instruction-following tasks, it may not perform as well on other tasks, such as MMLU. This limitation can largely be attributed to the narrow domain coverage of the training dataset. Future studies could address this by incorporating more diverse datasets from a wider range of domains.

We will include these discussions in the updated manuscript.

Thank you for your detailed review and insightful questions on our paper. We greatly value your insightful feedback and appreciation of our work's advancement and significance. Below, we address your concerns in detail.

Q1: Regarding the influence of source models.

A1: Great point. We agree that the performance of WRPO can be influenced by the choice and quality of source models. The choice of source models primarily depends on the specific goals. For instance, we can select domain-specific source models to enhance the target model's capabilities when it shows weakness in a specific domain. In our study, we focus on instruction-following tasks to ensure consistency with prior preference optimization research. Therefore, we selected ten prominent open-source LLMs with parameter sizes ranging from 9B to 123B, all of which exhibit strong performance on relevant benchmarks. As demonstrated in Table 5 of our paper, our experimental results show that increasing the number of source models generally leads to improved overall performance.

To further explore the impact of source model combinations, we conducted additional experiments using the AlpacaEval-2 benchmark. Specifically, we examined the influence of the response quality from source LLMs by comparing responses with different reward rankings. Our results presented in the following table indicate that responses from top-1 ranked source models consistently yield better results than those from top-2 ranked models. These results further highlight the importance of selecting high-quality responses to achieve optimal performance.

Moreover, we investigated the impact of model composition by dividing our ten source models into two balanced groups, each comprising five models with strong performance characteristics:

• Group 1: Gemma2-27B-IT, Gemma2-9B-IT, Qwen2-72B-Instruct, LLaMA3-70B-Instruct, and Yi-1.5-34B-Chat

• Group 2: Mistral-Large-Instruct-2407, Internlm2.5-20B-Chat, DeepSeek-V2-Chat, DeepSeek-Coder-V2-Instruct, and Phi-3-Medium-4K-Instruct

The experimental results reveal that various combinations of source models achieve comparable length-controlled win rates (LC). These findings demonstrate the consistent and robust performance of WRPO across various source model configurations. The insights derived from these experiments and analyses will be added to the updated manuscript.

Q2: Regarding the details and ablations of fusion coefficients tuning.

A2: In WPRO, we implement a dynamic adjustment mechanism for the fusion coefficient $\alpha$ to facilitate a gradual transition of the target model's distribution toward that of the source models. This approach is deliberately designed to be straightforward and computationally efficient, requiring minimal parameter tuning. The fusion coefficient $\alpha$ is initialized at 0.0 and increases linearly throughout the training process until it reaches a predetermined target value [1]. To determine the optimal target value, we employ a simple greedy search algorithm across [0.1, 0.3, 0.5, 0.7, 0.9], as described in Appendix A. This dynamic adjustment strategy effectively balances the contributions from both source and target models while addressing potential distribution discrepancies, making it suitable for various tasks and eliminating the need for complex parameter configurations or exhaustive optimization procedures.

Moreover, we conducted ablation experiments to compare two tuning strategies:

• Static strategy, where $\alpha$ remains constant at fixed values (0.1, 0.3, or 0.5) throughout training.
• Dynamic strategy, where $\alpha$ linearly increases from 0 to the target values (0.1, 0.3, or 0.5) during training.

The results on AlpacaEval-2 are presented below.

We find that the dynamic tuning strategy consistently outperforms the static strategy, further demonstrating the effectiveness of the dynamic tuning approach. We will provide a detailed explanation and incorporate these new experiments into the revised manuscript.

[1] BAM! Born-Again Multi-Task Networks for Natural Language Understanding (Clark et al., ACL 2019)

Q3: Regarding the computational resources of WRPO.

A3: Thank you for raising this question. Firstly, we would like to clarify that increasing the number of source models does not raise the time complexity of our method during training. The interaction with source models occurs exclusively during the preliminary phase before training, where we conduct offline sampling from source LLMs and utilize a reward model to evaluate and annotate responses, subsequently selecting one response with the highest reward score for each prompt. This step constitutes a fixed, one-time computational cost that is independent of the training process. Importantly, the source models do not participate in the actual training phase. Therefore, the inclusion of additional source models does not impose additional computational costs during training. Moreover, our comparative analysis demonstrates that WRPO introduces only a modest 16% increase in training time compared to DPO (which does not involve source models), regardless of the number of source models used.

We will include these detailed analyses in the revised paper to provide greater clarity.

Q4: Regarding the generalization of WRPO.

A4: Thank you for your thoughtful suggestion. Our experiments on the UltraFeedback dataset demonstrate WRPO's effectiveness in mitigating distributional shifts, suggesting its adaptability to other domains where distributional shifts may be more pronounced. Notably, applying WRPO to multilingual or cross-domain tasks would primarily require the use of domain-specific prompts, source models, and reward models to align with the particular characteristics of each task. Currently, we are conducting experiments in math and coding domains, and we will update the results once the experiments are concluded.

Dear Reviewer 42p3,

We are deeply grateful for the time and effort you have invested in providing insightful reviews and constructive feedback. Your comments have been instrumental in refining our work. We have carefully addressed each of your concerns and have incorporated additional experiments and analyses to strengthen our manuscript. The key points addressed are as follows:

• We have elaborated on the selection criteria and the impact of different source model combinations.
• We have detailed the dynamic tuning strategy for the fusion coefficient and conducted ablation experiments to compare it with static strategies.
• We have clarified that the inclusion of multiple source models affects only the offline preprocessing phase, without increasing the training complexity.
• We have further elucidated the generalization capabilities of WRPO in multilingual or cross-domain tasks.

We hope these revisions and expanded discussions have adequately addressed your concerns. As the Author-Reviewer discussion phase is nearing its conclusion, we would greatly appreciate any additional comments or questions that could help us further enhance our work.

Thank you again for your time and effort.

Best regards,

Authors

Q3: Regarding the computational resources of WRPO.

We supplement comprehensive runtime comparisons between DPO and WRPO using 8×A800 GPUs, examining scenarios with varying numbers of source LLMs. The detailed results are presented below:

Our empirical analysis demonstrates that WRPO maintains consistent computational efficiency across different numbers of source LLMs. Notably, WRPO introduces only a modest computational overhead of approximately 16% in training time compared to DPO (which does not involve source LLMs), regardless of the number of source LLMs involved.

Dear reviewer 42p3,

We sincerely appreciate your valuable feedback and the time you've dedicated to reviewing our paper. As we approach the final day of the discussion period, we kindly request any additional comments you may have on our revisions. We have invested significant effort in conducting additional experiments and addressing your queries, and would be grateful for your acknowledgment of our responses. Your feedback is crucial for us to effectively present our work to the research community. Please let us know if any points require further clarification.

Thank you very much and look forward to your replies!

Best regards,

Paper 6241 Authors

Dear Reviewer 42p3,

Thank you for your time and detailed feedback on our manuscript. As the reviewer-author discussion period ends today (December 2nd at 11:59 pm AoE), we would like to check if we have adequately addressed all your concerns.

Your insightful comments and questions have been instrumental in improving our work. We have carefully incorporated your feedback into the revised manuscript and hope that our responses and updates have successfully addressed all the points you raised.

We understand you have a busy schedule, but if you have any remaining questions or need further clarification, please let us know, and we will address them promptly. If you feel that we have satisfactorily addressed your concerns, we would greatly appreciate it if you could kindly consider revisting your initial score.

Thank you again for your valuable time and constructive feedback that has helped enhance the quality of our work.

Best regards,

The Authors of Paper 6241

Thanks for the author's thorough response addressing my concerns; I'd like to raise my score to 6.

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s significance and clarity. Below, we address your concerns in detail.

Q1: Regarding the efficiency of alignment.

A1: Thank you for raising this important point. To clarify, increasing the number of source models does not raise the time complexity of our method during the training phase. The interaction with source models is confined to the initial preprocessing phase before training. Specifically, this phase involves offline sampling, where responses are generated from source LLMs and evaluated using a reward model. For each prompt, the response with the highest reward score is selected. This step constitutes a fixed, one-time computational cost that is independent of the training process. Importantly, the source models do not participate in the actual training phase. Consequently, the inclusion of additional source models does not impose additional computational overhead during training. Moreover, our comparative analysis demonstrates that WRPO introduces only a modest 16% increase in training time compared to DPO (which does not involve source models), regardless of the number of source models used. We will include these detailed analyses in the revised paper to provide greater clarity.

Q2: Regarding the use of different source models.

A2: Great question. The selection of source models is primarily determined by specific objectives. When a target model exhibits limitations in particular domains, domain-specific source models can be strategically employed to enhance its capabilities. In our study, we focus on instruction-following tasks to ensure consistency with prior preference optimization research. Therefore, we selected ten prominent open-source LLMs with parameter sizes ranging from 9B to 123B, all of which exhibit strong performance on relevant benchmarks. As demonstrated in Table 5 of our paper, our experimental results show that increasing the number of source models generally leads to improved overall performance.

To further explore the impact of source model combinations, we conducted additional experiments using the AlpacaEval-2 benchmark. Specifically, we examined the influence of the response quality from source LLMs by comparing responses with different reward rankings. Our results presented in the following table indicate that responses from top-1 ranked source models consistently yield better results than those from top-2 ranked models. These results reinforce the value of selecting high-quality responses for achieving optimal performance.

Moreover, we investigated the impact of model composition by dividing our ten source models into two balanced groups, each comprising five models with strong performance characteristics. The groups are organized as follows:

• Group 1: Gemma2-27B-IT, Gemma2-9B-IT, Qwen2-72B-Instruct, LLaMA3-70B-Instruct, and Yi-1.5-34B-Chat

• Group 2: Mistral-Large-Instruct-2407, Internlm2.5-20B-Chat, DeepSeek-V2-Chat, DeepSeek-Coder-V2-Instruct, and Phi-3-Medium-4K-Instruct

The experimental results reveal that various combinations of source models achieve comparable length-controlled win rates (LC). These findings demonstrate the robust performance of WRPO across different source model configurations. The insights gained from these experiments and analyses will be incorporated into the updated manuscript.

Q3: Regarding the analysis of Figure 3 and Figure 4.

A3: Thank you for your thoughtful suggestion. We agree that Figures 3 and 4 could benefit from more detailed analysis and will expand the discussion as follows:

Figure 3 demonstrates the evolution of internal reward dynamics during preference optimization in the Target-SFT model across various preference pairs, with consistent learning rate and $\beta$ parameters. The internal reward margin, as defined in Eq. (7), comprises two components:

1. An on-policy reward margin: $(1-\alpha) \cdot (r(x, y_{w_t})-r(x, y_l))$
2. A hybrid-policy reward margin: $\alpha \cdot (r(x, y_{w_s})-r(x, y_l))$

Figure 3(a) presents the analysis of utilizing solely the on-policy reward margin ($\alpha = 0$). The observed reward margin approximates 0.2, indicating a relatively conservative optimization approach. This modest margin growth can be attributed to the model's limited exploration capability due to its exclusive reliance on on-policy samples.

In contrast, Figure 3(b) illustrates the effect of employing only the hybrid-policy reward margin ($\alpha = 1$). This configuration exhibits more aggressive optimization behavior, yielding margins exceeding 1.0. While this suggests enhanced discriminative capability between positive and negative samples, the substantial distribution shift inherent in the hybrid setting may compromise training stability and ultimately yield suboptimal results.

Figure 3(c) showcases our proposed weighted-reward mechanism, which synthesizes both on-policy and hybrid-policy reward margins through dynamic weighting. This approach achieves an optimal balance between the aforementioned extremes, generating moderate reward margins of approximately 0.5 and facilitating smooth margin transitions throughout the training process. The harmonious integration of hybrid-policy and on-policy components, as evidenced by the balanced optimization process, appears to be instrumental in the superior performance of our weighted-reward mechanism.

Figure 4 illustrates the ablation studies on the effectiveness of incorporating preferred responses from both source and target LLMs. We conduct these studies on two configurations: the baseline target model (Target) and its fine-tuned version (Target-SFT) to ensure a comprehensive evaluation. The analysis involves systematically removing either the source model's chosen response $y_{w_s}$ or the target model's chosen response $y_{w_t}$ from the optimization objective in Eq. (6) by setting $\alpha=0$ or $\alpha=1$, respectively.

In the Target setting, the removal of $y_{w_t}$ leads to a substantial decline of 25.8 points in the length-controlled win rate, indicating that the distribution shift between $y_{w_s}$ and $y_{l}$ creates challenges in directly utilizing source model responses for preference optimization. Moreover, this finding emphasizes the crucial role of $y_{w_t}$ in bridging this distribution gap. In the Target-SFT setting, although SFT helps mitigate the performance deterioration caused by removing $y_{w_t}$, its performance still lags behind our WRPO by 6.3 points, which combines both $y_{w_s}$ and $y_{w_t}$.

On the other hand, removing $y_{w_s}$ reduces WRPO to DPO based solely on self-sampled on-policy data. Notably, the exclusion of source model responses leads to performance declines of 3.5 points and 5.2 points in the Target and Target-SFT settings, respectively, highlighting the important role of $y_{w_s}$ in providing valuable preference signals through the weighted-reward mechanism.

We will include these discussions and analysis in the revised version.

Dear Reviewer McSp,

We sincerely appreciate the time and effort you have devoted to providing thoughtful reviews and valuable feedback. We have carefully addressed your concerns in detail and incorporated additional experiments and analyses, as summarized in the discussion:

• Demonstarted that the inclusion of multiple source models does not impact the time complexity of our method during the training phase.
• Detailed the selection and impact of different source model combinations.
• Expanded the discussion of Figures 3 and 4.

We hope these revisions and discussions have adequately addressed your concerns. As the Author-Reviewer discussion phase is ending soon, we would be grateful for any additional comments or questions that could further enhance our work.

Thank you again for your time and effort.

Best regards,

Authors

Q1: Regarding the efficiency of alignment.

We supplement comprehensive runtime comparisons between DPO and WRPO using 8×A800 GPUs, examining scenarios with varying numbers of source LLMs. The detailed results are presented below:

Our comparative analysis shows that WRPO maintains consistent computational efficiency across different numbers of source LLMs. Notably, WRPO incurs only a modest overhead of approximately 16% in training time on 8×A800 GPUs compared to DPO (which does not involve source LLMs), regardless of the number of source LLMs involved.

Dear reviewer McSp,

We sincerely appreciate your valuable feedback and the time you've dedicated to reviewing our paper. As we approach the final day of the extended discussion period, we kindly request any additional comments you may have on our revisions. We would be grateful for your acknowledgment of our responses. Your feedback is crucial for us to effectively present our work to the research community. Please let us know if any points require further clarification.

Thank you very much and look forward to your replies!

Best regards,

Paper 6241 Authors

Dear Reviewer McSp,

Thank you for your time and detailed feedback on our manuscript. As the reviewer-author discussion period ends today (December 2nd at 11:59 pm AoE), we would like to check if we have adequately addressed all your concerns.

Your insightful comments and questions have been instrumental in improving our work. We have carefully incorporated your feedback into the revised manuscript and hope that our responses and updates have successfully addressed all the points you raised.

We understand you have a busy schedule, but if you have any remaining questions or need further clarification, please let us know, and we will address them promptly. If you feel that we have satisfactorily addressed your concerns, we would greatly appreciate it if you could kindly consider revisting your initial score.

Thank you again for your valuable time and constructive feedback that has helped enhance the quality of our work.

Best regards,

The Authors of Paper 6241

Dear Reviewer McSp,

As we approach the final hour of the reviewer-author discussion period, we kindly remind you that this is our last opportunity for discussion during the rebuttal period. We would be immensely grateful if you could spare a moment to review our responses. Your feedback is invaluable to us, and we sincerely hope to receive your final thoughts before the deadline closes.

We deeply appreciate your time and dedication in evaluating our work.

With sincere gratitude,

Authors of ICLR Submission 6241