Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Thank you for your thoughtful and thorough review of our work.

[W2 & W3] We have carefully revised the paper in accordance with your detailed suggestions and have uploaded the updated version, following the ICLR guidelines.

• We included a more detailed explanation of the notation and reorganized the structure of the paper for better clarity.
• Due to the significant computational demands of PPO training and challenges with training stability, we were unable to provide the results in time. Therefore, we retained both Part 4.1 and Part 4.2 as initially presented.
  • Exp 4.1 serves as a toy example to demonstrate how the model transitions from the base model to the aligned model can be achieved using WSPO in some cases.
  • Exp 4.2 highlights the training stability of the WSPO algorithm.
  • Exp 4.3 provides a comprehensive evaluation.

[Q1.] Because of the configuration of llm-eval-harness repo.

For the MMLU dataset, the llm-evaluation-harness offers various evaluation settings. In our paper, we initially used the "continuation setting," where the model is not presented with answer choices for the questions. However, we have since switched to the "default setting" and updated the corresponding results in our paper.

	Qwen2-7B-base	Qwen2-7B-Instruct	Qwen2-7B-base + WSPO	Qwen2-7B-Instruct + WSPO
MMLU	69.43	69.94	69.37	69.44

[W1 & Q2.] Experiments for Llama3 families.

[Experiment] LLaMA 3.2 released lightweight 1B and 3B models on October 24, 2024. We conducted the following experiments using these models. We utilize 1B weak models to align 3B and 8B models, with the hyperparameters for each method specified in parentheses.

Arena-hard	Llama3.2-1B-Instruct	Llama3.2-3B-Instruct	Llama3.1-8B -Instruct
Instruct	11.40	29.60	50.30
dpo ($\beta=0.1$)	11.70	31.70	48.40
wspo ($\gamma=0.5$)	---	31.80	50.80

Arena-hard	Llama3.2-1B-Instruct	Llama3.2-3B-Instruct	Llama3.1-8B -Instruct
Instruct	11.40	29.60	50.30
SimPO	14.60 ($\beta=2.5, \gamma /\beta = 0.55$)	0.70 ($\beta=2.5, \gamma /\beta = 0.55$) ;26.50 ($\beta=10, \gamma /\beta = 0.30$)	0.00 ($\beta=2.5, \gamma /\beta = 0.55$) ;3.50 ($\beta=10, \gamma /\beta = 0.30$)
wspo ($\gamma=0.5$)	---	31.20	52.60

[Q3.] It also has an alignment enhancement phenomenon.

We use 7B model’s SFT and DPO checkpoint to compute the ratio as if they are $\pi_{r}^{weak}$ and $\pi_{ref}^{weak}$ and using this ratio as the label to align 7B model's SFT checkpoint, we derive the following results.

Arena-hard	Qwen2-7B-Base
SFT	4.70
DPO ($\beta=0.1$)	11.20
WSPO ($\gamma=1.0$)	10.90
WSPO ($\gamma=0.5$)	14.90
WSPO ($\gamma=0.1$)	15.30

When ($\gamma = 1.0$), the alignment strength is nearly identical to the DPO-aligned model's. However, as the alignment strength is reduced relative to the original DPO-trained model, an enhancement in alignment is observed.

We sincerely look forward to engaging in further discussions.

Dear Reviewer ssa6,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,

Authors

Thank you for your thoughtful and thorough review of our work.

[W3. ] Our method is to learn the predicted distribution difference before and after model alignment, rather than relying on a specific dataset.

However, we should acknowledge the importance of weak model alignment. After all, great models are built on great data.

[Experiment] To illustrate this, we use the rejected subset of the preference dataset, which may include toxic content. This dataset performs WSPO training to capture the predicted distribution difference. The results show that our method does not depend on the specific datasets.

Arena-hard	Qwen2-1.5B - Instruct	Qwen2-7B-Instruct
SFT	1.30	34.40
DPO ($\beta=0.1$)	3.40	34.00
WSPO ($\gamma=0.5$)	--	40.30

[W1 & Q1.] Experiments for Llama3 families.

[Experiment] LLaMA 3.2 released lightweight 1B and 3B models on October 24, 2024. We utilize 1B weak models to align 3B and 8B models, with the hyperparameters for each method specified in parentheses.

Arena-hard	Llama3.2-1B-Instruct	Llama3.2-3B-Instruct	Llama3.1-8B -Instruct
Instruct	11.40	29.60	50.30
dpo ($\beta=0.1$)	11.70	31.70	48.40
wspo ($\gamma=0.5$)	---	31.80	50.80

Arena-hard	Llama3.2-1B-Instruct	Llama3.2-3B-Instruct	Llama3.1-8B -Instruct
Instruct	11.40	29.60	50.30
SimPO	14.60 ($\beta=2.5, \gamma /\beta = 0.55$)	0.70 ($\beta=2.5, \gamma /\beta = 0.55$) ;26.50 ($\beta=10, \gamma /\beta = 0.30$)	0.00 ($\beta=2.5, \gamma /\beta = 0.55$) ;3.50 ($\beta=10, \gamma /\beta = 0.30$)
wspo ($\gamma=0.5$)	---	31.20	52.60

Section 3 of the paper [1] utilizes DPO as a distillation objective to transfer the transformer-based alignment capability into Mamba. Similarly, WSPO can also transfer alignment capabilities between models with different architectures.

Reference

[1] The Mamba in the Llama: Distilling and Accelerating Hybrid Models, https://arxiv.org/abs/2408.15237

[W2 & Q2. ] Some Empirical insights into alignment enhancement

• The weak model's parameter size might constrain weak model alignment, but transferring this alignment to stronger models may yield significant benefits.
• Our method avoids direct training on the preference dataset, thereby mitigating risks like overfitting and reward hacking.
• Model alignment methods such as DPO, SimPO, and others may lead to suboptimal alignment performance. In contrast, our approach can control alignment strength using the hyperparameter $\gamma$, which adjusts the alignment strength ratio between the strong and weak models.

[W4 & Q3.] Guidelines for hyperparameter

• [Practical Guidelines] In the paper, we derive that $\gamma = \lambda / \beta$, which represents the ratio of alignment regularization between the strong model and the weak model. Using weak models to align stronger models, typical choices like $\gamma = 0.1$ or $\gamma = 0.5$ work well. As mentioned in our paper, when $\gamma > 1$, the aligned strong models tend to favor the reference distribution. When $\gamma = 1$, the aligned strong models mirror the weakly aligned models. Notably, when $\gamma < 1$, alignment performance is often significantly improved.
• [Automatic Tuning Methods] One automated approach we can use is the adaptive-KL algorithm. This involves maximizing the reward term, $\beta \log \frac{\pi_r^{\text{weak}}}{\pi_{ref}^{\text{weak}}}$, and dynamically adjusting $\lambda$ .For example, $\lambda \leftarrow \lambda \cdot \alpha \text { if } KL >KL_{\text {target }}$ Or $\lambda \leftarrow \lambda / \alpha \text { if } KL<KL_{\text {target }}$ $\alpha$ is the adjustment factor.

We sincerely look forward to engaging in further discussions.

[Q4.] Computational efficiency compare to RLHF

1. Training a Reward Model:

   • For PPO in RLHF, we train a 1.5B reward model by adding a layer to the base LM to predict reward values.
   • For WSPO, we also train a 1.5B reward model using approaches like DPO, SimPO, and other algorithms.

   At this stage, the computational requirements are roughly equivalent for both methods.

2. Using the Reward Model for Training:

   • Once the reward model is trained, we use it to train both PPO and WSPO.
   • Performance benchmarking is conducted on a single node with 8x H100 GPUs under the following configuration:
     • Batch size: 32, Sequence length: 4K, Training steps: 5724

   The measured training times are as follows:

   Model Size	PPO	WSPO
   7B	95–120 hours	54 minutes

[Trade-off Analysis] Currently, we can use WSPO optimization as a precursor to PPO, as it may significantly accelerate PPO training. While PPO is more computationally intensive, it remains the most robust approach, enabling the exploration of optimizations beyond the dataset's distribution. Thus, PPO can serve as a valuable method to further enhance model performance.

[Q5.] Weak model alignment is important

• Weak model alignment is crucial. If a weakly aligned model perfectly matches the reference model, it will cease to perform gradient optimization effectively.
• Suboptimal models can still offer valuable insights. Weak model alignment might be constrained by the model's parameter size, but transferring this alignment to stronger models can yield significant benefits.

[Experiment] We use 0.5B models to align 7B models, while the alignment achieved by the 0.5B model is limited.

Arena-hard	Qwen2-0.5B - Instruct	Qwen2-7B-Instruct
SFT	0.90	34.40
DPO ($\beta = 0.1$)	0.90	34.00
WSPO ($\gamma = 0.1$)	--	40.60

We sincerely look forward to engaging in further discussions.

Dear Reviewer oPbS,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,

Authors

Thank you for your thoughtful and thorough review of our work.

[For weakness.] We have provided an intuitive explanation in the Method section and refined our description to make the theoretical derivations clearer and easier to understand. See the updated paper for details.

[Q1.] Here are three scenarios:

• Abundant high-quality, preference-chosen dataset:
  In this scenario, we can directly apply SFT to align the strong model. However, our method needs to incorporate additional weak models to aid the strong model's learning process.

• Abundant computational resources, robust reward model, and strong expertise in model training:
  In this case, PPO enables the model to explore unseen distributions within the dataset, enhancing its robustness. Here, our method demonstrates greater efficiency.

• Direct preference optimization:
  Techniques like SimPO offer a lightweight optimization approach. However, SimPO relies on two ad-hoc hyperparameters and, also requires a high-quality preference dataset. In contrast, our method is more invariant to hyperparameters and has stringent requirements on the dataset.

[Q2.] WSPO can apply to the probabilistic models

• Current vision-language tasks, are commonly composed of two components, one is an auto-regressive language model, another is an image encoder to extract representations into the core LLM (e.g. Llama 3.2-Vision[1]).
• LMs can be viewed as agents with a large decision space (i.e., the vocabulary size). In contrast, RL agents often operate within a much smaller decision space. However, as long as these agents rely on probabilistic modeling, we believe our method can be effectively applied. Our approach focuses on learning the distributional differences before and after fine-tuning, making it suitable for such scenarios.

[Experiment] We use the RLHF-v dataset (image-text pair preference dataset) to perform DPO and WSPO based on the vision-language models. We use 2B models to align 7B models. The evaluation results are shown as follows.

Arena-hard	Qwen2-2b-VL	Qwen2-7b-VL
Sft	1.40	5.30
DPO	1.50	4.90
WSPO	---	5.80

MMHal-Bench is a dataset consisting of image-question pairs, designed to assess hallucinations and response informativeness. Below are the evaluation results for Qwen2-7b-VL.

MMhal-Bench	Informativeness ($\Uparrow$, full score: 6)	Hullucination rate (($\Downarrow$, full score: 1))
Sft	3.91	0.23
DPO	3.80	0.27
WSPO	4.02	0.22

Reference:

[1]. https://huggingface.co/meta-llama/Llama-3.2-11B-Vision

We sincerely look forward to engaging in further discussions.

Dear Reviewer LZuG,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,

Authors

Thank you for your thoughtful and thorough review of our work.

[W1.] Suboptimal models can still offer valuable insights.

• If a weakly aligned model perfectly matches the reference model, it will no longer perform gradient optimization.
• However, aligning a weak model, as opposed to a strong one, provides greater flexibility in discovering a relatively optimal aligned model.
• Weak model alignment might be constrained by the model's parameter size, but transferring this alignment to stronger models can yield significant benefits.
• Our method introduces a way to adjust the alignment strength via the hyperparameter $\gamma$, which represents the ratio of regularization strength between the strong and weak models.

[Experiment] We use 0.5B models to align 7B models, while the alignment achieved by the 0.5B model is limited.

Arena-hard	Qwen2-0.5B - Instruct	Qwen2-7B-Instruct
SFT	0.90	34.40
DPO ($\beta = 0.1$)	0.90	34.00
WSPO ($\gamma = 0.1$)	--	40.60

[W2.] Our hyperparameter is easy to understand.

• Our loss function contains just one hyperparameter. In Section 5.2 of the Analysis paper, we visualize how different hyperparameter values influence the experimental results. Typically, values like 0.1 or 0.5 work well (e.g., for Qwen, Llama, etc.).
• This hyperparameter is easy to understand. It helps maintain alignment with the reference distribution while balancing the regularization strength between strong and weak models.

[Experiment] We selected the Llama3-8B-Instruct model as our weak reference model and the pre-aligned Simpo-aligned model from the SimPO project [1] as the weakly aligned model. By simply setting the hyperparameter $\gamma = 1$, we performed WSPO based on the Llama3-8B-Instruct model. The results are presented below, and, surprisingly, we also observed an alignment enhancement effect. This futther proves that in [W1], our method has the effect of adjusting the alignment strength.

Arena-Hard	LLama3-8B-Instruct
SFT	38.90
SimPO ($\beta=2.5, \gamma /\beta = 0.55$)	52.20
WSPO ($\gamma = 1.0$)	53.80

[W3. ] Trade-off between efficiency and parameter-tuned tasks.

• While we recognize that SimPO is lightweight, it requires tuning two hyperparameters, for which clear guidelines are lacking. As noted in the Hyperparameter Tuning section of their project page [1], this process can be challenging.
• One of the key contributions of our work is to prove that we can utilize the predicted distributions before and after model alignment as labels to guide the alignment process. As a result, we do not focus on comparing various advanced alignment algorithms.

[Experiment] We utilize 1B weak models to align 3B and 8B models, with the hyperparameters for each method specified in parentheses. SimPO hyperparameters are selected from their project page[1]. As shown in the following table, if slightly more resources are available, a method requiring less human intervention may be preferable.

Arena-hard	Llama3.2-1B-Instruct	Llama3.2-3B-Instruct	Llama3.1-8B -Instruct
Instruct	11.40	29.60	50.30
SimPO	14.60 ($\beta=2.5, \gamma /\beta = 0.55$)	0.70 ($\beta=2.5, \gamma /\beta = 0.55$) ;26.50 ($\beta=10, \gamma /\beta = 0.30$)	0.00 ($\beta=2.5, \gamma /\beta = 0.55$) ;3.50 ($\beta=10, \gamma /\beta = 0.30$)
wspo ($\gamma=0.5$)	---	31.20	52.60

Reference:

[1] https://github.com/princeton-nlp/SimPO

We sincerely look forward to engaging in further discussions.

Dear Reviewer 4UGn,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,

Authors