MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

We would like to thank the reviewer for the thorough review, which will help to improve our paper. Our reply can be summarized as the following:

1. Regarding notations

Thanks for your suggestions. To make the method easy to follow, we have simplified notations of weak teacher and strong student models in Section 4 of the rebuttal revision paper.

2. The definitions of “familiar” and “unfamiliar”

Thanks for your feedback. Since we assume different LLMs have different knowledge, familiar behaviors refer to self-generated samples, whereas unfamiliar behaviors refer to samples generated by other models. We have included these definitions in Sections 1 and 4 of the rebuttal revision paper.

3. Regarding positive behavior construction

Thanks for your detailed review and feedback. Our key idea of positive behavior construction is to select unfamiliar and high-quality labels for training the student model. To achieve this, we first identify unfamiliar labels as those generated by teacher models, rather than self-generated labels. Then, we use low perplexity to further filter high-quality labels among these unfamiliar ones. While higher perplexity does indicate greater unfamiliarity, it often corresponds to lower-quality labels due to increased noise. To clarify this motivation, we conduct an experiment using helpfulness questions. Specifically, we randomly sample 100 labels generated by teacher and student models, and then calculate the perplexity and reward for these labels. As shown in the table below, labels generated by teachers are categorized as unfamiliar based on the perplexity of the student model. Among these unfamiliar labels, the highest-quality ones are those generated by the 8B Llama3 teacher, which exhibit the lowest perplexity and the highest reward. Conversely, labels generated by the 7B Llama2 teacher have the highest perplexity but the lowest reward. These results have been included in Appendix G.4 of the rebuttal revision paper.

Thank you again for your careful review and feedback, and we are glad that the clarity has improved. Our reply can be summarized as the following:

1. Regarding perplexity-based filtering technique

Thanks for the feedback.

For relevant reference papers, previous works [1-3] have experimentally demonstrated the effectiveness of filtering high-quality data using low perplexity. Based on these findings, we use perplexity to filter high-quality data generated by multiple weak teachers. We have cited these relevant reference papers in Section 4.1 of the rebuttal revision paper.

For the experiment evidence, as shown in Figure 2, we find that the performance of strong students increases as the number of weak teachers increases at each iteration. This observation indirectly supports the use of low perplexity in selecting higher-quality positive samples from multiple weak teachers.

To further address concerns, we replace perplexity filtering with random sampling to directly assess the effectiveness of perplexity filtering. As shown in the table below, we observe that removing the perplexity filtering of weak labels (-ppl filtering) decreases the performance of helpfulness and harmlessness. This demonstrates that perplexity filtering is effective for filtering high-quality samples for helpfulness and harmlessness. We have included these results in Appendix G.2 of the rebuttal revision paper.

[1] When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale. NIPS@ATTRIB 2023.

[2] Scaling data-constrained language models. NIPS 2023.

[3] CCNet: Extracting high quality monolingual datasets from web crawl data. LREC 2019.

We hope this can help address your concerns; please also let us know if you have anything else to ask.

We thank the reviewer for the very detailed comments, our reply can be summarized as the following:

1. Regarding perplexity-based filtering technique

Thanks for the feedback.

For relevant reference papers, previous works [1-3] have experimentally demonstrated the effectiveness of filtering high-quality data using low perplexity. Based on these findings, we use perplexity to filter high-quality data generated by multiple weak teachers. We have cited these relevant reference papers in Section 4.1 of the rebuttal revision paper.

For the experiment evidence, as shown in Figure 2, we find that the performance of strong students increases as the number of weak teachers increases at each iteration. This observation indirectly supports the use of low perplexity in selecting higher-quality positive samples from multiple weak teachers.

To further address concerns, we replace perplexity filtering with random sampling to directly assess the effectiveness of perplexity filtering. As shown in the table below, we observe that removing the perplexity filtering of weak labels (-ppl filtering) decreases the performance of helpfulness and harmlessness. This demonstrates that perplexity filtering is effective for filtering high-quality samples for helpfulness and harmlessness. We have included these results in Appendix G.2 of the rebuttal revision paper.

2. Regarding the training order of models in algorithm

Thank you for your detailed feedback. The algorithm consists of two stages: initialization (line 974-976) and iterative optimization (line 977-996). For the initialization stage, the teacher model is trained first using golden labels. Then, the initialized teacher model provides supervision to initialize the student model. After that, for the iterative optimization stage, we iteratively optimize the student model and then optimize the teacher model.

This reason is that we find that the initialized student model is not well aligned with the teacher model, so we further optimize the student model to improve the alignment performance, and iteratively optimize the teacher and the student then. To address potential confusion, we have updated Algorithm 1 to clearly separate the initialization stage from the iterative optimization stage. Additionally, we have added more detailed explanations in Appendix A of the rebuttal revision paper.

3. Regarding the comparison setting in Table 1

Thank you for your feedback. Table 1 contains strong-to-weak alignment, self-alignment, and weak-to-strong alignment baselines. Different methods train the same 70b student model using different training strategies. Finally, we consistently evaluate the performance of 70b student models trained with different methods.

4. Regarding the third-party reward model

Thanks for the feedback.

For the specific use of the third-party reward model, following the previous paper [4], we concatenate the question and the model's answer as inputs to the reward model. The reward model's output is then scaled to a range of 0–1 using the sigmoid function. Finally, to facilitate comparison, we further scaled scores to 0–100 by multiplying by 100. We have updated the details of the reward calculation in Appendix D.1 of the rebuttal revision paper.

For the choice of the third-party reward model, as we discussed in section 5.4, we follow the previous paper [4] to choose the oasst-rm-2-pythia-6.9b-epoch-1 [5] as our third-party reward model, for the fair comparison. Additionally, Table 10 in the RewardBench paper [6] demonstrates the effectiveness of this reward model, which achieves 92.5 out of 100 in evaluating alignment performance for conversational tasks. Furthermore, as shown in Tables 2 and 3, GPT-4 and human evaluations provide results consistent with those of the reward model, confirming its reliability.

Dear Reviewer Wdjd,

Thank you for dedicating your time and effort to reviewing our submission. We sincerely appreciate your recognition of the key aspects of our work: clearly defining the problem, effectively demonstrating the motivations, and providing a solid foundation for future studies.

We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work. As the author-reviewer discussion period for ICLR 2025 will be over soon, please let us know if you require any additional information or clarification from our end. We are open to engaging in further discussions to enhance our submission. Thank you!

We sincerely thank the reviewer for valuable suggestions. These contributions are instrumental in enhancing the quality of our work. Below is a summary of our responses to your comments:

1. Evaluation on other alignment tasks

Thanks for the suggestion. Since we focus on aligning LLMs with helpfulness and harmlessness in weak-to-strong learning settings, we have conducted experiments on HH-RLHF and PKU-SafeRLHF. Experimental results on these datasets consistently demonstrate the effectiveness of our method. To further validate the performance of our method on general alignment tasks. We conduct experiments on the MT-Bench dataset [1]. This dataset encompasses a diverse range of tasks, including writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities questions. Following previous work [1], we use the GPT-4 to evaluate the model output with scores ranging from 1-10. Since MT-Bench contains general questions for assessing helpfulness, we directly evaluated methods trained on helpfulness datasets without additional fine-tuning. For self-alignment methods and MACPO, we choose checkpoints with the highest rewards in Table 1 for evaluation. As illustrated in the table below, our method, MACPO, consistently outperforms the baselines. Furthermore, these results illustrate the ability of our method to generalize to other alignment tasks. We have included these results in Appendix G.3 of the rebuttal revision paper.

2. Details of negative agents

Thanks for your feedback. As discussed in section 4.3, during our iteration optimization process, these negative agents are frozen, to consistently provide familiar negative behavior and avoid the degradation of the positive agents.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NIPS 23.

We appreciate your time and effort in reviewing this paper and providing these insightful comments. We respond as follows:

1. Clarification of our method

Thank you for the feedback. To address your concerns, we have clarified the terms "familiar behaviors" and "unfamiliar behaviors" by defining them explicitly: familiar behaviors refer to self-generated samples, whereas unfamiliar behaviors refer to samples generated by other agents. These definitions have been added in both Sections 1 and 4 for better understanding. Additionally, to enhance the explanation of our method, we have expanded the details of the optimization stages in Section 4 of the rebuttal revision paper.

2. Weak-to-strong learning setting

To practically study the problem of humans supervising superhuman models, following the previous work [1], we adopt the simple analogical setup that uses weak models to supervise strong models. Weak and strong models are defined based on the strength of their base abilities before the fine-tuning stage. In the fine-tuning stage, strong students are restricted to training on labels generated by weak teachers. This design tests whether weak labels can effectively guide the behavior of strong students. In this paper, we use LLaMA2-7B-base (fewer parameters) as the weak teacher and LLaMA2-70B-base (more parameters) as the strong student. The LLaMA2 paper [2] demonstrates the significant gap in base abilities between the 7B and 70B models, supporting the rationale for this setup.

3. Time consumption and performance comparison

Thanks for the suggestion.

Time consumption comparison at each iteration. After model initialization, compared to strong-to-weak alignment and self-alignment baselines, our method consumes similar time or less time at each iteration. Specifically, given the held-out question set, 7b LLMs take about 10 minutes to generate new samples, and the 70b LLMs take about 2 hours to generate new samples. For each iteration, MACPO requires sampling answers once from three 7b positive teacher models and one 70b positive student model, while RLAIF, RLCD and self-rewarding methods require sampling answers twice from 70b LLMs. Although SPIN only requires sampling once from 70b LLMs, it starts to decrease the alignment performance after the first iteration. We have included the computation efficiency discussion in Appendix E.2 of the rebuttal revision paper.

The iterative performance of strong-to-weak alignment methods. We extend RLAIF and RLCD into iterative alignment methods by resampling samples at each iteration. As shown in the table below, MACPO consistently outperforms these methods across multiple iterations. The reason is that strong-to-weak alignment methods ignore further improving the teacher agents. We have added these results in Appendix G.1 of the rebuttal revision paper.

For the iterative performance of self-alignment methods, as shown in Table 1, the performance of SPIN and Self-rewarding start to decrease after 1 or 2 iterations, which means that they have started to collapse. However, our approach can still further improve the performance of strong student LLM.

[1] Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. OPENAI 2023.

[2] Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta 2023.