Improving Model Alignment Through Collective Intelligence of Open-Source Models

We sincerely thank the reviewer for their detailed and thoughtful assessment of our work. We appreciate your recognition of the originality, significance, and methodological clarity of our MoAA framework.

The paper provides a strong context by … (Jiang et al., 2023).

Thank you for suggesting these valuable references. We actually cited both papers in our work.

No comparison with other … data generation.

Thank you for this constructive feedback. We agree that comparing with a broader range of alignment methods provides a more comprehensive evaluation of our approach. We have conducted extensive comparisons with several contemporary alignment pipelines:

1. We benchmarked against MagPie [1], which follows a similar SFT and DPO process with its generated data, and Meta-Rewarding LLM [2], an iterative alignment method utilizing self-judgment for improvement. Our method demonstrates competitive or superior performance against both approaches, as detailed in Appendix E and Table 13 of the manuscript.
2. As shown in Figure 4, we compared the quality of data generated by single models versus our MoA approach for MoAA-SFT. Notably, even when compared to GPT-4o-05-13, one of the most powerful closed-source models available at the time of writing, our synthesized data produced superior performance metrics on downstream tasks.
3. We also conducted comprehensive comparisons of different reward models within the alignment pipeline. As demonstrated in Table 6, our MoA-based judge was evaluated against individual LLM judges and specialized trained reward models. The results show that our method delivers highly competitive performance.

We hope this answer your concern and are happy to answer any remaining questions!

[1] MAGPIE

[2] Meta-Rewarding Language Models

Multi-turn data: … discontinuity in multi-turn data.

We thank the reviewer for rasing this concern. We design our pipeline to be able to handle the multi-turn data coherently. As briefly outlined in Section 3.1.2, our approach ensures conversational continuity through a context-aware generation process. Specifically, when generating multi-turn conversational data, we maintain coherence by providing both proposer and aggregator models with the complete conversation history. For each new turn, the previous question-answer pairs from the aggregator are included as context, allowing models to generate responses that maintain thematic consistency and proper reference resolution across turns. The aggregator has access to the full conversation history when refining proposer outputs. We have found this approach significantly reduces discontinuities in multi-turn data generation. We will include a more detailed description of this process in the revised draft.

How do you ensure the diversity … alignment performance?

This is a great question. Ensuring diversity in our synthetic data is essential for effective alignment. MoA inherently generates diverse outputs by combining responses from multiple LLMs with different architectures, training methods, and capabilities. This diversity in the underlying models naturally leads to varied perspectives in the generated data.

As demonstrated in the original MoA paper and our Section 4.2, our approach doesn't simply combine model responses - it organically integrates and refines them. Potential biases from individual models are often corrected during aggregation, especially when using our most capable model as the aggregator.

Our experimental results in Section 4.3 confirm this approach preserves diversity while enhancing data quality, with MoAA-trained models showing better generalization across diverse tasks compared to those trained on individual model data. We acknowledge that more systematic study of bias mitigation in collaborative alignment frameworks remains a valuable direction for future research.

"What are the potential computational costs … for fine-tuning?"

Thank you for this question! While MoA data generation requires more computation than single models, our approach remains cost-effective because:

• Data generation is a one-time cost, while inference efficiency of the resulting fine-tuned model is equivalent to any similarly-sized model. As shown in Table 11, our distilled model achieving 90.6% of MoA performance at only 5.4% of the computational cost during inference.
• Our parallel implementation runs proposer models simultaneously, significantly reducing wall-clock time.
• Using efficient open-source models generates high-quality data without expensive proprietary APIs, with demonstrated cost savings of 23% compared to GPT-4o.

Thank you again for your constructive feedback. We believe these revisions have strengthened our manuscript and addressed your concerns while further highlighting the contributions of our work.

We sincerely thank the reviewer for their constructive feedback and thoughtful evaluation of our work on Mixture of Agents Alignment (MoAA). We appreciate your recognition of our paper's contributions and the validity of our claims and experimental design. We will address each concern below.

For MoA as a reward model, I think it's important to compare to some other baselines ...

Thank you for bringing this up! We will make sure to include and discuss the Replacing Judges with Juries paper [1] in the updated manuscript. To compare our MoA as reward model to the juries [1], we implemented the their method and evaluated it on the PPE benchmark [2]. PPE consists of 18k diverse data points spanning human preference and reasoning tasks. It achieves an average of 0.607 while our method achieves 0.661. Note that in order to make the comparison fair, we use the same models in the MoA for the panel of juries. The detailed results of the juries method and other methods are displayed below:

[1] Replacing Judges with Juries

[2] How to Evaluate Reward Models

Previous work [1] has already shown that mixture of agents could … If one model is significantly stronger than other models, will the method still work?

This is an excellent question. We have conducted several experiments to address this specific concern in the paper and list them here for clarity.

First we include an imbalanced ensemble with one model (Gemma-2-9B-it) significantly outperforming others. Specifically, we evaluated a small-scale MoA setup (due to limited compute) with Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 as proposers, and used a two-layer MoA with Gemma-2-9B-it as the aggregator. Table 15 demonstrates that the fine-tuned gemma model shows better performance than the strongest individual model in the mix by a large margin. This finding challenges the conventional thinking that alignment requires supervision from models more capable than the target model.

Second, we conducted extensive ablations examining MoA performance with varying combinations of proposers and aggregators, as documented in Table 8 and Table 15. Our analysis revealed that MoA performance serves as a reliable predictor of the distilled model's final performance, with several key insights transferring directly to MoA distillation. Notably, we found that while the ordering of proposers has minimal impact on outcomes, the capability of the aggregator model significantly influences results.

Another contribution of our work is demonstrating that high-quality alignment can be achieved using exclusively open-source models. Our framework achieves competitive performance compared to strong proprietary models like GPT4-o-2024-05-13 for SFT tasks, both in terms of model performance (shown in Table 3) and cost-efficiency (with 23% cost reduction as detailed in Table 10). This finding challenges the prevailing assumption in the field that effective alignment necessarily requires access to powerful proprietary models as supervisors, potentially democratizing powerful alignment techniques for the broader research community.

The whole MoA procesure is heavily ... potential ways for the whole MoA process to be more time efficient?

Thank you for the feedback. This practical consideration touches on a key motivation behind our distillation framework. While data generation with MoA is indeed computationally intensive, our approach offers a valuable efficiency tradeoff: the one-time cost of MoA data generation yields a distilled model that is both efficient during inference and recovers most of the performance benefits.

To address the computational challenges during data generation, we implemented batched implementation enables all proposer models to generate responses in parallel, significantly reducing runtime. We've also explored promising efficiency improvements, such as enabling the aggregator to begin generating while proposers are still completing their work, which showed encouraging early results. Although a comprehensive exploration of MoA efficiency optimizations falls beyond the scope of this work, our preliminary investigations suggest several promising directions for making the MoA process more time-efficient. We believe these efficiency considerations represent an important avenue for future research that could further enhance the practical applicability of our approach.

We sincerely thank the reviewer for their thorough evaluation of our work. We appreciate your recognition of the contributions of our alignment framework. We are pleased that you found our experimental design sound and our analysis of the quality of the MoAA-generated SFT and preference synthetic data useful. Regarding the concerns raised in your review we answer below.

The key contributions of this paper heavily rely on a previous paper: 'Mixture-of-Agents…

While our work builds upon the Mixture-of-Agents (MoA) framework, we believe our contributions extend significantly beyond simply showcasing MoA's utility for alignment. Our work introduces several novel aspects:

1. Novel use of multi-LLM as a Judge: we introduced MoA as a competitive reward model that requires zero additional training. As far as we know, not until recently, the researchers in the community started to look into using combined LLMs for model alignment. For example, [1] routes preference pairs from specific domains to domain-specific expert judges. In contrast, our MoA framework combines and refines judgments from multiple judges, utilizing their collective expertise to improve alignment. And this distinction makes our approach unique, since don’t use a router which can introduce additional bias during training. In addition, our method leverages the strengths of multiple models collaboratively rather than relying solely on isolated, domain-specific experts.

   [1] Tengyu Xu, et al. (2024). The Perfect Blend: Redefining RLHF with Mixture of Judges. (online on 09/30/2024)

2. Cost-Efficient Performance: Our alignment method, purely relying on open-source models, is competitive against strong proprietary LLM such as GPT4-o-2024-05-13, for SFT, on both the model performances (as shown in Table 3 in our manuscript) and cost (saves 23% as shown in Table 10). This represents a potential departure from the standard paradigm where alignment has typically required access to proprietary models as supervisors.

3. Strengthening the Strongest Model in MoA: Furthermore, we tried finetuning the strongest model in the model mixtures of MoA, and still observed a clear performance boost with MoAA. We think this is a non-trivial finding because improving the strongest model in the mix provides evidence that our method can potentially push the frontier open-source models further without the supervision of stronger LLMs. Specifically, we evaluated a small-scale MoA setup (due to limited compute) with Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 as proposers, and used a two-layer MoA with Gemma-2-9B-it as the aggregator to generate the data mix. Table 15 demonstrates that the fine-tuned gemma model shows better performance than the strongest individual model in the mix by a large margin. This finding challenges the conventional wisdom that alignment requires supervision from models more capable than the target model

We sincerely thank the reviewer for the positive feedback and the supportive overall recommendation. We are pleased that you recognize our novel application of the MOA architecture for alignment through both synthetic data generation (MoAA-SFT) and preference optimization (MoAA-DPO). We appreciate your acknowledgment of the key contributions of our work, particularly the novel use of MOA for alignment tasks (both SFT and DPO), the significant reduction in data generation costs compared to GPT-4o and the decrease in inference costs for the aligned model while maintaining comparable performance.

Furthermore, we agree that while the MOA architecture itself builds upon existing work, our paper's primary contribution lies in its novel application to the alignment problem, which you have acknowledged as valuable. Thank you!

Thank you again for your review. We believe our work opens up exciting new directions for cost-effective alignment techniques using ensembles of open-source models.