Improving Model Alignment Through Collective Intelligence of Open-Source Models

1. This work can be summarized as directly applying MoA to create data, followed by standard SFT and DPO training. This limits its novelty and seems an incremental work based on MoA.

2. Similar to MoA, there are many empirical choices involved in MoAA that could affect its performance and its generalizability. For example, the ensemble of models used for MoAA shows great variance in the end performance (as also discussed in Appendix B). The authors in this work conducted an empirical sweep to determine the models used, but this makes it highly impractical if one wants to extend MoAA in other domains such as math, coding, or agentic tasks.

3. Main results in Table 1 (and similarly in Table 3 and 6) shows significant improvement from MoAA when comparing training with MoAA-generated data with prompts from UltraFeedback + UltraChat against no-training/training with single agent generated data. However, I believe the presentation (e.g., in Table 1) can be highly misleading, as one natural baseline should be comparing against directly training on the existing UltraChat and UltraFeedback with SFT/DPO - a simple setting that has been used in many work and requires no additional data curation. This comparison is necessary to help readers put into perspective the benefit introduced by MoAA, but is right now missing.

4. Since MoAA relies on using multiple large language models (e.g., three 70B+ models for SFT data and also three 70B+ models for DPO data), it becomes necessary to justify this compute-cost-performance trade-off against simply querying proprietary models such as GPT-4o (i.e., in Table 3 and is missing in Table 6). Specifically, in addition to performance, the authors should also compare the inference-cost of using MoAA v.s. GPT-4o to better put this trade-off into perspective.

1. The technical contributions in this work are relatively limited. The pipeline in this paper largely relies on existing work (i.e., MoA), merely using its results to construct the preference data for SFT and DPO.

2. Specifically, one-third of the article's methodology section is dedicated to introducing foundational concepts such as DPO and MoA from existing work. Apart from these existing works, the technical differences presented in this paper seem limited to: (1) Adjusting more suitable prompts for specific tasks. (2) Making some details compatible with multi-turn data. (3) Adding a dynamic criteria filtering component in section S3.2. Nevertheless, the ablation results in Appendix Table 12 indicate that the enhancement from criteria filtering is relatively minimal, only around one point. Therefore, the main improvement of this paper still primarily stems from the existing work MoA.

3. I also have some concerns about the comparison of this method with related works. Although MoAA improves the base model's capabilities, and the authors conducted extensive experiments to test the effectiveness of different teacher models, the comparisons are mainly between MoAA and the base model or other settings of MoAA. There are only Table 5 compares with two other reward models (i.e., ArmRM and PairRM), and MoAA seems to have failed to make a significant difference in the other model. It would be more comprehensive to include comparisons with other similar methods, rather than just analyzing the performance of MoAA under different settings. At least, compared MoAA to MoA to illustrate the improvement solely brought by MoAA. However, MoA achieve 65.1 and 9.25 in AlpacaEval2(LC) and MT-Bench, which have outperform the performance of MoAA.

4. The experiments in this paper are all based on the small models of 8/9B. Whether this method can scale to larger models and whether it can lead to significant improvements in larger models is questionable, which may limit the broader impact of this work.

5. Although MoA can enhance performance and reduce bias compared to a single model, it inevitably introduces higher costs for dataset construction. It is clearly unreasonable to demand that the costs of MoAA be lower than previous methods without using MoA. However, providing some comparisons in terms of dataset construction efficiency, such as the number of times LLMs are called or consumed tokens, would offer a more comprehensive view. Besides, the related work MoA has given a "Budget and Token Analysis" in Section 3.4.

6. The MoA part uses WizardLM-8x22B, Qwen2-72B-Instruct, Gemma-2-27B-it, and LLaMA-3.1-70B-Instruct as proposers and Qwen1.5-110B-Chat as aggregators. These are all stronger models than Llama-3.1-8B or Gemma-2-9B. It is common for training on data distilled from stronger models to improve performance. It would be even more compelling if the trained model could surpass the models corresponding to those data sources, rather than just fitting a better model. However, Table 1 does not compare the performance of MoAA with the most strong two models: WizardLM-8x22B and Qwen1.5-110B-Chat.

1.⁠ ⁠Limited Novelty: The paper primarily extends existing methodologies, such as the Mixture of Agents (MoA) framework, for generating synthetic data for supervised fine-tuning and preference optimization. Its contributions largely revolve around practical refinements and adaptations of known techniques. 2.⁠ ⁠Weak Baselines: Baselines only consist of open-source instruction-tuned models without comparing against stronger baselines such as [1,2]. 3.⁠ ⁠Limited Model Evaluation: Evaluation is limited to Llama 3.1 8B and Gemma 2 9B models. Expanding to a broader range of model sizes and architectures (Llama 3.2 1B and 3B, Qwen 2 0.5B and 1.5B) can provide a good insight into the generalizability of the method.

[1] META-REWARDING LANGUAGE MODELS: Self-Improving Alignment with LLM-as-a-Meta-Judge, https://arxiv.org/abs/2407.19594v2 [2] MAGPIE: ALIGNMENT DATA SYNTHESIS FROM SCRATCH BY PROMPTING ALIGNED LLMS WITH NOTHING. https://arxiv.org/abs/2406.08464v1

1. The novelty is limited as the MoA was previously proposed and not optimized in this work.
2. There is no ablation on the choice of mixed models.
3. While mixing models can enhance robustness, using this mix as a reward model seems to degrade performance on certain tasks, despite an improved average score driven largely by high safety ratings. However, the number of safety tasks in UC and UF is relatively low, so I'm curious about the difference between using MoA as the reward model versus directly using a strong LLM as the reward model in this context.
4. MoAA’s effectiveness should be validated across additional complex tasks, such as math, reasoning, and coding, to determine its generalizability.

(1) While the empirical results are compelling, the paper lacks a more in-depth theoretical analysis of the MoAA approach and its potential limitations. (2) Lack of comparisons with additional baselines: The evaluations could be strengthened by including comparisons with a wider range of baselines, such as other model ensemble methods or preference optimization techniques. (3) Potential scalability concerns: The paper does not discuss the computational and memory requirements of the MoA framework, which may be a practical concern when scaling to larger models or datasets.