Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

Thanks a lot for reading our paper and for your insightful comments.

Q1: Technical contribution.
A1: Please refer to our general response G1 for an overview of the technical contributions. Additionally, we would like to clarify that our proposed method differs from self-consistency. To demonstrate this distinction, we provide additional results in Appendix C.2, comparing the performance of Self-MoA, Mixed-MoA, and Universal Self-Consistency (USC) $1$ on the AlpacaEval 2.0 benchmark.

While the key difference between Mixed-MoA and Mixed-USC (or Self-MoA and Self-USC) lies in the type of prompting—MoA methods use MoA prompts, while USC methods use USC prompts—our results show that MoA approaches consistently outperform their USC counterparts. This suggests that instead of simply identifying the most consistent response, MoA and Self-MoA actively encourage the model to synthesize references, resulting in superior responses.

Q2: Introduction of Self-MoA.
A2: We have included a brief introduction to MoA in the Related Work section (Appendix A). We agree with the reviewer that providing a more detailed introduction to both MoA and Self-MoA would enhance the paper's readability. To address this, we will include a dedicated background section in the camera-ready version.

Q3: Clarification of linear regression.
A3: The primary objective of Section 3.1 is to compare the strength of the effects of quality and diversity. Linear regression is chosen as it provides a direct and intuitive statistical tool to quantify these effects. To account for potential nonlinearity, we also experimented with different quality measurements, including average, k-norm, and centered-1/k-norm, to ensure robustness in our analysis.

$1$ Chen, Xinyun, et al. "Universal self-consistency for large language model generation." arXiv preprint arXiv:2311.17311 (2023).

Thanks a lot for reading our paper and for your insightful comments.

Q1: Clarification of Section 3.2.
A1: We agree with the reviewer that addressing both "When does it matter?" and "When doesn't it matter?" is crucial. To explore these questions, we conducted a comprehensive study across various tasks, including AlpacaEval 2.0, MT-Bench (newly added), MMLU, CRUX, and MATH. Additionally, we evaluated a large proportion of possible proposer combinations for the last three tasks. The results from these single-task evaluations consistently demonstrate that Self-MoA outperforms Mixed-MoA when one model significantly outperforms others.

Building on this observation, we attempted to construct scenarios where Mixed-MoA could have an advantage:

1. Constructing a mixed task where no single model performs well across all subtasks.
2. Identifying two models with very similar performance on a specific task to see if combining them improves performance.

We implemented both scenarios in Section 3.2 (the second is hard to construct so we only provide one case study in the last paragraph). Even in such constructed settings, Self-MoA remains a strong baseline, while Mixed-MoA provides only marginal improvements. Furthermore, in Section 4, we observe that incorporating a second model can enhance Self-MoA-Seq performance when the first model reaches saturation.

Additionally, as noted in our discussion with Reviewer ajnS (see Tables 1 and 2 in our individual response to Reviewer ajnS), Mixed-MoA may be a more practical choice in certain scenarios, such as online settings where identifying the best model is difficult. For instance, Mixed-MoA shows better performance in specific categories of MT-Bench.

To enhance the clarity of Section 3.2, we will incorporate these discussions into the camera-ready version.

Q2: Clarification of Table 1.
A2: You are correct in interpreting the "Individual" column—it represents the evaluation of each model independently, without employing any form of MoA prompting. Both the 2-Layer (MoA-Lite) and 3-Layer Mixed-MoA configurations use the outputs of six individual models as proposers, with Qwen1.5-110B-Chat serving as the aggregator.

For the 3-Layer MoA, the forward passes are distributed as follows: six forward passes in the 1st layer (one for each proposer), six in the 2nd layer (each proposer generates again), and one in the 3rd layer (aggregator generates once), resulting in a total of 13 forward passes.

To enhance clarity, we have updated the naming conventions in Table 1 in the revised version of the paper.

Q3: Typos in Figure 2.
A3: Thanks for spotting the typo. We have fixed it in the updated version of the paper.

Thanks a lot for reading our paper and for your insightful comments.

Q1: Soundness of Table 3.
A1: Please refer to our general response G2.

Q2: Soundness of Table 1 and Table 2.
A2: We agree with the reviewer that “times of generations” could be a more precise metric than “forward pass.” However, we would like to clarify that the comparison between Self-MoA and MoA in Table 1 is indeed fair. Specifically, Self-MoA + WizardLM-2-8x22B and MoA-Lite (as named in $1$ ) share an identical inference pipeline. Both methods employ a two-layer framework: proposers generate responses in the first layer, while the aggregator synthesizes responses in the second layer.

As both approaches utilize Qwen1.5-110B-Chat as the aggregator, the sole distinction lies in the proposers. Self-MoA uses only WizardLM-2-8x22B, whereas MoA-Lite (2-layer MoA) relies on all six models listed in the “Individual” section of Table 1. Given that 2-layer Self-MoA significantly outperforms 2-layer MoA, it is reasonable to conclude that ensembling multiple outputs from a single model is more effective than ensembling outputs from multiple models.

To improve clarity, we have updated the naming in Table 1 for better readability in the revised version.

Building on the findings from Table 1, we conducted the experiments presented in Table 2 to further explore and demonstrate the potential of Self-MoA. In this set of experiments, we selected state-of-the-art models as proposers and aggregators and reported the corresponding results. As the reviewer correctly noted, the purpose of Table 2 is not to directly compare Self-MoA and Mixed-MoA, as that comparison has already been addressed in Table 1, as discussed earlier.

Q3: Comparison on MT-Bench.
A3: We agree with the reviewer that providing additional results on MT-Bench adds value to our work. Please refer to Table 8 in Appendix C.1 for a detailed comparison. As discussed in A2, a fair comparison is between 2-layer MoA and 2-layer Self-MoA, both using Qwen1.5-110B-Chat or GPT-4o as the aggregator. The results demonstrate that Self-MoA consistently outperforms its Mixed-MoA counterpart, with the best performance achieved when GPT-4o is used as the aggregator. Notably, this is accomplished with less computation compared to a 3-layer MoA configuration using GPT-4o.

While the performance improvements are relatively modest, as current single models already achieve high win rates on MT-Bench (as suggested in $1$ ), these results further validate the effectiveness and efficiency of the Self-MoA approach.

I thank the authors for providing additional MT Bench results. It does add more evidence that the approach works for chat applications.

However, my main concern about this paper stands: "In the end, the paper asks the overarching question "Is mixing different large language models beneficial?" and the paper's Table 3 essentially does show: yes it is, although the authors do not acknowledge this. The value of MoA lies in the ability to potentially construct a well rounded model from multiple different experts."

Regarding G2, while yes the average does correspond to average performance when getting a random task from some stream of queries equally weighted, "dddddd" is still the weakest model for MMLU and on the bottom end for MATH. "dddddd" loses in 2 of 3 tasks. This is true in the new variance normalized Table 9 results also. I do not think an overall average is sufficient to claim that MoA is not needed. This is not just a theoretical thing: in real production settings, the worst loss bucket can represent the most embarrassing losses. MoA offers a tool for addressing specific loss buckets via ensembling.

For example, I can come up with a simple example where Self-MoA would fail: three models, A only does coding and nothing else, B only does math and nothing else, C only answers biology questions and nothing else. In this case there is no good singular model choice for Self-MoA. Are there some real world settings where it is beneficial to select one model to sample from? Absolutely! The authors show this very well. However, this is not sufficient to claim the generality of Self-MoA as a method.

In addition, the "TaskBest" results (the strongest results) clearly proves that "mixing different large language models [is] beneficial" as it aggregates outputs from multiple LLMs. This essentially directly contradicts the abstract, the main thesis of the paper.

I think this paper has value to the community, and offers interesting insights, but as it stands the framing of the work tries to sell Self-MoA over MoA when it doesn't have to. In the process, this framing introduces claims that are unsound, and I cannot raise my score at this time. The paper instead could have served as a valuable analysis of MoA and potential improvements to MoA (e.g. when is it better to sample from a single model, TaskBest being some sort of Routed MoA, etc). I hope the authors consider this for future revisions (not necessarily ICLR).

Thanks a lot for reading our paper and for your insightful comments.

Q1: Technical novelty.
A1: Please refer to our general response G1.

Q2: The connection between Mixed-MoA and Self-MoA-Seq.
A2: As noted by the reviewer, the original MoA paper introduces a multi-layer structure to scale up computational capacity. However, Mixed-MoA, when applied without a sequential structure, remains constrained by context length limitations.

Additionally, our Self-MoA-Seq experiments incorporate elements inspired by Mixed-MoA. Specifically, we observed that introducing a second model of similar quality (e.g., Llama-3.1-8B-Instruct) can enhance performance after the first model reaches saturation. Conversely, adding a second model of lower quality (e.g., DeepSeek-Coder-V2-Lite-Instruct) degrades performance.

Q3: Experiments with GPT-4o.
A3: We appreciate the reviewer’s suggestion to include GPT-4o in our experiments, as it further strengthens our findings. Accordingly, we report results using GPT-4o as the aggregator on the AlpacaEval 2.0 and MT-Bench benchmarks. The latter is newly added in Appendix C.1, as suggested by Reviewer ajnS.

On AlpacaEval 2.0, GPT-4o as the aggregator provides a 4% performance gain for Mixed-MoA. However, Mixed-MoA with GPT-4o still falls short compared to Self-MoA, regardless of whether GPT-4o or another model is used as the aggregator or proposer. On MT-Bench, GPT-4o proves to be a stronger aggregator than the default Qwen1.5-110B-Chat, improving the average score by 0.3–0.4 for both Mixed-MoA and Self-MoA. As a result, Self-MoA with GPT-4o as the aggregator achieves the best performance overall.

These additional results highlight that GPT-4o is a more powerful aggregator than smaller models for both Mixed-MoA and Self-MoA. More importantly, they further demonstrate the consistent effectiveness of Self-MoA compared to Mixed-MoA.

Table 1: GPT-4o results on AlpacaEval 2.0.

Table 2: GPT-4o results on MT-Bench.

Thank you for your prompt and insightful feedback!

For "2-Layer Self-MoA w/ GPT-4o + WizardLM-2-8x22B," we used WizardLM-2-8x22B as the proposer and GPT-4o as the aggregator. Similarly, for "2-Layer MoA w/ GPT-4o," we employed six individual models from the Mixed-MoA setting as proposers, with GPT-4o serving as the aggregator.

Following your suggestion, we conducted experiments to evaluate the setting where GPT-4o is used as both the proposer and the aggregator. The updated results are presented below. Interestingly, in two tasks, "2-Layer Self-MoA w/ GPT-4o + GPT-4o" did not achieve the best performance. This suggests that GPT-4o may not serve as an optimal aggregator for its own outputs. A similar observation was also noted in A7, where WizardLM-2-8x22B was not the most effective aggregator for its own outputs. Selecting the most effective aggregator for a given set of proposers remains an intriguing direction for future research.

Table 1: GPT-4o results on AlpacaEval 2.0.

Table 2: GPT-4o results on MT-Bench.

We hope our reply addresses all your questions and will be happy to have further discussions if you have additional questions or comments.

There are various changes in the paper, can you please highlight (using a different color) what are changed in the paper, compared to the version that was reviewed. e.g. Table 1, "Self-MoA + WizardLM-2-8x22B" seems to be changed to "2-Layer Self-MoA + WizardLM-2-8x22B". This is fine, but I am not sure if there are any other changes.

Thank you for your valuable feedback and suggestion. We updated Table 1's naming in the main paper to improve clarity as suggested by reviewers. All other changes have been highlighted in red in the Appendix. Additionally, we have now specifically highlighted the change to Table 1 in the updated version for improved transparency. We appreciate your input and hope this addresses your concern.