Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

We appreciate your thoughtful feedback and for recognizing the contributions of our work. Your comments and suggestions have helped us identify areas for further clarification and improvement. Below, we address each of your concerns in detail.

---

1. Evaluation on Additional Tasks (StarCraft II, Hanabi, etc.)

Thank you for your suggestion regarding the inclusion of additional tasks, such as StarCraft II or Hanabi. We want to clarify that our paper does not evaluate on only two tasks, but on two broad categories of tasks: information exchange and multi-agent debate. Within each category, we evaluate across multiple specific tasks. In total, we conduct experiments on 8 tasks, with the MMLU benchmark even containing 57 sub-tasks. We argue that this constitutes a comprehensive and varied evaluation. This number and diversity of tasks in our evaluation are already extensive compared to many other works in LLM-based MAS.

While we agree that tasks like StarCraft II and Hanabi would be valuable to explore, these are relatively less studied in the context of LLM-based MAS. We argue that the benchmarks we’ve chosen are sufficient for demonstrating the effectiveness of Optima.

We appreciate your suggestion and agree that these tasks could serve as interesting directions for future work, and we are currently considering them for follow-up experiments.

---

2. Number of Agents in Experiments

We understand your concern regarding the limited number of agents in our experiments. As the first work to optimize multi-agent communication via iterative training, we chose to focus on the cleanest scenario involving two agents. This allowed us to focus on the core challenges of optimizing communication efficiency without introducing the additional complexities that would arise with more agents, such as communication topology, speaking order, and coordination between multiple agents.

Still, we recognize the importance of demonstrating the scalability of Optima to more agents. As such, we have conducted additional experiments:

1. We directly applied the model trained in the 2-agent scenario to a 3-agent communication setting:

Although the performance slightly decreased compared to the 2-agent setting, this is expected due to the increased complexity of information exchange in multi-agent scenarios, where more agents require more sophisticated coordination. This result demonstrates the generalizability of the trained models to scenarios with additional agents.

1. Since we assign one agent as solver, and the other as critic in multi-agent debate task, it is hard to generalize the already trained model to more agent scenarios. Therefore, we additionally train the models on both types of tasks in the most basic 3-agent scenario, where agents speak in order. We show that Optima still offers satisfying performance:

As anticipated, in the information exchange task, the 3-agent setting generally performs worse than the 2-agent setting due to the more distributed nature of the information, but Optima still offers performance gain against baselines. In the debate task, Optima also continues to provide a performance boost while significantly reducing token usage. As noted earlier, the 3-agent setting introduces additional complexities, such as speaking order and communication topology, which need to be carefully managed for optimal performance. We believe that with improved configuration and adjustments in the 3-agent setup, Optima can still be effectively utilized to train models in these scenarios.

We take your point seriously, and we believe that these additional experiments will help demonstrate the adaptability and scalability of Optima to scenarios involving more agents.

We thank you for your thoughtful review of our work. We appreciate your positive remarks regarding the clarity and detail of our framework, as well as the promising results we presented. However, we respectfully disagree with some of the assessments, particularly regarding the novelty of our work, the fairness of our comparisons, and the scalability of our approach. We would like to address your concerns in detail.

---

1. Unfair Comparison: Token Consumption During Inference and Post-Training

We understand your concern about comparing token consumption during inference while ignoring the tokens consumed during post-training. However, our major goal in this work is to demonstrate the efficiency of the final system during inference, which is the key metric for real-world deployment. Since our objective is to optimize token efficiency during inference, we believe that focusing on inference tokens is entirely reasonable.

While we agree that the training cost is important, it is not relevant to our claim regarding inference efficiency. To clarify, the post-training process consumes a certain amount of tokens, but this cost is negligible compared to pre-training tokens. More importantly, the improvements in inference efficiency outweigh the costs incurred during our Optima post-training. Specifically:

• For iSFT, the training (including initialization and 5 iterations) can be completed in under 12 hours using 8 A100 GPUs for almost all the tasks.
• For iDPO, due to the MCTS component, the training takes longer, but can still be completed within 24 hours on 8 A100 GPUs for almost all the tasks.

These timeframes, given the substantial gains in inference efficiency, are highly acceptable and reflect a reasonable tradeoff. Requiring the inclusion of post-training tokens in the efficiency comparison is unreasonable, as it diverges from the usual practice in the field, where post-training costs are typically not included in such comparisons. This is especially true when the focus is on optimizing inference efficiency.

Furthermore, if we were to include training tokens when comparing with baselines, then no training-based method would appear effective, as the number of tokens consumed during post-training is generally much larger than the tokens used during inference. If the evaluation is restricted to inference-only techniques without additional training tokens, such as prompt engineering, the scope for meaningful improvements in efficiency would be highly constrained. Such techniques, while lightweight, tend to provide limited improvements and would restrict the allowable methods to superficial approaches that fail to address the deeper challenges of optimizing efficiency.

---

2. Lack of Novelty

We respectfully disagree with the assessment that our work lacks novelty. While iSFT and iDPO have been explored in previous works, our contribution lies in the novel problem focus, the novel findings, and the novel application of these techniques. Specifically:

• Novel Problem Focus: Our work addresses the critical but under-explored issue of communication efficiency in LLM-based MAS. This challenge, which has been long overlooked, is central to the successful deployment of multi-agent systems, especially when operating at scale.
• Novel Findings: We demonstrate that our approach leads to systems that improve inference scaling laws. This finding extends beyond the original goals of iSFT and iDPO (focusing only on effectiveness), opening new research directions regarding the optimization of LLM-based MAS, and even more broadly, the LLM systems.
• Novel Application: Our approach applies iSFT and iDPO to multi-agent systems, which requires adaptations to the reward design, data construction, and filtering processes. These modifications are crucial for ensuring that the techniques are effective in MAS scenarios, making our approach both innovative and impactful.

We believe that these contributions, combined with our comprehensive empirical evaluation, position our work as a significant advancement in the field, rather than an incremental improvement.

Thank you for your follow-up response and for acknowledging the importance of inference efficiency and the improvements achieved by Optima. We appreciate this ongoing discussion and would like to address your remaining concerns. Before proceeding, we believe it would be helpful to review and clarify the key points from both sides.

---

1. Clarifying Our Points of Divergence

We believe we have reached consensus on the key achievements of Optima: it produces more effective and efficient models, with significant gains in inference efficiency. However, our perspectives diverge on whether training tokens should be included in efficiency comparisons:

• Our position: Since the goal of Optima is to optimize inference efficiency, it is reasonable to compare Optima-trained models with plain models, focusing solely on inference metrics. This approach aligns with standard practices in the field, such as comparing RLHF-trained models to SFT models based on chat ability, without including training tokens in the comparison.
• Your concern: Additional training introduces new costs, so training tokens should be included, or inference-time MCTS methods should be incorporated as baselines for fairness.

---

2. Addressing the Inclusion of Training Tokens

We understand your concern about including training tokens, but we believe this perspective may overlook key principles of evaluating training-based methods:

1. Optima's goal is to create inference-efficient models. Just as RLHF comparisons focus on chat ability without accounting for training tokens, we evaluate inference efficiency because it reflects our method’s primary target. Including training tokens dilutes the focus on this core objective.
2. It is a well-established practice to exclude training tokens in comparisons. This norm exists because post-training methods are primarily evaluated based on their ability to inject desired properties into the model. As long as the required resources are practical—e.g., Optima training takes under 12–24 hours on 8 A100 GPUs—these methods are considered effective. We encourage you to reflect on why this practice is widely accepted and how it aligns with Optima’s focus.
3. As we have said, if training tokens are included, no training-based method would appear efficient. For example, training-based methods like RLHF would always seem inferior in token usage compared to lightweight inference-only techniques like prompt engineering.

---

3. Addressing the Suggestion to Include MCTS-Based Methods

Including MCTS-based methods as inference-time baselines is inappropriate for several reasons:

• MCTS is inherently inefficient for inference, as it relies on generating multiple trajectories for each response. This contradicts Optima’s goal of improving inference efficiency.
• MCTS requires access to the correct answer or a reliable reward signal to evaluate trajectories, which is unavailable during inference. Therefore, incorporating MCTS-based methods would not align with the practical objectives of our work.

---

4. Training Tokens in Context of Ablation Studies

Our ablation study can be seen as a comparison between Optima method and plain iSFT and iDPO, which also fine-tune the models. The findings in our ablation study show that removing token-efficiency-related reward terms from Optima leads to models with significantly higher token usage or inferior performance.

Thus, our ablation studies provide evidence that Optima acts as an effective tradeoff between the effectiveness and efficiency when compared with other fine-tuning methods such as iSFT and iDPO.

---

5. The Complementarity of Training and Inference Techniques

We want to emphasize that training-based methods like Optima are not in conflict with inference-time techniques such as self-consistency or MCTS. Instead, they are complementary:

• As shown in Table 1 and Figure 3 of our paper, models trained with Optima can still benefit from inference-time techniques, further improving performance and efficiency.
• Combining training-based and inference-time methods allows for better results than either approach alone.

We respectfully disagree with the notion that these techniques should be viewed as mutually exclusive.

Thank you for your thorough review and constructive feedback. We appreciate your recognition of our paper's clear structure and methodology. We would like to address your concerns and clarify several important points:

---

1. Novelty and Contribution

We respectfully disagree with the assessment of limited novelty. While iSFT and iDPO have indeed been explored previously, our work's novelty lies in:

• Novel Problem Focus: We identify and address the critical but long-ignored problem of communication efficiency in LLM-based MAS. To our knowledge, we are the first to systematically study and optimize this aspect.
• Novel Findings: We demonstrate that our approach yields systems with better inference scaling laws, a finding that extends beyond the original goals of these training algorithms and opens new research directions.
• Novel Application: The application of these algorithms to multi-agent scenarios represents an innovation. The challenges of maintaining effective communication while optimizing for efficiency require adaptation of these techniques, such as reward designing, reward balancing, data construction process and data filtering.

The significance of our work lies not just in the methods used, but in identifying and successfully addressing a crucial challenge in MAS deployment.

---

2. Experimental Comparisons

We appreciate your concern about comparison fairness. Our comparisons aim to demonstrate that Optima can successfully inject the ability to be both concise and accurate into models. To achieve this, we compared trained and untrained models to show that the framework performs as intended. However, we take your point seriously, and conducted additional experiments by training the models on the MATH and GSM8k training set to evaluate the impact of training on the training data, as they have human annotated CoT-like data. Note that this comparison may be unfair to us, as we only use model-generated answer to do self-improve, while training directly on the annotated CoT data benefit from human annotation. We do grid search over the hyperparameters:

• lr: [1e-6, 5e-6, 1e-5]
• epoch: 1, 2, 3
• batch: 128

We found the results with lr=1e-6, epoch=2, bsz=128 for MATH, and lr=1e-6, epoch=3, bsz=128 for GSM8k are the best:

Interestingly, we did not observe a significant difference between the performance of trained and untrained models. This is possibly because Meta has already extensively trained Llama on high-quality math data. This suggests that the observed improvements may not be solely attributable to training on the training set. We appreciate your feedback, as it helps us further clarify the robustness of Optima's contributions.

---

3. Lack of Multi-Agent Aspects

Thank you for raising this point. Still, we believe our work is fundamentally grounded in multi-agent scenarios:

• Our framework specifically targets multi-agent communication efficiency and effectiveness
• Both evaluation settings (information exchange and multi-agent debate) represent typical multi-agent scenarios
• The reward function and training procedure are designed specifically for multi-agent interaction

While our methods have potential applications beyond MAS, we view this generalizability as a strength rather than a limitation.

---

4. Technical Writing Corrections

We appreciate your careful attention to detail. We will revise the abstract to avoid mischaracterizing SFT and DPO as RL methods, and add explicit formula for the language modeling loss.

4. Scalability

To assess the scalability of Optima, we directly transferred the model trained under the 2-agent scenario to a 3-agent setting. The results are summarized below:

Although the performance slightly decreased compared to the 2-agent setting, this is expected due to the increased complexity of information exchange in multi-agent scenarios, where more agents require more sophisticated coordination. This result demonstrates the generalizability of the trained models to scenarios with additional agents.

Since we assign one agent as solver, and the other as critic in multi-agent debate task, it is hard to generalize the already trained model to more agent scenarios. Therefore, we additionally train the models on both types of tasks in the most basic 3-agent scenario, where agents speak in order. We show that Optima still offers satisfying performance:

As anticipated, in the information exchange task, the 3-agent setting generally performs worse than the 2-agent setting due to the more distributed nature of the information, but Optima still offers performance gain against baselines. In the debate task, Optima also continues to provide a performance boost while significantly reducing token usage. As noted earlier, the 3-agent setting introduces additional complexities, such as speaking order and communication topology, which need to be carefully managed for optimal performance. We believe that with improved configuration and adjustments in the 3-agent setup, Optima can still be effectively utilized to train models in these scenarios.

---

5. Data Integrity Verification

To address concerns about potential test data leakage:

• We strictly adhered to official train-test splits for all datasets
• We additionally conduct verification by checking for any overlap between training and test questions, and we find no instances of test data appearing in training sets

This confirms the validity of our reported performance improvements. We will add it to the appendix. We appreciate your attention to this important methodological consideration.

---

Our Results' Broader Impact

Beyond addressing specific concerns, we'd like to again argue that Optima represents an important advancement in LLM-based MAS and LLM systems more broadly:

• We achieve up to 90% reduction in token usage while maintaining or improving performance, opening new possibilities for practical deployment. We are the first to highlight the importance of simultaneously optimizing effectiveness and efficiency, and propose an effective training framework for this purpose.
• The improved inference scaling laws suggest better compute utilization, making multi-agent systems more viable for real-world applications.
• Our successful application of iterated self-improvement opens promising new research directions about training-based self-improvement of LLM MAS.

We believe these contributions, combined with our thorough empirical validation and careful attention to methodological concerns, make Optima a valuable addition to the field.

---

We welcome any requests for clarification or additional questions. If our responses have adequately addressed your concerns, we kindly ask you to consider raising the score. We truly appreciate your careful review.

---

[1] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

Thank you for your thoughtful review and for recognizing the timeliness and significance of the challenge our work addresses. We also appreciate your positive assessment of our extensive empirical evaluation. We are pleased to address your specific concerns:

---

1. Theoretical Explanation of Reward Function

Our reward function design follows the empirical approach common in traditional RL literature (e.g., +1 or +10 for successful trajectories and -1 or 0 for failures). While primarily empirically-driven, our design still follows clear principled considerations:

• The token efficiency term ($R_\text{token}$) directly incentivizes concise communication, while the language modeling loss term ($R_\text{loss}$) promotes natural and interpretable exchanges. High rewards indicate that a model achieves accuracy (via $R_\text{task}$), conciseness (via $R_\text{token}$), and readability (via $R_\text{loss}$), which is desired for our final goal.
• Our ablation studies (Table 3) demonstrate each component's crucial role - removing either term significantly degrades either efficiency or performance.

This empirically-validated approach aligns with successful practices in RL applications where reward shaping is often guided by desired behaviors rather than theoretical guarantees.

---

2. Results on Other Models

We have conducted additional experiments using Llama 3.2 3B on several benchmark to address your concern about the performance of Optima on different models. The results, as shown below, demonstrate Optima’s consistent advantages in both effectiveness and efficiency compared to the baselines:

These results confirm that Optima consistently achieves superior performance with significant token efficiency, even on a different model. We will add the experiments and relevant analysis in the appendix, and mention it in the main text.

---

3. Safety and Alignment Considerations

While safety alignment isn't the primary focus of Optima (which targets task effectiveness and efficiency), we take your concern seriously. We follow similar approach proposed in LLM-as-judge [1] and conduct additional analysis using GPT-4o, asking it to judge whether the communication trajectory contains unsafe content. Given the high cost of evaluating all trajectories, we follow the principles of sample size determination for proportions to estimate the proportion of unsafe content with 99% confidence. Specifically, for each dataset, we sample $n\times \sqrt{\frac{N-n}{N-1}}$ samples, where $n=\frac{2.576^2\cdot 0.5^2}{(1-0.99)^2}$ and $N$ is the dataset size. This formula ensures that the calculated error rate on the sampled trajectories falls within the 99% confidence interval of the error rate for the entire dataset. After conducting this analysis, we found no unsafe content in any of the sampled communication trajectories.

We will add the analysis and the prompt to our appendix to improve our work. Thank you for your advice.