We sincerely thank Reviewer 3bEo for the valuable feedback. We have addressed all the comments you provided. Please find our point-by-point responses below. Any further comments and discussions are welcome! We apologize for the lack of clarity and the missing details in the initial submission. We have added extensive implementation details to enhance the understanding of our work. All modifications are highlighted in blue in the updated manuscript.

Q1. Scalability of Coordination Mechanism

"Scalability of the Coordination Mechanism: Could the authors clarify how EcoNash's coordination structure scales as the number of agents grows, particularly with a single Coordinator LLM? Are there any plans or existing methods for adapting the framework to include multiple coordinators or a more decentralized approach to avoid potential bottlenecks?"

Reply:

Thank you for your response. We are actively researching this area. As shown in Sec 4.4, Figure 2, and Figure 3, our findings using LLaMA 3.1 70B indicate that performance declines when a single Coordinator LLM handles more than six Execution LLMs. However, additional experiments with LLaMA 3.1 405B suggest that a stronger Coordinator LLM can effectively manage more Execution LLMs, with performance only starting to decline when coordinating more than nine LLaMA 3.1 70B models.

This performance difference is closely related to the capability of the Coordinator LLM itself. Beyond forming strategies and formats based on the given question (as demonstrated in Appendix D.2, where LLaMA 3.1 70B generates high-quality strategies and formats), the Coordinator LLM must also evaluate the responses from the Execution LLMs, generate rewards, and update parameters based on the reward-derived loss to approximate a BNE. In Table 6, we provide results using random rewards for evaluation, which show a significant drop in performance.

Therefore, we believe introducing an External Reward Model could be a reasonable solution, despite the additional computational overhead. Additionally, employing a more powerful model as the Coordinator LLM or fine-tuning the evaluation capability of the Coordinator LLM through specific methods would also be promising approaches.

Q2. Clarification on Assumptions in Convergence Proofs

"Clarification on Assumptions in Convergence Proofs: The theoretical results on convergence to BNE are valuable. However, could the authors clarify the assumptions made regarding the reward structures, learning rates, and agent behavior during convergence? Additionally, how sensitive are the convergence results to these assumptions?"

Reply:

We have revised Appendices A.3, B.2, and B.3 to further clarify our BNE convergence and provide additional details regarding the theoretical underpinnings. Specifically, Appendices B.2 and B.3 explain why EcoNash achieves a superior regret bound compared to MAD. The core assumptions behind these results are Assumption 5 (Q-function Estimation Error) and Assumption 6 (Policy Suboptimality), which are based on the following conditions:

1. Convex and compact policy space.

2. Lipschitz continuous policy functions.

3. Bounded and continuous Q-functions.

4. Proper learning rate scheduling.

Regarding learning rate sensitivity, we emphasize that excessively high learning rates can destabilize our on-policy learning process. Additionally, since we use on-policy learning within the datasets, we must ensure that our replay buffer is not too large to prevent instability.

In contrast, Assumption 7 highlights that in multi-agent systems with the following conditions:

• Non-cooperative zero-sum or constant-sum game structure.

• Strategic uncertainty due to incomplete information.

• No mechanism for joint strategy optimization.

• Competitive reward structure.

Policy suboptimality remains bounded from below. This necessitates the introduction of a coordinator to guide the system and the design of a bounded and continuous reward space to ensure effective coordination.

Q6: Implementation of Self-Evaluation Rewards

"The self-evaluation reward mechanism is intriguing for improving reasoning consistency. Could the authors elaborate on how this reward is designed and applied across different tasks? Are there any specific criteria used for self-evaluation that may vary by task type?"

Reply:

Thank you for your insightful question. Below, we explain how the self-evaluation reward mechanism is designed and adapted to different tasks, focusing on mathematical reasoning and planning benchmarks.

Task-Specific Criteria for Self-Evaluation

1. Mathematical Reasoning Tasks (e.g., GSM8K, SVAMP, MATH, GSM-Hard)

Mathematical reasoning tasks emphasize correctness and step-by-step reasoning clarity, given the structured nature of the problems.

Evaluation Criteria:

• Correctness: The Task-Specific Reward ($r_i^{\text{TS}}$) evaluates how accurate the final answer is relative to the ground truth.
• Reasoning Steps: Intermediate steps are assessed for logical consistency and clarity, with partial credit given for accurate reasoning even if the final answer is incorrect.

Example: For the problem $2x + 3 = 7$, the reward would consider:

1. Correct isolation of $x$ (i.e., $x = 2$).
2. Clear explanation of steps, such as subtracting 3 and dividing by 2.

This ensures the model learns to prioritize both process clarity and solution accuracy.

2. Planning Tasks (e.g., TravelPlanner, Strategy QA)

Planning tasks require agents to generate actionable, coherent plans under various constraints.

Evaluation Criteria:

• Relevance: Responses are evaluated for alignment with key task objectives, such as adhering to user-defined constraints (e.g., budget, time).
• Coherence: Plans must exhibit logical structure, with each step building naturally on the previous ones.

Example: For TravelPlanner, where agents create travel itineraries under budget constraints:

1. The evaluation rewards agents for satisfying constraints while optimizing time and cost.
2. Penalties are applied for infeasible or irrelevant actions.

This approach ensures that agents focus on generating practical and effective solutions.

Dynamic Reward Adaptation

To adapt to different tasks, we dynamically adjust the weights ($\alpha_1, \alpha_2, \alpha_3$) of the reward components—Action Likelihood, Task-Specific Reward, and Collaborative Contribution Reward. This ensures that the reward system remains flexible:

• Mathematical Tasks: Focus on correctness and reasoning clarity ($\alpha_1, \alpha_2$).
• Planning Tasks: Emphasize relevance, coherence, and adherence to constraints ($\alpha_2, \alpha_3$).

This dynamic adjustment ensures agents optimize for the most relevant criteria for each task type.

We hope this explanation addresses your question and are happy to provide further clarification if needed.

Dear Reviewer 3bEo,

We sincerely thank you for your efforts in reviewing our work and for your great support! We also greatly appreciate your insightful questions and hope that our responses have helped to clarify them.

Please let us know if you need any further information or if there are additional points you would like to discuss with us. Thank you very much!

Best regards,

Authors of #14101

We sincerely thank Reviewer 2o1n for the valuable feedback. We have addressed all the comments you provided. Please find our point-by-point responses below. Any further comments and discussions are welcome! We apologize for the lack of clarity and the missing details in the initial submission. We have added extensive implementation details to enhance the understanding of our work. All modifications are highlighted in blue in the updated manuscript.

Q1: Accuracy of BNE in the MA-LLM Framework (Section 3.1.1)

"The authors can present the paper's content more clearly and make it easier to follow. I found a lot of crucial details not adequately explained in theoretical results, experimental setup and training, and general examples/intuition that would make the paper more appealing to the multi-agent reasoning community. See questions below (1-6, 8)."

Reply:

Thank you for your valuable comments. We appreciate the opportunity to clarify and improve our manuscript. Let me address each point:

1. "3.1.1: What is a type function?"

Reply:

In our revised framework, we have moved away from the type function formulation to a more intuitive belief-based representation. Instead of using type functions, we directly model agent states through belief states $\mathbf{b}_i \in \mathbb{R}^d$ and belief networks $B_i(\mathbf{\tau}_i, O_i; \theta_i^B)$. This approach better captures the dynamic nature of LLM interactions while maintaining mathematical rigor. The detailed revision can be referred to Sec 3.1.1 which is highlight by blue.

2. "Problem with the notation: shouldn’t tuple of $\theta$s and actions be ordered, so $\theta_i$ and $\theta_{-i}$ does not account for the orderings? The authors should clarify that."

Reply:

You raise a good point about the ordering issue. In our revised framework, we've eliminated the $\theta_i$ and $\theta_{-i}$ notation to avoid ambiguity. Instead, we use belief network parameters $\theta_i^B$ for individual agents and the group-level representation $\mathbf{E}$ generated by $f_e(\{\mathbf{b}_i\}_{i=1}^N; \theta_e)$ to capture inter-agent relationships. This notation better reflects the hierarchical structure of our framework.

3. "Appendix A.3: $H$ is entropy and $I$ is mutual information—if so, it should be stated or defined?"

Reply:

Thank you for your comment. We have revised the entire Appendix A.3 section, providing complete explanations for all the assumptions and their relation to EcoNash. We have explicitly defined $H$ as entropy and $I$ as mutual information to enhance clarity. Specifically, here is the definition of of Assumption 3:

Definition: Belief Entropy

For a given time $t$, the belief entropy $H_t$ is defined as the Shannon entropy of the aggregated belief embeddings:

where $B_i$ represents the belief network of agent $i$.

Assumption: Game Regularity

There exists a constant $\eta > 0$ such that for any $t_1 < t_2$, if the entropy difference satisfies $H_{t_1} - H_{t_2} \leq \log 2$, then the following holds:

where:

• $I(\cdot; \cdot \mid \cdot)$ represents the conditional mutual information,
• $\theta_i^B$ are the parameters of the belief network for agent $i$,
• $\xi(\mathbf{e}_i, \mathbf{E})$ denotes the coordination outcome based on the agent's prompt embeddings $\mathbf{e}_i$ and the group-level representation $\mathbf{E}$,
• $D_t$ refers to the dataset or information set available at time $t$.

This assumption ensures stability in how the belief network parameters influence coordination outcomes over time.

Dear Reviewer 2o1n,

We sincerely thank you for your efforts in reviewing our work and for your great support! We also greatly appreciate your insightful questions and hope that our responses have helped to clarify them.

Please let us know if you need any further information or if there are additional points you would like to discuss with us. Thank you very much!

Best regards,

Authors of #14101

Q2: Code provided through a URL

"The provided code (through a URL) is very minimal/barebones. It includes data files and code for Game of 24—which is not even reported in the actual paper."

Reply:

Thank you for pointing this out. We have refined and submitted our complete codebase, along with detailed explanations in the README file as https://anonymous.4open.science/status/EcoNash-867A. We will continue to maintain and fully open-source the code.

Regarding the Implementation of the Game of 24:

Initially, we conducted experiments on the Game of 24 as a mathematical planning task. However, we observed that tree search-based methods (e.g., ToT, MCTS-Rollout, TM-LLS) consistently outperformed methods like CoT and self-consistency on this task. This created an imbalance in comparison, as the evaluation unfairly favored certain approaches rather than providing a well-rounded assessment across diverse tasks.

Additionally, using the Game of 24 for comparing token consumption is not entirely reasonable, as search depth becomes a significant factor, as highlighted in https://arxiv.org/pdf/2309.17179:

"We argue that the Path@1/Path@N metric may not be reasonable. We also include the number of computations used in Path@1 generation (average number of tokens in sentence-level and average number of forward computations in token-level for solving a single problem). We refer readers to the second row of Fig. 2 for the Path@N result, with token/forward number as the x-axis. TS-LLM variants consume much more computation than CoT, making the comparison unfair.”

Considering these observations, we decided to focus on a more complex and practical task—the travel planner—as our planning task benchmark, as shown in Table 2. This benchmark provides a broader and more realistic evaluation scope, aligning better with our objective of testing generalizable planning methods.

Q3: Potential unfair comparison

"The comparison is between EcoNash (a method that involves fine-tuning the model) and other methods (RAP, ToT, rStar, etc.)—essentially comparing a fine-tuned model with prompting methods. The higher performance is not entirely surprising."

Reply:

Thank you for pointing this out. However, we would like to clarify that the EcoNash framework does not fine-tune the LLMs directly. Instead, we focus on adjusting the prompt embeddings and training the belief networks to modify the outputs of the LLMs. Through the coordinator's guidance, we aim to steer the multi-agent LLM system towards a Bayesian Nash Equilibrium (BNE) in an incomplete information game setting.

To address your concern about unfair comparison, we have added other baselines that involve learned action-value functions (arXiv:2309.17179 and arXiv:2309.15028). We have included the updated results in Table 1 of the revised manuscript. Comparing with these baselines enhances the persuasiveness of our method by demonstrating its advantages over approaches that also learn action-value functions.

Analysis

1. GSM8K: EcoNash achieves a 2.7% higher average score compared to TS-LLM and a 4.9% improvement over PPO-MCTS. This demonstrates its superior performance in solving standard mathematical problems.

2. GSM-Hard: EcoNash achieves a 10.9% improvement over TS-LLM and a 13.5% improvement over PPO-MCTS. This highlights its significant advantage in tackling complex mathematical challenges.

3. SVAMP: EcoNash outperforms TS-LLM by 3.0% and PPO-MCTS by 3.0%, showcasing its efficiency in arithmetic and mathematical reasoning tasks.

4. StrategyQA: EcoNash achieves a 4.2% higher average score compared to TS-LLM and a 5.1% improvement over PPO-MCTS, demonstrating its superior understanding and decision-making capabilities in strategic question-answering tasks.

5. MATH: EcoNash outperforms TS-LLM by 4.9% and PPO-MCTS by 8.2%, highlighting its strength in addressing advanced mathematical problems.

We sincerely thank Reviewer PYcK5 for the valuable feedback. We have addressed all the comments you provided. Please find our point-by-point responses below. Any further comments and discussions are welcome!

Q1: Lack of key details and missing training information

1. "Currently, the paper lacks quite a few key details to properly understand the work."

2. "Training details are almost completely missing."

Reply:

We apologize for the lack of clarity and the missing details in the initial submission. We have added extensive implementation details to enhance the understanding of our work. All modifications are highlighted in blue in the updated manuscript. Here is a summary of the changes:

1. Section 3.1: Revised the description of the BNE in the multi-agent LLM framework for better clarity, specifically elaborating on how we quantify the effectiveness of different strategies.

2. Section 3.2: Updated Appendix A.3 to provide further explanation on the basis and reasoning behind Lemma 1. Refined Appendices B.2 and B.3 to detail the computation of the Bayesian regret bounds for EcoNash and MAD based on Lemma 1.

3. Section 3.3.1: Updated Figure 1 to more clearly illustrate the specific procedures in the inference and optimization phases.

4. Section 3.3.2 (Reward Setting): Added more detailed descriptions of the reward setting in Appendix B.4 to supplement the main text.

5. Section 3.3.2 (Individual Network / Centralized Mixing Network): Made systematic modifications to both the individual belief networks and the centralized mixing network to ensure theoretical completeness and learning stability.

6. Section 3.3.2 (Belief Encoder): Added a detailed definition of the belief encoder and its optimization details to clarify how it aggregates group-level information.

7. Section 3.3.2 (Early Stopping): Provided detailed criteria for early stopping to ensure efficient optimization and convergence.

8. Appendix B.5: Included a comprehensive table of hyperparameters corresponding to Section 3.3.2 to enhance reproducibility.

9. Appendix D.2: Added more strategy examples generated by the coordinator to provide additional insights.

10. Table 1: Introduced additional performance comparisons with baselines that use learned action-value functions, specifically:

    • [1] Feng, Xidong, et al. "Alphazero-like tree-search can guide large language model decoding and training." arXiv preprint arXiv:2309.17179 (2023).
    • [2] Liu, Jiacheng, et al. "Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding." First Conference on Language Modeling. 2024.

Optimization Procedure:

1. Each execution LLM maintains its belief state $\mathbf{b}_i \in \mathbb{R}^d$ and receives observations $O_i = [e_t, e_s, \mathbf{b}_i]^\top$, where $e_t$ encodes the task, and $e_s$ represents the coordinator's strategy.

2. The belief network is defined as $B_i(\mathbf{\tau}_i, O_i; \theta_i^B)$, which updates each agent's state based on its history $\mathbf{\tau}_i$ and current observation, generating prompt embeddings $\mathbf{e}_i$.

3. The belief network outputs:

   • A set of prompt embeddings $\mathbf{e}_i^t$ for $i = 1, \dots, N$,
   • A set of individual Q-values $Q_i^t$ for $i = 1, \dots, N$.

4. The belief encoder aggregates the belief states from all agents to generate a group-level representation using multi-head attention with $H$ attention heads to capture inter-agent relationships.

5. The Centralized Mixing Network is designed to coordinate belief information from execution LLMs, receiving:

   • A set of prompt embeddings $\mathbf{e}_i^t$ for $i = 1, \dots, N$,
   • The group-level representation $\mathbf{E}^t$,
   • A set of individual Q-values $Q_i^t$ for $i = 1, \dots, N$.

6. After achieving commitment, the mixing network computes:

   • The similarity difference (SD) loss,
   • The global value function $Q_{\text{tot}}^t$.

7. A combined loss is then used to update all parameters.

Dear Reviewer PYck,

We sincerely thank you for your efforts in reviewing our work and for your great support! We also greatly appreciate your insightful questions and hope that our responses have helped to clarify them.

Please let us know if you need any further information or if there are additional points you would like to discuss with us. Thank you very much!

Best regards,

Authors of #14101

Dear Reviewer PYck,

We deeply appreciate your detailed review and insightful comments. Your feedback has significantly improved the quality of our submission.

Revisions Made

In our rebuttal, we have comprehensively addressed your concerns and enhanced the manuscript's readability by providing additional explanations. Specifically, we have:

• Added detailed information on the implementation in Sections 3.1.1 and 3.3.2.
• Included proofs in Appendices A.3, B.2, and B.3.
• Elaborated on the reward settings in Appendix B.4.
• Detailed the dataset setup in Appendix B.5.
• Provided a table of hyperparameters in Appendix B.6.
• Expanded the formats and strategies generated by the coordinator in Section D.2.

We kindly request your feedback on our responses, especially as the deadline for submitting the revised PDF is approaching. Your further insights would be invaluable to ensure that our revisions meet the necessary standards.

Thank you once again for your time and constructive input, any additional comments and discussions are welcome!

Best regards,

Authors #14101

We sincerely thank Reviewer bum5 for the valuable feedback. We have addressed all the comments you provided. Please find our point-by-point responses below. Any further comments and discussions are welcome!

Q1: The connection between Section 3.2 and the following method

"I don't see a strong connection between Section 3.2 and the following method."

Reply:

Thank you for your valuable feedback. In Section 3.1, we define the Bayesian Nash Equilibrium (BNE) within the context of multi-agent LLMs, including the evaluation method and the proof of its existence. In Section 3.2, we build upon these definitions to introduce Lemma 1, which allows us to evaluate the performance of multi-agent LLM systems through Bayesian regret analysis. Our analysis demonstrates that EcoNash achieves a sublinear regret bound, in contrast to the linear regret of existing multi-agent debate methods. In the subsequent Section 3.3, we present the specific implementation of our architecture based on the assumptions satisfying Lemma 1 (as detailed in the Appendix). This implementation is then correlated with the results in Section 4 (Experiments), providing a cohesive flow from theoretical foundations to practical application.

Q2: Presentation problem in Figure 1

"Figure 1 should be clearer."

Reply:

We appreciate your feedback. We have updated our manuscript and provided a new Figure 1. In the revised figure, we have separated the inference and optimization phases into left and right parts, respectively. Additionally, we have provided detailed structures and data flow for each network in the optimization part. This redesign aims to enhance clarity and improve the visual representation of our framework.

Q3: Missing details about belief encoders and hyperparameters

"Lack of information about the model structure of the belief encoders and hyperparameters of optimizing them."

Reply:

Thank you for pointing this out. We have revised Section 3.3.2 (Optimization Phase) in the updated manuscript (see the blue-highlighted parts). We have added detailed descriptions of the belief encoders:

“The belief encoder $f_e(\cdot; \theta_e)$ aggregates the belief states from all agents to generate a group-level representation $E = f_e(\{b_i\}_{i=1}^N; \theta_e)$, using multi-head attention with $H$ attention heads to capture inter-agent relationships. Each head is computed as:........ ”

We have also added Appendix B.5 to detail all the hyperparameters used to train EcoNash, providing transparency and facilitating reproducibility.

Q4: Additional performance comparison with a baseline with a learned action-value function

"Since the proposed method needs to tune a Q function, the author should include a baseline with a learned action-value function, e.g., [1][2]."

Reply:

Thank you for this valuable suggestion. We have compared our method with the two methods you mentioned ([1] and [2]) that use learned action-value functions. We have included the updated results in Table 1 of the revised manuscript. Comparing with these baselines enhances the persuasiveness of our method by demonstrating its advantages over approaches that also learn action-value functions. Specifically:

Analysis

1. GSM8K: EcoNash achieves a 2.7% higher average score compared to TS-LLM and a 4.9% improvement over PPO-MCTS. This demonstrates its superior performance in solving standard mathematical problems.

2. GSM-Hard: EcoNash achieves a 10.9% improvement over TS-LLM and a 13.5% improvement over PPO-MCTS. This highlights its significant advantage in tackling complex mathematical challenges.

3. SVAMP: EcoNash outperforms TS-LLM by 3.0% and PPO-MCTS by 3.0%, showcasing its efficiency in arithmetic and mathematical reasoning tasks.

4. StrategyQA: EcoNash achieves a 4.2% higher average score compared to TS-LLM and a 5.1% improvement over PPO-MCTS, demonstrating its superior understanding and decision-making capabilities in strategic question-answering tasks.

5. MATH: EcoNash outperforms TS-LLM by 4.9% and PPO-MCTS by 8.2%, highlighting its strength in addressing advanced mathematical problems.