Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization

We thank the reviewer for the constructive comments. Below, we provide detailed responses to each of your points.

1. Performance Gain vs. Computation Cost:

We acknowledge your observation regarding the performance gain relative to the computation cost increase in experimental results. We want to clarify that current multi-agent methods inherently involve a trade-off between performance gains and computational costs and were discussed limitedly in previous works. Instead, we take it as an important factor.

It is important to note that our methods are orthogonal to different prompting methods, as shown in Table 2. While prompt engineering methods can improve performance without increasing #API calls, our framework demonstrates improvement across different prompting methods by leveraging the collaborative strengths of multiple agents.

Compared with single execution, the seemingly marginal improvement from all current methods actually results from the limited collaborative capacity of the backbone model. We still view the LLM-agent collaboration as a promising direction for better utilization of LLMs. And we thought of two possible approaches to address it in the future work: (1) build LLMs that inherently incorporate collaboration mechanisms; (2) investigate specific scenarios that single LLMs struggle to complete, e.g., agents have exclusive resources or are distributedly deployed.

2. Necessity of Multi-Agent Collaboration:

Please refer to General Response 2 for overall responses to your concern about the collaborations of agents with different roles. Here're some detailed explanations:

Your question about the role of a doctor in solving a mathematics problem is insightful. There are chances that agents with certain roles refuse to answer specific queries. Generically, agents' behaviors are unpredictable without their actual responses, indicating that ideal agent team optimization could not be achieved based on human priors (also see General Response 5). Since different roles provide diversity as well as potential poor performance, we propose a posterior agent team optimization method in DyLAN to mitigate this gap.

Specific to your concern, as we described in General Response 2, Doctor, Programmer, Economist, and Mathematician agents may exclusively have correct answers to different mathematical queries, and it's difficult to distinguish whose answer is the correct one. Moreover, they may all have false answers but come up with a correct one after a discussion on the reasoning process / factual knowledge / probable hallucination. In conclusion, given that we couldn't predict whether an agent will respond faithfully according to its backbone model, give better solutions due to the stimuli of tools and role prompts, or even refuse to answer, collaboration is necessary and is usually (if not always) better for accuracy on some topics even if they have a single correct solution.

We will clarify this point by stating the discussion explicitly and demonstrating clearer cases in the case study.

3. Risk of Non-Expert Consensus Leading to Incorrect Solutions:

We want to clarify that in most subjects of MMLU, there are only 1-2 expert agents among seven candidates in our experiments, and no expert agents for a few subjects, aka. we have already conducted experiments with a majority of non-expert agents. Please refer to General Response 3 for further discussion about the imbalance of experts and non-experts.

Regarding the potential risk of non-experts reaching a consensus on an incorrect solution, our Agent Team Optimization algorithm plays a key role in mitigating this, as described in General Response 3. In extreme cases, where all agents are designed to perform poorly in specific tasks, possible techniques could be designed in addition to agent team optimization, such as automatically creating agents before optimization with validation, which is not our primary focus.

4. Clarification of "Sensitivity":

In the last paragraph of Page 1, the term "sensitivity" refers to the system's low adaptability to the domain and complexity of different tasks. In a static setup, the lack of flexibility can lead to suboptimal performance when faced with tasks that deviate from the expected domain or complexity. Our dynamic multi-round approach in DyLAN and Agent Team Optimization method is designed to adapt more effectively to a broader range of tasks. We will clarify the term in the updated version.

We thank the reviewer for the constructive comments. Below, we address each of your concerns in detail.

1. Novelty of the Framework:

Please refer to General Response 1 for the overall explanation of our contributions. We understand your perspective on the perceived limited novelty of our framework, given its use of standard technologies. However, we argue that the novelty of our work lies in the unique concept of agent team optimization, and the innovative and effective method based on the unsupervised metric Agent Importance Score. We will emphasize and clarify the point in the updated version of our paper.

2. Baseline for Agent Team Optimization:

Please refer to General Response 5 for the additional baseline (Human Prior Selection) for Agent Team Optimization. We demonstrated that the current agent team optimization method is posterior and based on actual interactions and peer ratings between agents. It exhibits significant improvement compared to prior selections. As a detailed explanation, prior selections seem to overweigh the statement from role prompts and fail to predict each agent's actual behaviors and the functions of tools from short descriptions. We will add the results and the discussion in the updated paper.

3. Shapley Value as a Metric for Individual Contributions:

Shapley Value is well-used for measuring marginal (individual) contributions in multi-agent settings. We chose not to use Shapley Value as a metric for agent team optimization because it is inherently a supervised metric, requiring labeled data to accurately assess the contribution of each agent, let alone its high computational costs. It's impractical in real scenarios and limited in generalizability. In contrast, our Agent Importance Score is an unsupervised metric designed to evaluate agent contributions without needing labeled data. This approach is more aligned with the practical constraints and diverse applications of LLM-agent collaborations, where labeled data may not always be available or feasible to obtain. Appendix D.5 elaborates on Shapley Value as the self-evident metric for validating Agent Importance Score. We will emphasize this point in the updated version of the paper.

We thank the reviewer for the constructive feedback and comments. Below, we address each of your points in detail.

1. Clarification on Agent Differentiation and Construction:

Please refer to General Response 4 for detailed clarification.

In DyLAN, the prompts are not the same between prompt-based agents according to Section 3.1 & 4.1 and Appendix E. This approach allows the agents to specialize in specific tasks or subtasks, contributing their unique strengths to the collaborative effort and utilizing different tools in various tasks. While diverse prompts may not significantly affect performance in some tasks like MATH, this specialization is crucial for achieving high accuracy in other scenarios like general reasoning and code generation.

Specific to your concern, agents with specific expertise could be automatically settled in each time step by the Agent Team Optimization process. Orthogonally, agents can be manually set in different time steps; e.g., we designate four code writers and four judges in different time steps for code generation tasks, as exhibited in Figure 5. Thus, agents could collaborate at both task and subtask levels.

Our revised paper will provide a more precise explanation of the process and construction of agent teams.

2. Agent Team Optimization Process:

The agent team optimization in DyLAN is indeed an additional step. Please note that the process is unsupervised, meaning it does not rely on labeled data. As described in Appendix C.1, the additional consumption of API calls is 8.30 on average for general reasoning tasks and 23.04 on average for code generation. We did not add these costs to the collaboration process of optimized teams of agents because the unsupervised optimization could be performed offline, and the scores are reusable. Moreover, according to the first paragraph of Section 4.3, only a small amount of data is enough for optimization towards higher accuracy.

Also, the unsupervised optimization process required no extra data as an additional step to inference. Please note that it is not an overfitting since no label is used during the process. We will emphasize these details in our revised paper to provide a comprehensive understanding of the agent team optimization process.

3. Selection of Top k Agents and Efficiency:

k is a super-parameter that is tuned to balance efficiency and performance. We assign k=2 because it is the minimal number for agents’ interactions, and empirically found it brings a great trade-off between effectiveness and efficiency. We will include the information on how k is selected in our revised paper.

4. Performance Improvement Relative to Backbone Models:

Regarding the observation in Table 5 about the smaller gains over the baseline when using GPT-4 compared to GPT-3.5, we believe this is due to the simplicity of HumanEval benchmark instead of the ineffectiveness of the method, since the score is 90+ and the improvement seems to be marginal. It is easy to understand that very weak LMs cannot collaborate and solve tasks, but DyLAN with GPT-3.5 actually gains excellent improvement. Thus, the improvement of backbone models actually makes collaboration and agent team optimization more effective. That is because capacities of reasoning, instruction following, etc., determine the effectiveness of LLM-agent collaborations.

As for empirical results, due to budget limits, we have only tested DyLAN on the humanities category of MMLU with GPT-4-0314, and we found an improvement from 83.9 (single execution) to 87.9 (+4.0), greater than +2.3 (from 59.8 to 62.1) with GPT-3.5 in Table 3. Nonetheless, we acknowledge that, ideally, if single agents are capable of solving tasks in whatever complexity at 100% accuracy, DyLAN will be meaningless. Nevertheless, in fact, even GPT-4 can only reach <50% accuracy or even lower in some challenging tasks, including MATH (we have tested), WebArena [1], SWE-bench [2], etc., denoting strong potential for LLM-agent collaboration and agent team optimization methods in DyLAN.

We are also experimenting with how our framework can be adapted to leverage the collaborative potential of LLMs in a more challenging task - WebShop. If time permits, we will also update the results.

[1] WebArena: A Realistic Web Environment for Building Autonomous Agents. https://arxiv.org/abs/2307.13854
[2] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? https://arxiv.org/abs/2310.06770

We thank the reviewer for the constructive feedback and comments. Below, we address each of your concerns in detail.

1. Clarification on the contributions and motivation

Please refer to General Response 1 for the summary of the contributions. We want to clarify that "role-playing" agents are not our focus and are only used partially in both the formulation and experiments.

2. Addressing Factual Errors:

We acknowledge the confusion caused by certain aspects of our presentation. Here are clarifications:

2.1. Empirically, we have compared the performance of single and multiple agents (in Table 2, 3, and 5), and our findings align with previous research underscoring the benefits of multi-agent collaboration.

2.2. In response to your suggestion regarding tool usage, we refer to our General Response 4, where we elaborate on tool usage in DyLAN, as described in Section 3 & 4.1.

2.3. In Figure 5, Unit Tester is indeed an LLM agent equipped with Code Interpreter as a tool, according to the last sentence of Section 4.1 and the last paragraph of Appendix C.1. We didn't deploy extra tools except code interpreters because of the alignment towards baseline methods.

2.4. We ignore the scoring process in Figure 5 mainly due to simplicity since there are 8 agents in Figure 5. We chose to demonstrate the scoring process in Figure 6 with only 4 agents.

We will clarify these points and rectify any wording issues in our revised paper.

3. Necessity and Performance of Agent Collaboration vs. Single Agent:

Please refer to General Response 2 for the discussion of the effectiveness of LLM-agent collaborations. Detailed explanations about comparing the collaboration with single LLM agents are as follows.

Our work is grounded in the philosophy that while a single, perfect agent capable of efficiently handling all tasks is an ideal goal, it is not yet achievable. In this context, our Dynamic LLM-Agent Network (DyLAN) framework aims to leverage the strengths of multiple agents in a collaborative setting to achieve better results and cost-efficiency, as well as better generalizability compared to a single agent.

From our observations in current experiment settings: (1) Expert is hard to manually select when there are many candidates; (2) The selection from human priors (by LLMs) mismatches the optimal team of agents and cause severe performance drop (refer to General Response 5).

In response to your comments, Mathematician and Programmer agents alone are not enough for all mathematical and programming problems, respectively. For instance, Programmer and Economist agent can outperform Mathematician agent in elementary mathematics, and a single programmer in Figure 5 (based on GPT-35-turbo-0301) could only achieve 73.2 on HumanEval compared to 82.9 from DyLAN. Collaboration. We'll add a different case to demonstrate the effectiveness of collaborations, e.g., DyLAN solves a query where all single agents fail.

4. Clarification of the Collaboration Process:

To be specific, in Figure 5, Programmer and Algorithm Developer agents write incorrect programs at t=1, and at t=3, they receive the judgment from judges at t=2, thereby perceiving other cases along with according judgments from Unit Tester, Syntax Checker, and other judges. Programmer at t=3 refined its solution, which turned out to be correct, while Algorithm Developer still failed. The process demonstrates that code writers refine their solutions based on the feedback from judges and ensemble cases from other code writers. After Inference-Time Agent Selection, the other two agents reach a consensus and vote for the final, correct answer.

In short, the interplay of agents' distinct expertise leads to a more robust and accurate solution than what a single agent could achieve. We will add these descriptions to the caption of Figure 5. Additionally, to demonstrate the necessity and effectiveness of Agent Importance Score in Figure 5, we will include more detailed examples in the revised version.

5. Missing References:

We apologize for the oversight in not citing the relevant paper. Though "role-playing" is not our core focus, we acknowledge its importance in the field of LLM agents. The reference will be duly added in the next version of our paper.

We thank the reviewer for the constructive feedback and comments. Below, we respond to each of your concerns in detail.

1. Clarity and Problem Formulation:

We appreciate your suggestion regarding our formulations. We agree that Figure 5 is a crucial element in conveying the concept and operations of DyLAN. Currently, we demonstrate the formal model for agent interactions as a feed-forward network $S=(A, E)$, in Section 3.1, and we described how task query $q$ is solved by the system's output $o$ in the last paragraph. Based on your suggestion, we will consider including a formal model, including the decision-making process earlier in the text, to provide readers with a clearer and more immediate understanding of the problem being solved.

2. Handling Imbalance Between Expert and Non-Expert Agents:

Your concern about the potential imbalance between expert and non-expert LLMs in DyLAN is insightful. Please refer to General Response 3 for detailed explanations. Based on empirical results, we explained that Agent Team Optimization could deal with the imbalance between expert and non-expert agents. Implementing multiple rounds of agent selection is likely to be effective in addressing extreme cases, as you suggested. We believe this approach will enhance the robustness of DyLAN against the risk of poor accuracies due to an imbalance in expertise.

To step further, we want to share some thoughts on building metrics that reflect an agent's capacity for certain tasks. A new work demonstrates several aspects for evaluating agents under multi-agent settings [1]. We think it is possible to extend its metric to certain timesteps inside DyLAN for a clear understanding of agents' capacities. We will add the discussion to our updated paper.

3. Selection and Effectiveness of Rankers:

Regarding the rankers used in DyLAN, we conducted an ablation study to evaluate the performance of various ranking methods in Appendix D.3. The choice of a specific ranker was based on its ability to improve the overall performance with reasonable computational costs. As described in the second paragraph in Section 2, there are mainly two types of rankers - listwise and pairwise. The former ranks candidates in one pass, and the latter ranks candidates by scoring them based on the comparison result of each pair of candidates. We tested three kinds of pairwise candidates, including LLM-Blender (using GPT-3.5 to judge each pair), Elo Score (using GPT-3.5 as a judge within theoretic probability distribution models), and Sliding Window (using GPT-3.5 with only $nk$ times comparison for top k), along with listwise ranker (based on GPT-3.5).

We eventually found their performance roughly the same for DyLAN, and the listwise method saved computational costs. It might be because of the strong discrimination ability of GPT-3.5. We use the listwise ranker for all other experiments, and it indeed demonstrates effectiveness and efficiency on downstream tasks. Also, we acknowledge that multiple rankers could be potentially beneficial for DyLAN but may introduce extra costs. We will state these discussions explicitly in the updated paper.

In conclusion, we believe that our paper's additional explanations, experiments, and reorganization, as suggested in response to your feedback, will significantly strengthen our work. We are committed to advancing the field of LLM-agent collaborations and appreciate your valuable input in helping us refine our approach.

[1] MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration. https://arxiv.org/abs/2311.08562

5. Analysis and Insights on Agent Team Optimization:

The effectiveness of our agent team optimization approach is a key contribution of our work. The Agent Importance Score is designed to capture the contributions of individual agents within a collaboration based on actual responses.

First, we want to clarify the motivation of the scoring method. The scoring method is inspired by neural Importance Score in traditional machine learning studies, as explained in Section 2. While the algorithm couldn’t be directly transferred, we propose a three-step algorithm (as described in Section 3.3) by leveraging peer ratings to achieve effectiveness and efficiency. There are also several human-team optimization findings supporting our methodology, as shown in Section 2.

We also provided empirical results to demonstrate Agent Importance Score as the indicator of individual contributions. (1) The direct proof is the performance improvement after optimization in downstream tasks. In the last paragraph in Section 4.2 and the second paragraph in Section 4.3, we clearly describe the benefits of Agent Team Optimization and the impact of team sizes. (2) We also compared the scoring distribution with Shapley Value (Appendix D.5). We found that Agent Importance Score has a closer distribution to Shapley Value, which shows that the scoring process grasps the individual contributions of each agent to a fair extent.

Some insights could be derived from the existing results. As shown in Table 6, Agent Team Optimization is capable of differentiating contributory agents from the others. Some contributory agents match human prior, e.g., Mathematician for college mathematics, and some seem to be unrelated or not directly related, e.g., Psychologist for public relations. It reveals that Agent Team Optimization could not only retrieve domain experts but might also retrieve agents with other domain knowledge that might help (refer to Appendix D.1 & D.4). Moreover, peer ratings are more reliable than self-confidence due to previous studies [6], which provides extra solid support for our method.

Please refer to General Response 5 for the additional experiment of another baseline for Agent Team Optimization. We will add the discussion in the next version of our paper.

6. Extension to Online Agent Team Optimization:

First, we want to clarify that Agent Team Optimization does not depend on the label or the ground truth of datasets. The optimization is based on the unsupervised metric Agent Importance Score. Thus, the optimization method can already be performed posteriorly during inference at this stage.

Regarding online agent team optimization methods, DyLAN currently employs Inference-Time Agent Selection as a naive approach of online optimization, as described in Appendix B (Limitation). The challenge is to capture the actual multi-round behaviors of agents in the middle of collaboration. Inference-Time Agent Selection only considers single-round responses and does not consider the ratings from the agents themselves. There is potential for further development in this area, allowing for more concrete and adaptive online agent team optimization.

7. Optimal Threshold of Answer Similarity for Early Stopping:

In Appendix C.1, we described using exact match for classification queries and BLEU scores for open-ended generation tasks, including code generation tasks. We set the threshold of BLEU score to 0.9 empirically for all open-ended generation tasks without specific tuning since it’s not our primary focus. In Appendix B (Limitation), we acknowledge that the threshold and choices of methods could be further optimized, and based on our observation, it does affect the performance, though marginally.

[6] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. https://arxiv.org/abs/2306.13063

We thank the reviewer for the constructive feedback and comments. Below, we respond to each point in detail.

1. Summary of Key Contributions and Novelty:

Please refer to General Response 1 for the summary of contributions. Our core contribution of improving LLM-agent collaborations by optimizing agent teams is distinct from previous methods or concepts, especially as we proposed an unsupervised metric Agent Importance Score to implement the posterior optimization.

2. Experimental Validation and Real-World Applicability:

Our experiments were designed to provide a clear and objective evaluation of DyLAN's performance. We chose established general datasets and the strongest baselines (before the submission time) for their recognized validity in the research community. All baselines are chosen for fair reasons. Specifically, CodeT is the strongest method based on GPT-35-turbo before submission. Even though Reflexion managed to reach SOTA using GPT-4, it still performs worse than CodeT on GPT-35-turbo, as shown in Table 5.

We acknowledge that CodeT is not a multi-agent method, but it serves as a strong baseline even with multi-agent methods coming out near the submission date. Because we were not able to re-implement all of them, we post a few reported results from recent (concurrent) multi-agent works on HumanEval with different backbone models:

Regarding MMLU and MATH datasets, most multi-agent works [4] have not reported their results on the datasets or used the same backbone model, and it’s out of budget for us to reimplement those methods on GPT-35-turbo. Nonetheless, we demonstrate our fair reasons for baseline selections and the effectiveness of DyLAN.

We acknowledge your point about the potential benefits of evaluating DyLAN on more complex, real-world tasks. As part of the rebuttal response, we are extending our evaluation to more diverse and challenging scenarios, such as WebShop [5], to demonstrate the applicability of our framework further. Due to the time limit, we plan to update the results later.

3. Clarification on Computational Overhead and Efficiency:

Regarding the computational overhead of techniques like peer rating and consensus checks, we have already included these factors in the number of API calls reported in our experiments:

• In Appendix C.2, we mentioned that the rating is implemented by the suffix of prompts. Thus, the rating process does not affect #API calls. Indeed, this kind of prompting would increase token consumption. However, we find it insignificant since CoT prompts, reasoning processes, mathematical formulas, and codes take up most of the token consumption.

• In the last two paragraphs of Appendix C.1, we mentioned that consensus checks are processed by template matching with pure scripts. Thus, the overhead is small and does not change #API calls as well.

We appreciate your concern about our potential mistakes. We will explicitly describe the overhead above in the next version of the paper.

4. Clarity and Conciseness of Presentation:

We appreciate your feedback on our paper's presentation. We will undertake a thorough review to eliminate repetitive content and improve clarity and conciseness.

[1] AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. https://arxiv.org/abs/2308.10848
[2] CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In proceedings of the 37th Conference on Neural Information Processing Systems. https://arxiv.org/abs/2303.17760
[3] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. https://arxiv.org/abs/2308.00352v5
[4] AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. https://arxiv.org/abs/2308.08155
[5] WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. https://arxiv.org/abs/2207.01206