Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems

We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1.1: Inadequate Justification for Pruning Methodology The one-shot pruning approach, though efficient, lacks theoretical grounding in the context of MAS.

We sincerely appreciate your thoughtful feedback! Please allow us to introduce how AgentPrune is motivated and grounded. The idea behind AgentPrune is quite intuitive: data structures such as graphs or networks are often highly redundant. This concept has been explored, defined, and addressed in thousands of studies, covering areas such as neural network pruning [1,2], graph sparsification [3,4], graph-parameter co-pruning [5], neural architecture search (NAS) [6] and traditional multi-agent reinforcement learning (MARL) [7]. These studies have demonstrated that whether dealing with neural networks composed of neurons or graphs made up of nodes and edges, many components can be pruned without significantly impacting the system's utility.

As formalized in Section 2, LLM-MAS can be naturally defined as a spatial-temporal graph. Consequently, examining its redundancy and removing it becomes a natural and meaningful task. One-shot pruning, as a classic pruning method, has shown significant success in network pruning [2], graph pruning [4], NAS [6] and MARL [7]. Thus, we selected one-shot pruning as the initial approach for achieving efficient LLM-MAS. We further hope that AgentPrune can pave the way for the community to explore more advanced techniques for enhancing MAS efficiency, with our well-formalized problem definition.

Weakness 1.2: Fixed structure across all tasks The decision to prune a fixed percentage of tokens and communication paths across all tasks does not account for the varying communication needs of different agents or tasks. This oversimplified pruning criterion may lead to suboptimal performance in scenarios where more dynamic and context-sensitive pruning is required.

To address your concerns, we have undertaken the following: (1) an analysis of why AgentPrune works, (2) an empirical examination of AgentPrune's transferability, and (3) the introduction of how AgentPrune can achieve task adaptivity.

(1) Why AgentPrune Works We respectfully revisit the underlying rationale of AgentPrune, which utilizes a small subset of samples from a dataset to train and subsequently prune a communication structure that generalizes across the entire dataset. This approach is effective because queries within the same dataset typically belong to the same task category, such as mathematical reasoning or code generation, which often share similar reasoning processes and division of labor among agents. At the same time, we respectfully note that leveraging a fixed communication structure for tasks of the same domain is not an uncommon practice, which is also seen in several well-established multi-agent frameworks, including LLM-Debate, MetaGPT, GPTSwarm, CAMEL, AutoGen and Exchange-of-Thought.

(2) AgentPrune Exhibits Cross-Dataset Generalization While the workflow discussed earlier operates within a single dataset, we wish to highlight that AgentPrune demonstrates remarkable generalization across similar task domains. Specifically, we evaluated the communication structures optimized on GSM8K and SVAMP by directly transferring them to other datasets without any further optimization. The results, shown in Table A, reveal strong transferability:

Table A. The transferability analysis of AgentPrune, with the backbone LLM-MAS being complete graph.

Notably, directly applying the structure optimized on GSM8K to AQuA yields better performance than the structure specifically optimized on AQuA itself. This finding underscores AgentPrune's impressive cross-task generalization capabilities.

(3) Achieving Task Adaptivity with AgentPrune We sincerely appreciate your insightful observation regarding the limitations of applying a fixed percentage of communication paths across diverse tasks, especially in dynamic scenarios. To address this concern, we propose a minimal enhancement to AgentPrune, requiring fewer than 40 lines of code changes. This enhancement enables task-adaptive allocation of communication structures through the integration of a lightweight task evaluator $\psi$:

• During training, the evaluator takes the task description $q$ as input and outputs a task-adaptive threshold $thres$, calculated as:
  $\psi(q) = \operatorname{Sigmoid}(\operatorname{FFN}(\operatorname{Embedding}(q))),$
  where $\operatorname{Embedding}(\cdot)$ can be implemented via any text embedding model like SentenceBert or MiniLM. The subgraphs $\\{\\mathcal{G}^\mathcal{S}\_k, \mathcal{G}^\mathcal{T}\_k\\}_{k=1}^M$ extracted in Eq. (9) and (10) are constraint by $thres$, ensuring:
  $\mathbf{A}(\mathcal{G}^\mathcal{X}\_k) \subseteq \mathbb{1}[\mathbf{A}(\mathcal{G}^\mathcal{X}) < thres] \odot \mathbf{A}(\mathcal{G}^\mathcal{X}), \mathcal{X}\in\\{\mathcal{S},\mathcal{T}\\}$
  which allows $\psi$ to be updated jointly with the graph masks via backpropagation.

• During inference, given a query/task $q$, the task evaluator $\psi$ computes the task-specific sparsity. Using this threshold, the optimized graph masks $\mathbf{S} = \\{\mathbf{S}^{\mathcal{S}}, \mathbf{S}^\mathcal{T}\\}$ are pruned to obtain a tailored subgraph $\mathcal{G}^{sub}_q$ for each task.

This straightforward yet impactful component facilitates dynamic configuration of the communication graph. We have made its implementation publicly available in https://anonymous.4open.science/r/agentprune_rebuttal-FE38, where the implementation of $\psi$ can be found at https://anonymous.4open.science/r/agentprune_rebuttal-FE38/AgentPrune/llm/llm_embedding.py. The overall modification involves fewer than 40 lines of code modifications compared to the original codebase.

In the response to Weakness 2.2, we will experimentally evaluate whether this enables AgentPrune to further satisfy the requirements for dynamic scenarios.

Weakness 2.1: How pruning affects quality and depth The paper provides little analysis of how pruning affects the quality and depth of agent interactions or task accuracy, especially in more nuanced MAS tasks.

Thank you for your insightful comment! Here, we address your concerns regarding AgentPrune's impact on interaction quality and depth separately:

Quality To evaluate how AgentPrune impacts the quality of agent interactions, we measured the accuracy of multi-turn dialogues before and after incorporating AgentPrune across three MMLU subsets. The results are summarized in Table B:

Table B. Accuracy of agent interactions with and without AgentPrune on LLM-Debate. We use four gpt-3.5-based agents.

The results reveal two main observations. First, with AgentPrune, the system achieves peak performance more quickly. For instance, in the Social Science subset, AgentPrune enables the system to reach an accuracy of 74.8% in Round 2, comparable to the Round 3 accuracy of vanilla LLM-Debate. Second, AgentPrune improves the ultimate accuracy of the system. In the STEM subset, for example, it boosts final accuracy by 3.6%, surpassing the peak performance of the vanilla LLM-Debate. These findings demonstrate that AgentPrune enhances the quality of interactions, fostering more effective and redundancy-free agent discussions.

Depth In our experiments, the number of interaction rounds was fixed for all LLM-MAS setups to ensure fair comparisons. Therefore, AgentPrune does not explicitly alter the depth of interactions. However, as shown in Table B, it enables the system to converge more quickly to accurate consensus, optimizing the trade-off between depth and quality within the same interaction constraints.

Weakness 2.2: Pruning's impact on collaborative tasks For instance, the pruning’s impact on collaborative tasks requiring high contextual awareness is underexplored.

To address your concerns, we will: (1) clarify that the benchmarks currently used already require high contextual awareness, and (2) provide additional experimental results for AgentPrune on more dynamic tasks.

Firstly, we humbly emphasize that some of the baselines we selected inherently demand high contextual awareness. For example, the code generation benchmark HumanEval assigns distinct roles to different agents (e.g., code reviewer, programmer, product manager), each receiving unique context and handling specific tasks. This process often requires iterative refinement of code to achieve the desired outcome. Thus, we respectfully assert that the current experimental results already reflect AgentPrune's effectiveness in tasks requiring high contextual awareness.

Secondly, we supplement our experiments in more dynamic and open-ended environments. Specifically, we evaluated AgentPrune's performance and token efficiency on two subtasks from the GAIA [8] benchmark: web browsing (355 queries) and diverse filetype reading (129 queries), when combined with GPTSwarm. These subtasks involve either highly dynamic open web environments or complex toolchains and are widely recognized as agentic benchmarks that necessitate high contextual awareness. The results are summarized in Table C.

Table C. Performance and token efficiency of AgentPrune combined with GPTSwarm. Both web browsing and file reading used Level 1 difficulty tasks. GPT-4 and AutoGPT results are taken from the GAIA benchmark.

Our observations are as follows:

1. AgentPrune remains effective in highly dynamic environments.
   The web browsing task includes complex and dynamic scenarios such as website widgets access and Google Street View, while file reading involves intricate tool usage like Excel summarization and PowerPoint viewer. Despite the challenges posed by such dynamic and complex benchmarks, AgentPrune achieves significant token savings, up to 60%, when integrated with GPTSwarm.

2. Task-adaptive AgentPrune provides additional benefits.
   As discussed in our response to Weakness 1.1, AgentPrune w/ $\psi$ dynamically adjusts the sparsity of the communication structure based on task requirements. This adaptation delivers two key improvements:

   • Performance Gains: AgentPrune w/ $\psi$ achieves a 1.8% improvement in web browsing and a 1.89% improvement in file reading.
   • Greater Token Efficiency: In file reading, AgentPrune w/ $\psi$ delivers 24.7% more token savings compared to the vanilla AgentPrune, thanks to its task-adaptive complexity adjustment.

These results further validate that AgentPrune are well-suited for tasks requiring high contextual awareness and excel in dynamic and complex environments.

Finally, we present a case study demonstrating the task-adaptiveness of AgentPrune. As shown in Figure 39 of our updated manuscript, we observe that for elementary-level queries, AgentPrune effectively adjusts to a higher sparsity, resulting in a highly simplified communication structure. In contrast, for more complex queries, a more intricate structure is allocated. We believe these discussions sufficiently validate that AgentPrune excels in addressing dynamic and open-world challenges.

Weakness 3: Formate issue of Figure 1

Thank you for your valuable suggestions! Based on your feedback, we have carefully revised Figures 1 and 4 to enhance their readability:

• For Figure 1 (introduction figure), we

  • Standardized all fonts to Georgia for consistency.
  • Reorganized the legend to improve clarity and ease of understanding.

• For Figure 4 (overall framework), we

  • Simplified the color scheme, unifying red for the left side and blue for the right side.
  • Limited all fonts to Times New Roman and Georgia to avoid excessive formatting.
  • Removed overly lengthy text, making the framework more concise and intuitive.
  • Eliminated unnecessary unannotated symbols or elements, placing greater emphasis on AgentPrune’s pruning workflow.

These updates are reflected in the revised manuscript available on OpenReview. We hope these changes address your concerns effectively.

Question 1: Experiments that require high contextual awareness If you could include experiments that require high contextual awareness, demonstrating that your model can improve both quality and token efficiency in such scenarios, I would be happy to raise my evaluation score.

We sincerely appreciate your insightful feedback and generosity! In our response to Weakness 2.2, we supplemented experiments demonstrating AgentPrune's performance in more dynamic and context-intensive tasks, specifically web browsing and file reading. These results confirm that AgentPrune effectively maintains solution quality while significantly reducing token consumption. We hope this addresses your concerns and would be delighted to clarify any further questions!

Question 2: Deeper analysis And I would appreciate a deeper analysis of the types of errors or ineffective communications that AgentPrune eliminates, as this would clarify the practical implications of its pruning method.

Thank you once again for your thoughtful question! To address your concerns, we have categorized two primary pruning focuses of AgentPrune in practical implementations. Detailed illustrations can be found in the revised manuscript (Figures 37 and 38). Specifically:

• Malicious or misleading information: As demonstrated in Figure 37, Web Browser 1 incorrectly claims that the "Top 5 Silliest Animal Moments" includes the fictitious "kakapo parrot." This misinformation propagates to downstream agents, such as inspectors and aggregators, leading to incorrect conclusions. After applying AgentPrune, 3 out of 5 outgoing edges from Web Browser 1 are pruned, allowing the system to generate the correct answer.

• Redundant information: In Figure 38, the math analyst provides no novel insights after receiving input from the math solver, merely reiterating the solver’s output. In such cases, AgentPrune significantly reduces the redundant agent's outgoing edges, effectively limiting unnecessary information flow and optimizing token usage.

We hope these analyses clarify the mechanisms and success of AgentPrune in enhancing system efficiency and performance.

Ethics Concerns

Regarding the ethics concerns you raised, we respectfully clarify that after thoroughly reviewing all relevant ICLR conference policies, we found no rule prohibiting authors from updating their anonymous code repository after the submission deadline. Unlike ICML, which explicitly addresses this in its guidelines (https://icml.cc/Conferences/2024/AuthorInstructions), ICLR does not impose such restrictions. In fact, such updates align with ICLR's encouragement of open-source research and enhancing code usability.

Additionally, we would like to emphasize that the last update to our anonymous repository (on October 19, 2024) involved fewer than five lines of code changes and minor updates to the README file. These modifications were solely intended to promote the open-source spirit and ensure that our code remains accessible and easy to use for the research community.

[1] Only train once: A one-shot neural network training and pruning framework, NeurIPS 2021

[2] Opq: Compressing deep neural networks with one-shot pruning-quantization, AAAI 2021

[3] Demystifying graph sparsification algorithms in graph properties preservation, VLDB 2023

[4] One-shot neural network pruning via spectral graph sparsification, TAG-ML 2023

[5] A unified lottery ticket hypothesis for graph neural networks, ICML 2021

[6] Single path one-shot neural architecture search with uniform sampling, ECCV 2020

[7] Multi-Agent Game Abstraction via Graph Attention Neural Network, AAAI 2020

[8] GAIA: A Benchmark for General AI Assistants

Dear Reviewer rYs7,

We sincerely appreciate your generous commendation and thorough feedback on our work. Your brilliant insights deeply inspired us, and we are genuinely committed to addressing them with the utmost care. To aid in better understanding our rebuttal and revisions, we have summarized your key concerns and our responses below:

• Inadequate Justification for Pruning Methodology Weakness 1 We have carefully clarified (1) how AgentPrune is conceptually grounded and motivated, (2) the strong cross-dataset generalization capabilities of AgentPrune's fixed structure, and (3) that AgentPrune can achieve one structure per task with fewer than 40 lines of code modification.
• How pruning affects quality and depth Weakness 2 & Question 2 We provided additional analysis of how AgentPrune influences the quality and depth of MAS interactions.
• Pruning's impact on collaborative tasks Weakness 2 & Question 1 We validated AgentPrune’s adaptability and performance in dynamic collaborative tasks such as web browsing and file reading.
• Formatting issues Weakness 3 We have revised Figures 1 and 4 to enhance their clarity and presentation quality.

Thank you once again for your dedication throughout the review process. We fully and humbly understand that you might have a tight schedule, and we would be sincerely grateful if you could let us know whether our rebuttal has sufficiently addressed your concerns and whether it might warrant a reconsideration of the score. Thank you immensely!

Warm regards,

Authors

Dear Reviewer rYs7,

We would like to thank you for your insightful review and dedication to the review process. With your help, the quality of the manuscript has been significantly improved. Due to the approaching deadline for submitting the revised manuscript, we are now submitting a revised version with highlighted changes.

Also, we would like to respectfully convey our willingness to address any further concerns you may have. We aim to make the most of this opportunity for revision, as we deeply value your invaluable insights.

Thank you once again for your support.

Warm regards,

Authors

Dear Reviewer rYs7,

We sincerely apologize for reaching out again and fully understand that your time is extremely valuable. If this message causes any inconvenience, we deeply regret and apologize for the interruption.

The reason for our follow-up is that your review has been really insightful and constructive, and therefore we are eager to know whether our responses have adequately addressed your concerns.

To further address your expectations regarding collaborative tasks requiring high contextual awareness, we extend our experiments beyond the previously included web browsing and file reading tasks by incorporating results on the ALFWorld benchmark. ALFWorld, a diverse suite of synthetic, language-based, interactive decision-making tasks in household environments, serves as a robust evaluation platform for assessing AgentPrune’s effectiveness in applications involving interactive or online decision-making. The results are summarized below:

Table A. Results on ALFWorld benchmark.
We employ a four-agent system (instantiated by GPT-3.5), where the roles include task suggestor, executor, grounding agent, and household expert. Results of methods marked with * are borrowed from original papers. For simplicity, in-context examples and parts of the exploration trajectory have been omitted, focusing only on components that contribute to success or failure across three attempts.

We observe that in open-world text-based scenarios—including tasks such as finding hidden objects (e.g., locating a spatula in a drawer), moving objects (e.g., transferring a knife to a cutting board), and manipulating objects with other objects (e.g., chilling a tomato in the fridge)—AgentPrune is still capable of achieving performance improvement with 36% reduction in prompt token consumption. We humbly believe this further corroborates AgentPrune’s utility in highly collaborative, context-sensitive tasks.

After all these efforts, we would greatly appreciate your guidance on the following points:

• For Weakness 1, whether our justification of the one-shot pruning methodology is sufficiently clear.
• For Weakness 2, whether the additional experiments in collaborative scenarios, which involve more dynamic and context-sensitive pruning, meet your expectations.
• For Question 2, whether the deeper analysis we provided for AgentPrune is sufficient.

We have done our utmost to provide more experimental results and analyses in response to your invaluable feedback. However, with the PDF revision deadline fast approaching, we are uncertain if these efforts fully address your insightful comments. We would greatly appreciate your reply and any further suggestions you may have.

Thank you once again for your time and invaluable insights!

Best regards,

All Authors!

Dear Reviewer rYs7,

Warm greetings as we step into December!

With the author-reviewer discussion deadline approaching in two days, we have provided additional explanations regarding your observation on Weakness 1: Inadequate Justification for Pruning Methodology to facilitate further communication.

You mentioned that AgentPrune employs an "oversimplified pruning criterion." However, we find this critique could similarly apply to a substantial number of existing pruning methods. AgentPrune leverages magnitude-based pruning with trainable masks, a paradigm widely adopted in the literature. For instance, representative examples include:

• LTH [1] (ICLR 2019 Oral; 4000+ citations; magnitude pruning with trainable masks)
• GLT [2] (ICML 2021; 200+ citations; magnitude pruning with trainable masks)
• RigL [3] (ICML 2022; 600+ citations; magnitude pruning with weights)
• LLM-Pruner [4] (NeurIPS 2023; 370+ citations; gradient magnitude pruning)
• Wanda [5] (ICLR 2024; 340+ citations; magnitude pruning with weights and activations)

These examples demonstrate that magnitude-based pruning, whether in network or graph pruning contexts, is not only common but also well-established and grounded in prior work.

Finally, we respectfully reiterate that we have conducted comprehensive analyses and experiments to address all your concerns in detail. With the discussion deadline so close, we are eager to know if our responses have alleviated your concerns and if there is room for you to revisit your evaluation score, as you generously mentioned earlier.

Thank you once again for your valuable feedback and kind consideration.

Best regards,
Authors

[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR 2019
[2] A Unified Lottery Ticket Hypothesis for Graph Neural Networks, ICML 2021
[3] Rigging the Lottery: Making All Tickets Winners, ICML 2022
[4] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023
[5] A Simple and Effective Pruning Approach for Large Language Models, ICLR 2024

Dear Reviewer rYs7,

Thank you for your evaluation of our work and the valuable feedback you have provided. As the deadline for the author-reviewer discussion is fast approaching, we sincerely hope you can join us for the final discussions to further enhance the quality of our manuscript.

We have tried our best to give more experimental results and corresponding analysis according to your requirements. However, we do not know whether this can be a good answer to your comments. We look forward to your reply and corresponding suggestions!

Thank you immensely for your time and insights!

Best regards,

All authors!

Thank you immensely for your time and efforts, as well as the helpful and constructive feedback! Here, we give point-by-point responses to your comments.

Weakness 1: It appears necessary to fix the number and roles of agents before implementing AgentPrune.

Your observations are truly insightful! We address your questions regarding the number of agents and the roles of agents separately below:

On the number of agents, we would like to respectfully clarify that it is more appropriate to say AgentPrune necessitates an upper bound on the number of agents, rather than the specific number of agents. In certain scenarios, AgentPrune can completely prune all the agents of a specific agent (nodes), effectively removing them from the system. This can occur when the number of agents is disproportionately high relative to the task complexity or when adversarial agents are present in the system. We have supplemented the revised manuscript with relevant case studies in Appendix I.1 and I.2 to illustrate this behavior. This flexibility in handling agent numbers distinguishes AgentPrune from many existing methods, such as DyLAN, LLM-Debate, AutoGen, and MetaGPT, which require the agent count to be fixed and immutable.

On the roles of agents, we acknowledge that the roles of agents shoule be predefined prior to the optimization process. However, this requirement is consistent with the majority of mainstream multi-agent systems, including GPTSwarm, DyLAN, LLM-Debate, and MetaGPT.

Weakness 2: To effectively train each version of the framework, a substantial amount of data is required to optimize the connectivity parameters through sufficient rollouts.

We would like to respectfully point out that AgentPrune does not require a substantial amount of data to optimize the spatial-temporal graph masks. On the contrary, the training cost is negligible compared to the inference cost. To demonstrate this, we present in Table B the training and total (training + inference) token costs of AgentPrune when combined with various LLM-MAS backbones. As shown, AgentPrune requires only around 2%–6% of token consumption to complete the spatial-temporal connectivity optimization. Once this optimization is achieved, all subsequent inferences benefit from a token-efficient and economical communication structure.

Table B. Training and total token costs of AgentPrune (AP) applied to different LLM-MAS backbones on the GSM8K dataset.

Question 1: I would like to know the required quantity of data to be generated as well as the number and types of mask configurations necessary for training in Sections 3.2 and 3.3

In Table B, we present the token consumption during the connectivity optimization stage of AgentPrune. In practice, optimization is performed using only $\\{5, 10, 20\\}$ data queries, as stated in Line 402, which demonstrates the exceptionally low data requirements for training of AgentPrune. The relevant parameter sensitivity analysis can be found in Appendix H.4.2.

Regarding the specific mask configuration, we provide a brief summary: for any multi-agent system and its corresponding spatial-temporal communication graph $\mathcal{G}$, AgentPrune assigns trainable spatial and temporal graph masks, denoted as $\mathbf{S}^\mathcal{S}$ and $\mathbf{S}^\mathcal{T}$. Both masks are initially set to $0.5 \cdot \mathbf{I}\_{|\mathcal{V}|}$, where $\mathbf{I}\_{|\mathcal{V}|} \in \mathbb{R}^{|\mathcal{V}|\times |\mathcal{V}|}$ is a matrix of ones. After training and optimization over $Q' \in \\{5, 10, 20\\}$ queries, guided by Eq. (8), AgentPrune performs one-shot pruning of $\mathbf{S}^\mathcal{S}$ and $\mathbf{S}^\mathcal{T}$ based on a specified sparsity ratio $p\% \in \\{50\%, 30\%\\}$. This results in the pruned graph $\mathcal{G}^{sub}$, as defined in Line 310.

As evident, the mask optimization process in AgentPrune is straightforward, with only two key parameters, $Q'$ and $p\%$. Should you have further questions regarding the mask configuration, we are more than happy to provide additional clarification.

Dear Reviewer FgBo,

We would like to extend heartfelt thanks to you for your time and efforts in the engagement of author-reviewer discussion. To facilitate better understanding our rebuttal and revision, we hereby summarize your key concerns and our responses as follows:

1. Does AgentPrune need to fix the number and roles of agents? We respectfully clarify that (1) AgentPrune can adaptively regulate the number of agents, and (2) similar to many mainstream multi-agent systems, AgentPrune requires fixed agent roles.
2. Does AgentPrune require extensive training data? Our experiments demonstrate that AgentPrune requires only 2–6% of the overall token consumption for training.

For other issues not mentioned here, please refer to our detalied rebuttal response. We sincerely hope this addresses your concerns! We respectfully look forward to further discussion with you.

Warm regards,

Authors

Apologies for the late response.

Overall, the authors' responses effectively addressed my concerns.

Regarding the token consumption in connectivity optimization, my previous question primarily focused on the number of structural samples required for the policy gradient (i.e., M). In your responses to other reviewers, I noticed that M = 10 and Q' = 10 were mentioned. Does this imply that the policy gradient process identified the optimal structure using approximately 100 samples?

We would like to sincerely thank you for your inspiring discussion and stronger support of our work. We are more than happy to see that our rebuttal has properly addressed your concerns!

We would like to express our deepest respect for your meticulous review! We can genuinely sense that you have dedicated considerable time to thoroughly reviewing our manuscript (even including the appendices) and providing very specific insights and feedback. We must acknowledge that this is one of the most enlightening and helpful reviews we have received in recent years! In response to your efforts, we have carefully prepared a point-by-point reply:

Weakness 1.1: wide application of AgentPrune In other words, using AP is reasonable from an economics point of view only at large scales, where there are enough training examples and enough expected future queries to warrant such an optimization step. Still, it's hard to make a case for widely applicable cost savings, especially since each graph appears to be task-specific and thus not transferable.

We acknowledge your insight that both AgentPrune and other multi-agent network optimization techniques, such as GPTSwarm and AFlow [1] (a recent multi-agent search framework open-sourced by the MetaGPT team), rely on the availability of a training set, which limits their applicability in broader scenarios. However, we respectfully present the unique characteristic of AgentPrune: while it does depend to some extent on training resources, this dependency is minimal.

From an experimental perspective, we provide a comparison of the training costs of AgentPrune and GPTSwarm in Table A. As shown, the training token consumption of AgentPrune is significantly lower than that of GPTSwarm. It requires only 3%-4% of the total token usage in the entire experiment to complete the spatial-temporal connectivity optimization.

Table A. Training and total token costs of AgentPrune-C and GPTSwarm on the GSM8K dataset.

In terms of the required number of training queries, AgentPrune only needs $\\{5, 10, 20\\}$ queries for optimization, as stated on Line 402, which contributes to its token-saving efficiency. Given this context, we believe that in scenarios with limited data, developers may be unable to use methods like GPTSwarm for initial optimization (since this requires hundreds of queries). However, AgentPrune could be a viable choice.

Weakness 1.2: unfair comparison Comparing with optimization-free baselines (e.g. CoT) is also a bit unfair, since they do not require this initial training step.

Thank you for your comment! We acknowledge that comparing our approach with optimization-free baselines, such as CoT and ComplexCoT, may not be fair, as they utilize only a single LLM-based agent and do not involve a training process. However, following the methodology of several well-known multi-agent studies, such as EoT[2], AgentVerse[3], GPTSwarm, and DyLAN, which also compare CoT with multi-agent approaches, we chose to follow their setting.

Weakness 2: misleading result emphasizing the reduction in prompt tokens instead of the cost itself (e.g. in Table 3) is slightly misleading; In the same vein, reporting in Table 2 the performance gain w.r.t. Vanilla sort of hides the fact that AgentPrune has slightly worse performance compared to Reflexion prompting (which is not a dealbreaker, but it should still be noted).

Thank you for pointing that out, and we apologize for any potential confusion caused! Following your suggestion, we have updated Tables 3 and 5 in the revised manuscript to more comprehensively present savings in both token consumption and cost. Additionally, we have restructured Table 2 to highlight the ranking order of the different methods. We hope this revision helps to present the results in a more balanced light.

Weakness 3: missing experimental details Finally, there are some missing experimental details in the paper, such as: What function $\phi$ is used as in the experiments? What connection probability is used to generate the random graph? What’s the diversity penalty for the OpenAI models? While I found the answers to these questions in the code, given the non-static nature of repositories these values should also be reported in the experimental setup appendix of the code.

Thank you for your valuable feedback! Following your suggestion, we have added more detailed experimental settings in Appendix G.2 and Appendix G.1.1 of the revised manuscript, including but not limited to:

• The implementation details of function $\phi$
• The generation process of the random graph communication structure
• The initialization of the graph masks $\mathbf{S}^\mathcal{S}$ and $\mathbf{S}^\mathcal{T}$
• The temperature setting for GPT models

We hope these additions enhance the completeness and reproducibility of the paper.

Minor notes: typos

We greatly appreciate your thorough review! Based on your feedback, we have carefully re-examined the manuscript and made corrections to the typos you pointed out, including spelling errors and issues with numerical precision. All these revisions have been presented in the update manuscript, available on OpenReview.

Question 1: applicable dataset scale What’s the minimum scale (in terms of e.g. dataset size and number of queries) where AgentPrune is more cost-effective than other techniques?

Your question is highly insightful! To address it, we provide a detailed cost analysis based on Section 3.4 and Appendix E of the manuscript, demonstrating that AgentPrune achieves significant cost savings even on small dataset scales.

Notations Consider a dataset comprising $Q$ queries, where $Q'$ queries are used for training. Let the communication graph be $\mathcal{G}$, with $K$ dialogue rounds. Assume the average token consumption per spatial, temporal, and query message is $c_\mathcal{S}$, $c_\mathcal{T}$, and $c_q$, respectively.

Cost Analysis For a vanilla multi-agent system, the total token consumption can be approximated as:

With AgentPrune, the token cost, as given in Eq. (16), becomes:

The cost saving introduced by AgentPrune can then be quantified as:

When $\Delta > 0$, AgentPrune reduces overall system costs. From $\Delta > 0$, we derive the following constraint:

This inequality defines the range of dataset sizes $Q$ and training set sizes $Q'$ where AgentPrune is effective.

Illustrative Example While the above constraint is rigorous, it may lack intuitive clarity. To provide a concrete example, we assign realistic parameter values:

• Vanilla multi-agent system: random graph for spatial communication; LLM-Debate for temporal communication; five LLM-based agents.
• Parameters: $C_q = 30, C_\mathcal{S}=100, C_\mathcal{T}=100, |\mathcal{E}^{\mathcal{S}}|=10， |\mathcal{E}^{\mathcal{T}}|=25$. The other parameters are the same with the original manuscript, like $M=10, p\\%=50\\%, Q'=10$.

Substituting these into the equation yields $Q > 102$. This indicates that AgentPrune achieves cost savings on datasets with more than 102 queries. Notably, this threshold is far below the size of datasets we use, such as HumanEval (164 queries) and GSM8K (8.5k queries), as well as popular alternatives like MBPP (1000 queries) [4] and GAIA (450 queries) [5], all of which exceed this requirement.

Conclusion This analysis underscores AgentPrune’s broad applicability. It demonstrates that even datasets as small as ~100 queries can benefit from significant cost reductions (and potentially improved performance) through AgentPrune.

Question 2: Transferability of AgentPrune Do networks trained on a task transfer to similar tasks?

Your question is crucial for enhancing the practical applicability of AgentPrune! To address it, we conducted transferability experiments across four mathematical reasoning datasets, evaluating the performance of communication graphs optimized by AgentPrune when transferred to other datasets without further optimization. The results are summarized as follows:

Table A. The transferability analysis of AgentPrune, with the backbone LLM-MAS being complete graph.

We draw several key conclusions: (1) AgentPrune-optimized communication graphs exhibit strong transferability on similar tasks. For instance, directly applying the structure optimized on GSM8K to AQuA outperforms the structure specifically optimized on AQuA itself. (2) Transferability is influenced by the knowledge capacity of the dataset. GSM8K, with thousands of mathematical queries, yields communication graphs that generalize well to other datasets. For example, the GSM8K-optimized structure achieves near or even better performance compared to vanilla optimization on MultiArith and AQuA, including a $1.03\%$ improvement on AQuA. In contrast, SVAMP, composed of merely elementary-level queries, demonstrates relatively limited generalization capacity. Its optimized structure sometimes leads to minor performance drops, such as a $0.71\%\downarrow$ decrease when transferred to MultiArith.

Overall, we respectfully submit that AgentPrune demonstrates strong generalization within similar task domains. Hopefully this can address your concerns, and we are more glad to respond to any further questions you may have!

[1] AFlow: Automating Agentic Workflow Generation

[2] Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication, EMNLP2023

[3] AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors

[4] Program Synthesis with Large Language Models

[5] GAIA: A Benchmark for General AI Assistants

Dear Reviewer sHy9,

We extend our highest respect for the time and effort you devoted to reviewing our manuscript—we can truly sense the considerable attention you have given to every detail. To facilitate better communication and express our gratitude, we sincerely summarize our rebuttal as follows:

1. In what scenarios is AgentPrune useful? Weakness 1 We deeply value your insightful observations! At the same time, we respectfully demonstrate that while both AgentPrune and existing multi-agent optimization methods (e.g., GPTSwarm) require certain optimization resources, AgentPrune's dependence on training resources is significantly lower than that of GPTSwarm, making it applicable to a wider range of scenarios.
2. Does AgentPrune exhibit transferability? Question 2 We validated the strong transferability of AgentPrune across four datasets: AQuA, MultiArith, SVAMP, and GSM8K.
3. What is the minimum dataset size for AgentPrune to remain cost-effective? Question 1 We provided the lower bound for the applicable dataset size of AgentPrune, along with an illustrative example to clarify its effectiveness.
4. Misleading results Weakness 2 Following your recommendation, we reorganized Tables 2, 3, and 5 to ensure a more balanced presentation of the results.
5. Missing experimental details Weakness 3 We have included additional experimental parameter details in Appendix G.2 and Appendix G.1.1.

For any other issues not mentioned here, please refer to our detailed rebuttal response. We sincerely and humbly believe we have responded to your concerns properly, and we once again thank you for your meticulous and generous review!

Warm regards,

Authors

We sincerely thank you for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1: Comparison with random pruning The paper start with showing random pruning can provide performance improvement, but the proposed method is not compared to the random pruning, so it is hard to understand how much improvement compared to random pruning we have.

Thank you for your insightful comment! To better illustrate the differences between AgentPrune and random pruning, we supplement the analysis with results on six datasets: MMLU, GSM8K, MultiArith, SVAMP, AQuA, and HumanEval, as shown below:

Table A. Performance comparison between AgentPrune (AP) and random pruning (RP). The pruning ratio is fixed at 50%.

The results demonstrate that random pruning consistently underperforms AgentPrune across various LLM-MAS backbones, causing notable performance degradation. While this degradation is relatively modest in dense communication structures like the Complete Graph, it becomes more pronounced in inherently sparse structures such as Random Graph and GPTSwarm. For example, random pruning results in a significant 5.79% drop in pass@1 on HumanEval+GPTSwarm. In contrast, AgentPrune achieves a 0.47% performance gain while requiring only 27.2% of the vanilla GPTSwarm's prompt tokens (as shown in Table 3). This improvement can be attributed to AgentPrune’s refined optimization strategy and precise connection importance evaluation.

Weakness 2: Training cost It is not unclear how much training and query/tokens is need to train the system, and how different number of training (one-shot) is needed and how they affect the performance.

Thank you for highlighting this issue! To address your concerns, we provide additional analyses on: (1) the training token cost of AgentPrune and (2) the impact of the number of training queries on its performance.

As shown in Table B, we report the training token cost and the total token cost of AgentPrune across different datasets. The results reveal that AgentPrune requires only a minimal proportion of training tokens relative to the total inference token cost, ranging from 2.80% to 6.82%. This demonstrates the high cost-efficiency of AgentPrune.

Table B. Training and total token costs of AgentPrune applied to different LLM-MAS backbones on the GSM8K dataset.

In Table C, we conduct a sensitivity analysis of the number of training queries used by AgentPrune on MMLU and HumanEval datasets. The results indicate that the performance of AgentPrune saturates with approximately 20 training queries, which highlights that AgentPrune can effectively learn a generalizable communication structure early in the training process using only a small number of samples.

Table C. Sensitivity analysis of the number of training queries for AgentPrune-C on MMLU and HumanEval datasets.

Weakness 3: Missing ablation study: Missing ablation study about the benefit of each of the proposed methods (i.e. distribution approximation, low-rank sparsity)

Thank you for highlighting this critical point! However, we respectfully point out that we have conducted ablation experiments on low-rank sparsity and agent profiling in Section 4.5 and Appendix H.4.1, with the results detailed in Table 6. It is important to note that we did not present ablation results for distribution approximation, as removing it would render AgentPrune non-functional, i.e., the same as random pruning.

Please allow us to elaborate further on this: during training, the graph masks are updated according to Eq. (8), where the first term corresponds to distribution approximation and the second term corresponds to low-rank sparsity. If the first term is removed, the second term reduces to a rank minimization problem with a closed-form solution, which can be computed without gradient backpropagation. (This is analogous to training a neural network using only L0 regularization on the parameters without providing downstream supervision signals.) In this case, the trainable masks would not receive any meaningful supervision signal, and the training process would become entirely random. Therefore, we did not present results regarding distribution approximation. We hope this clarifies your concerns.

Question 1: What's the training time for this proposed method. Does it overfit to the training data? How does the K' affects the performance.

Thank you again for your thoughtful comments! Below, we address each of your questions individually:

Regarding training time, we have supplemented the manuscript with training time details, as shown in Table D:

Table D. The training and overall time consumption of AgentPrune.

As shown, AgentPrune requires minimal training time, constituting only 7–18% of the overall wall-clock time. More importantly, it reduces overall execution time. For example, on GSM8K, AgentPrune reduces total time consumption by 38 minutes compared to the vanilla system, which stems from AgentPrune’s ability to prune unnecessary communication paths, significantly reducing redundant LLM API calls.

Regarding data overfitting, as demonstrated in Table C, AgentPrune requires only a small number of training samples to perform effective one-shot pruning. Its generalization ability has been thoroughly validated across diverse datasets, as detailed in Section 4.2. We respectfully believe this evidence strongly indicates that AgentPrune does not suffer from overfitting issues.

Regarding how $K'$ affects performance, our response to Weakness 2 includes a corresponding analysis and experiments on this topic. We hope this clarification adequately resolves your concerns.

Question 2: Does the mask change depending on the input data?

We address this question in two phases. During the training phase, the spatial-temporal graph masks dynamically adjust based on the MAS outputs for the input data, as detailed in Line 933 of Algorithm 3. In the inference phase, AgentPrune utilizes the sparse yet informative communication graph $\mathcal{G}^{sub}$ obtained through one-shot pruning to resolve the remaining queries. Since $\mathcal{G}^{sub}$ has already captured the critical knowledge within the current task domain, the multi-agent system can effectively complete tasks without further updates or iterations, at a significantly reduced computational cost, as demonstrated in Section 4.2.

Question 3: How does the proposed method perform compared to random pruning?

Thank you for your feedback! We have addressed this in our response to Weakness 1, where we provide a detailed comparison between AgentPrune and random pruning.

Question 4: Can distribution approximation and low-rank sparsity work in isolation? what's their effect on the accuracy and communication reducing?

We appreciate your insightful comment, which strengthens our work! In our response to Weakness 3, we have supplemented the discussion on the ablation study of AgentPrune.

Dear Reviewer CryL,

We deeply appreciate your dedication to engaging in author-reviewer discussions. Recognizing the existing discrepancies in the understanding of our manuscript, we have outlined your key concerns and our responses for enhanced communication:

1. Is AgentPrune more effective than random pruning? Weakness 1, Question 3 We empirically validated that AgentPrune significantly outperforms random pruning across six datasets.
2. What is the training cost of AgentPrune? Weakness 2, Question 1 We have comprehensively reported the training token cost, the number of queries required, and the training time for AgentPrune.
3. Missing ablation study Weakness 3, Question 4 We respectfully clarify that the relevant ablation studies are presented in Section 4.5 and Appendix H.4.1.

For other issues not mentioned here, please refer to our detailed rebuttal response. We sincerely hope this addresses your concerns! We humbly look forward to further discussion with you.

Warm regards,

Authors

Dear Reviewer CryL,

We would like to thank you for your insightful review and dedication to the review process. With your help, the quality of the manuscript has been significantly improved. Due to the approaching deadline for submitting the revised manuscript, we are now submitting a revised version with highlighted changes.

Also, we would like to respectfully convey our willingness to address any further concerns you may have. We aim to make the most of this opportunity for revision, as we deeply value your invaluable insights.

Thank you once again for your support.

Warm regards,

Authors

Dear Reviewer CryL,

This is a gentle reminder that the discussion phase will end in less than 4 days, but we have not yet received your feedback on our rebuttal. We respectfully understand that due to your other important commitments, your time is precious and the demand is high. But we are eager to collaborate with you to improve this paper, and we have made extensive contributions and efforts to this end. We sincerely hope that you find our response convincing and kindly consider revisiting your rating.

We would like to express our gratitude once again to the reviewer for their time and all constructive feedback provided!

Thanks and Regrads,

Submission 2208 authors

Dear Reviewer CryL,

We would like to sincerely you for your serious comments and time invested in our work. We have revised our paper and added relevant discussions and experiments.

At present, all your concerns are responded in the rebuttal and revised version of the paper. However, as the revision deadline approaches (only 1 day remaining), we kindly request your feedback to confirm that our response and revision effectively address your concerns. If there are any remaining issues, we would greatly appreciate the opportunity to address them to ensure the quality of our work. We sincerely hope that you find our response convincing and kindly consider revisiting your rating.

Thanks and Regrads,

Authors

Dear Reviewer CryL,

Thank you for your evaluation of our work and the valuable feedback you have provided. As the deadline for the author-reviewer discussion is in only 8 hours, we sincerely hope you can join us for the final discussions to further enhance the quality of our manuscript.

We have tried our best to give more experimental results and corresponding analysis according to your requirements. However, we do not know whether this can be a good answer to your comments. We look forward to your reply and corresponding suggestions!

Thank you immensely for your time and insights!

Best regards,

All authors!