G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks

We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1: Would switching between different agent roles have an impact on the experimental results?

Thank you for the insightful comment! To address your concerns, we analyze the transferability of G-Designer to unseen agent roles as follows:

Table A. Generalization of G-Designer to unseen agent roles. We train G-Designer on the "dataset for optimization" and evaluate it on the "dataset for test," where the test set may include agent roles unseen during training.

The results indicate that G-Designer demonstrates strong transferability across datasets, effectively adapting from HumanEval to GSM8K and from MMLU to HumanEval, among others.

Weakness 2: Typos

Thank you very much for your careful review of our manuscript. We have carefully addressed the typos you pointed out and made the necessary corrections throughout the document. Your feedback has been very helpful in improving the clarity and accuracy of the paper.

We appreciate your attention to detail and your valuable contribution to the review process.

We would like to express our deepest respect for your meticulous review! In response to your efforts, we have carefully prepared a point-by-point reply:

Weakness 1: Supplement more agent-specific benchmarks.

Thank you very much for your insights. We conducted experiments on the more agent-specific GAIA benchmark to assess the effectiveness of our method. The experimental results are as follows:

Table A. Performance comparison on GAIA benchmark.

This demonstrates that our method effectively outperforms baselines even in more agentic benchmarks like GAIA. We sincerely hope this addresses your concerns.

Weakness 2: Include the recent multi-agent papers.

Your erudition and wisdom have been immensely helpful to us! We commit to including the papers you mentioned in the revised manuscript.

Weakness 3: Are the authors planning to test more advanced or open-source LLMs?

To address your concern, we use the newer and popular closed-source model GPT-4o-mini and the open-source model DeepSeek-V3 as base models, and compare the experimental results of our method with Vanilla and GPTSwarm in Table B.

Table B. Performance results with more advanced LLMs.

Due to the time constraints of the rebuttal, we were unable to compare our method with more baselines. However, the experimental results presented above demonstrate that our method is also applicable to newer LLMs.

Weakness 4: Do the authors plan to evaluate G-Designer’s defense capabilities against a broader range of attacks?

To address your concerns, we evaluated G-Designer under the PoisonRAG [1] attack, with results presented in the table below:

Table C. Performance comparison on MMLU. We employ the configuration from PoisonRAG, inserting erroneous messages into the contextual memory of the attacker agent, enabling them to disseminate conclusions derived from these messages to others.

The results indicate that even under memory attacks, G-Designer effectively mitigates interference from malicious agents through its dedicated sparsification mechanism.

[1] Poison-rag: Adversarial data poisoning attacks on retrieval-augmented generation in recommender systems.

Weakness 5: Do the authors plan to explore alternative anchor graph initialization?

The suggestion you proposed is very thoughtful! In Table C, we explored the impact of different initial topologies on the experimental results.

Table D: Different Topologies for anchor graph initialization on GSM8K dataset.

Using different topologies as initial anchors shows minimal performance differences, although the cost is higher. This aligns with our analysis in Section 5.4, where we concluded that the success of G-Designer is mainly attributed to its efficient communication graph generation.

We sincerely thank you for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1: Temperature Settings in the Experiments

Thank you very much for your thorough review! In the multi-agent system, to facilitate more diverse communication among different agents, we followed DyLAN and set the temperature to 1. For the Single Agent scenario, to ensure more reproducible experimental results, we set the temperature to 0.

For a fairer comparison, we have also presented the performance of the Single Agent method with the temperature set to 1 in Table A. Experimental results show that the change in temperature from 0 to 1 does not significantly affect the average accuracy of single-agent methods.

Table A. Performance of single agent methods with temperature set to 1.

Weakness 2: The reason for setting the number of dialogue rounds.

Your question is very valuable! In Table B, we present the changes in accuracy and cost with different numbers of dialogue rounds on the HumanEval dataset. Intuitively, increasing the number of optimization rounds leads to more refined and accurate results, yielding substantial performance improvements, but also comes with increased cost. To balance performance with token savings, we consistently set $K = 3$.

Table B. We report the performance of G-Designer on HumanEval benchmark with different $K$ values.

As regards the automation of $K$, we respectfully argue that G-Designer can easily achieve this via incorporating existing early-stopping mechanisms like in MacNet or DyLAN. Nevertheless, G-Designer not particularly leverage these, as its core contribution lies in the automation of MAS topology.

Weakness 3: The impact of different anchor settings on the results.

The suggestion you proposed is very thoughtful! In Table C, we explored the impact of different initial topologies on the experimental results.

Table C: Different Topologies for anchor graph initialization on the GSM8K dataset.

Using different topologies as initial anchors shows minimal performance differences, although the cost is higher. This aligns with our analysis in Section 5.4, where we concluded that the success of G-Designer is mainly attributed to its efficient communication graph generation.

Weakness 4: Have the authors considered using the latest GPT or LRMs?

To address your concern, we use the newer and popular closed-source model GPT-4o-mini and the open-source model DeepSeek-V3 as base models, and compare the experimental results of our method with Vanilla and GPTSwarm in Table D.

Table D. Performance results with more advanced LLMs.

Due to the time constraints of the rebuttal, we were unable to compare our method with more baselines. However, the experimental results presented above demonstrate that our method is also applicable to newer LLMs.

Weakness 5: Have the authors considered testing with more agents?

Thank you for this thoughtful inquiry! In Table 6 of our paper, we provide a comparison of accuracy, time, token consumption, and cost across different agent number configurations. The experimental results show that G-Designer is still able to maintain its cost efficiency with more agents.

Weakness 1: What happens to the communication protocol if a graph has a directed cycle?

Thank you for the insightful inquiry! In Equation (15), we extract an enforced acyclic graph $\mathcal{G}\_{com}$ from the potentially cyclic topology $\tilde{\mathbf{S}}$. Specifically, in our provided anonymous code repository, the function check_cycle() in GDesigner/graph/graph.py is responsible for avoiding cycles when constructing $\mathcal{G}\_{com}$. We will emphasize this point more clearly in the revised version.

Weakness 2: Why are these 3 criteria considered?

1. Effectiveness: Extraordinary problem-solving is one of the core motivations behind MAS.
2. Complexity Adaptiveness aims to achieve optimal performance with minimal resource consumption. This is emphasized or pursued by many recent multi-agent papers, such as AgentPrune (ICLR 2025), RouterDC (NeurIPS 2024), and GraphRouter (ICLR 2025).
3. Adversarial Robustness: As attack & defense mechanisms in Agent/MAS gain increasing attention (e.g., TrustAgent (EMNLP 2024)), we believe safety and robustness will be critical for the trustworthy deployment of MAS in real-world applications.

We respectfully acknowledge that while we have highlighted what we consider critical, we do not claim this framework to be exhaustive. We would greatly appreciate any additional insights you might suggest and would be happy to incorporate them into our future work.

Weakness 3: The starting anchor is simple.

We would like to respectfully clarify that the anchor is not an "almost random guess". Instead, as stated in line 247, it incorporates prior knowledge of the workflow sequence (for example, a simple but standard coding procedure). In Section 5.4 and Table 3, we demonstrate that introducing the starting anchor results in a practical performance improvement.

Weakness 4: Equation 14 are not correctly referred.

Thank you for your thorough review! We will correct the reference to Eq. (14) in Lines 278–288 in the revised version.

Weakness 5 & 7: No details about the training of the proposed method are given.

• Training set: For both our method and the baselines that require training, we use the same training set. As mentioned in Line 321 (Right) of the paper: 40 samples for MMLU/MultiArith/HumanEval, 80 samples for GSM8K/SVAMP/AQuA;
• Test set: provided in Table 4;
• Hyperparameter tuning: We use a validation set of size 100 for each benchmark and apply a grid search to find the optimal parameters. To reduce the complexity of hyperparameter tuning for users, we provide a set of generalized hyperparameters in Section 5.1.

Weakness 6: The paper lacks proper motivation.

Thank you for bringing this issue to our attention! We humbly provide a more intuitive explanation of each component:

• Section 4.1: Makes it possible to process MAS with VGAE.
• VGAE encoder $q(\cdot)$: Captures the semantic relationships among different agents regarding a given task.
• VGAE decoder $p(\cdot)$: Directly leveraging complex MetaGPT/LLM-Debate/GPTSwarm will cause unnecessary costs on simple queries.
• Equation 14: If the optimized topology is not regularized, it may suffer from malicious edges or unnecessary edges. This practice has been well-validated in the traditional graph structure learning literature, such as NeuralSparse (ICML 2020), PTDNet (WSDM 2021).

Weakness 7: If the baseline approaches need additional training?

In the baselines of our paper, DyLAN, LLM-Blender, and GPTSwarm require training.

We compare with other training-free methods because all previously published training-required methods (including DyLAN (COLM'24), LLM-Blender (ACL'23) and GPTSwarm (ICML'24)) have compared with training-free methods. Therefore, we follow the same setting as theirs for a comprehensive evaluation.

Besides, we respectfully argue our approach has the following key advantages in a fair way:

(1) G-Designer has better performance than training-free/required topologies; (2) its training cost is an order of magnitude lower than all training-required methods (Table 2). Additionally, our inference cost is among the lowest (Figure 4).

Weakness 8: Clarify the details of all the agents used in Table 1.

The profiles and tools of different agents vary significantly, resulting in distinct encoded node features. Consequently, the constructed graph varies significantly. We provide a case study (Figure 6) to illustrate this. Examples of different agent role/tool descriptions can be found in our provided code in GDesigner/prompt/gsm8k_prompt_set.py.

Weakness 9: The variance of G-Designer is high in Table 1?

We would like to clarify a possible misunderstanding: the subscripts in Table 1 indicate improvements over the vanilla baseline rather than variance.

Besides, we would like to emphasize that all results in Table 1 are averaged over three runs.