Multi-agent Architecture Search via Agentic Supernet

We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1: Clarification on parameter $\phi$ The paper lacks clarity regarding the parameter $\phi$ in Eq(6), where "$Q\_\phi$ is parameterized by $\phi$."

Thank you for pointing out! The $\phi$ represents the parameters of the controller network $\mathbb{Q}\_\phi$, which is essentially composed of the sampling functions for each layer, $\pi\_l: q \rightarrow \mathcal{V}\_\ell$ (where the definition of $\pi_\ell$ is given in Eq. (9)). They are MoE-style networks that select the activated operators for the $\ell$-th layer based on the query. Therefore, $\phi$ can also be expressed as $\phi = \\{\pi\_1, \cdots, \pi\_\ell\\}$.

We hope this clarifies the nature of the controller network.

Weakness 2: The selection of threshold value What’s the reason the threshold value thres in Eq(9) is set as 0.3?

We supplement the parameter sensitivity analysis of threshold value in Eq. (9) as follows:

We observe that while increasing the threshold value leads to some performance gains, the improvement plateaus beyond 0.3. Additionally, a higher threshold increases inference costs due to the activation of more operators per layer. Therefore, we ultimately set $thres = 0.3$.

Weakness 3: Performance discrepancy There appears to be a discrepancy in the reported performance improvements.

Thank you for your detailed review! After thorough inspection, we found that 11.82% was a typo, as the main text and experimental results were not properly synchronized. It should be corrected to 16.89%, which is derived from the performance difference between MaAS and MacNet on MBPP dataset.

Weakness 4 the improvements highlighted in red are derived from comparisons with the Vanilla baseline in Table 1, while the actual improvements over “handcrafted or automated multi-agent systems” baselines such as AFlow appear modest.

Thank you for your insightful suggestion! We chose to present the improvements over vanilla LLMs in Table 1 following prior practices in GPTSwarm (ICML 2024) and AgentPrune (ICLR 2025), which adopt the same approach.

To address your concerns, we have:

1. Provided a version of Table 1 with standard deviations, replacing the subscript values with the standard deviations from three runs, as shown in Table1-stdev (https://anonymous.4open.science/r/maas-rbt/table1-stdev.png).
2. Explained the key advantages of MaAS over the SOTA baseline AFlow:
   • Although MaAS achieves moderate improvements over AFlow on certain benchmarks, it shows clear advantages on a broader set of benchmarks (e.g., +1.14% on GSM8K, +2.58% on MultiArith).
   • MaAS achieves these gains with significantly lower computational costs—its training cost is only 15% of AFlow’s, and inference cost is just 25%.

We sincerely hope this demonstrates MaAS’s superiority over AFlow in both cost efficiency and performance.

Minor Typos

Thank you for the meticulous review! We have repaired the mentioned typos in our revised manuscript.

We would like to express our sincere respect for your insightful review! In response to your comments, we have carefully prepared a point-by-point reply:

Essential References Not Discussed

Thank you for the valuable supplement! We have added this important citation in our revised manuscript.

Weaknessm 1: Additional benchmarks One thing I can suggest is to test on additional benchmarks such as ALFWorld, SciWorld, and ToolBench.

Thank you immensely for the instructive advice! Our performance on ALFWorld is summarized in the table below. Since ALFWorld involves multiple trials, MaAS samples an architecture for each episode, meaning the same agentic workflow is used across multiple trials. Other trainable baselines, such as GPTSwarm and AFlow, are implemented similarly.

As observed, MaAS achieves a performance improvement of up to 22.54% on the embodied task ALFWorld while maintaining a relatively low average cost, demonstrating its cost-effectiveness.

Sincere thanks for the thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments.

Weakness 1: Lack of variance estimation

Thank you for your insightful suggestion! In fact, all results in Table 1 represent the average of three runs. Following prior practices in GPTSwarm (ICML 2024) and AgentPrune (ICLR 2025), we chose to present the performance difference from the vanilla LLM in the main result table (rather than reporting standard deviations).

To further address your concern, we provide a version of Table 1 with standard deviations, as shown in Table1-stdev (https://anonymous.4open.science/r/maas-rbt/table1-stdev.png). We sincerely hope this resolves your concern.

Weakness 2 It seems the experiments are not cross-validated (a single train-test split) or randomized with multiple attempts.

Thank you! The results are reported as the average of three runs, and we have supplemented them with standard deviations in our response to Weakness 1.

Weakness 3: Clarification on Hyperparameters Could you pls clarify which benchmark task yield the optimal parameter set in Figure 7? And are the optimal parameters subject to change given a different task?

Thank you for your insightful inquiry! In fact, our proposed MaAS has not undergone extensive hyperparameter tuning, and the current setting ($L=4, K=4$) is not necessarily the optimal one in terms of performance. As shown in Figure 7, increasing $L$ from 4 to 8 leads to a 0.9% improvement in pass@1.

However, we have consistently observed the following trends across multiple benchmarks:

• Beyond $L=4$, the performance gain from increasing the supernet depth becomes marginal.
• Beyond $K=4$, the performance gain from increasing the sampling times also plateaus.

Above is the rationale behind our choice of $L=4, K=4$. Since our current sensitivity study is based on HumanEval, we further provide a sensitivity analysis on GSM8K to substantiate our findings:

Comment 1: Typos and Minors

Thanks immensely for pointing out! We will add the missing left parenthesis and underline the tied runner-up in Table 2.

Suggestion 1: Ablation & Sensitivity analysis on other benchmarks

Thank you for your valuable insights! In our response to Weakness 3, we have supplemented the sensitivity analysis for GSM8K. Due to time constraints and API resource budget, we were unable to promptly provide ablation/sensitivity analyses for all datasets. However, we sincerely commit to including the corresponding analyses for other datasets in the appendix of the camera-ready version. Once again, we truly appreciate your feedback!

Suggestion 2: More details on transferability and inductive analysis

Thank you for your valuable feedback! We sincerely commit to including additional details on transferability and inductive analysis in the appendix.

Question 1 How will the proposed framework scale with number of agentic operators? Could you elaborate more on the potential impact of search efficiency?

To address your question, we gradually increase the number of operators on HumanEval and report the performance, cost, and time-related metrics as follows:

As observed, with more operators, performance exhibits a steady improvement, while cost remains largely stable, and training/inference time does not increase significantly. We believe this demonstrates the efficiency advantage of the agentic supernet.

Question 2 In Figure 7, the performance of sampling times K does not monotonically increasing, e.g. dip at K=6. What's the possible reason and indication?

Your keen insight is truly appreciated! To further investigate this phenomenon, we conducted a finer-grained hyperparameter analysis on $K$, with results summarized in the table below:

The results indicate that when $K\geq4$, MaAS's performance stabilizes within the range of 91.20–92.20 on HumanEval, suggesting that the benefit of additional sampling saturates at $K=4$, with subsequent fluctuations remaining within a normal range. We hope this properly addresses your concern.

Weakness 1: Insufficient Clarity in Presentation

Thank you for the insightful comment! Each layer shares the same set of operators, except for the first layer, where the early-exit operator is excluded. The operators in each layer produce outputs in parallel, which are then concatenated and passed as input prompts to the next activated operator, following standard practices in Mixture-of-Agents (ICLR 2025) and GPTSwarm (ICML 2024).

Weakness 2: Lessons from DARTS+

Thank you for this highly insightful discussion and enlightenment!

• Collapse Issue We empirically demonstrate that the agentic supernet does not encounter the collapse issue observed in traditional DARTS, as presented in Table-collapse (https://anonymous.4open.science/r/maas-rbt/table-collapse.md).
• Overfitting Issue Intuitively, we argue that MaAS's agentic supernet is inherently resistant to overfitting due to two key factors: (1) Query-aware sampling. In fact, prior work on customizable NAS like GRACES [1], has demonstrated the advantage of input-dependent supernet in out-of-distribution generalizability. (2) Cross-domain training data. The training data itself can span multiple domains (e.g., GAIA benchmark includes web searching and file analysis), inherently promoting cross-domain generalization.
• Biased Sampling Issue We commit to incorporating this intriguing discussion in our revised manuscript, borrowing lessons from FairNAS/DARTS-/DARTS-PT.

[1] Graph Neural Architecture Search Under Distribution Shifts, ICML'22

Weakness 3.1: Transferability of agentic supernet

Following your suggestion, beyond the numerical transferability study in Table 8, we further visualize the underlying mechanism of MaAS’s transferability, with results and analysis presented in Figure-transfer (https://anonymous.4open.science/r/maas-rbt/figure-transfer.md).

Weakness 3.2: Cross-domain optimization of agentic supernet

We would first like to point out that, the GAIA benchmark we used inherently falls under cross-domain optimization (web searching + file reading). To further address your concerns, we report the results of training the agentic supernet under a math/coding cross-domain setting:

(M->MATH, G->GSM8K, H->HumanEval)

Notably, when trained on the HumanEval+GSM8K mixture, the performance surpasses that of training on HumanEval or MATH alone.

Weakness 4: Ablation on (1) supernet and (2) query-based sampling

To validate that the driving force of MaAS relies not only on query-based sampling but also on the agentic supernet, we construct a baseline called Agent-Archive, which consists of an agentic system archive populated with operators and agentic workflows. Results are in Table-archive (https://anonymous.4open.science/r/maas-rbt/table-archive.md).

Comment 1 How agentic supernet shifts the focus from cost reduction to adaptability

We respectfully state that MaAS's adaptability stems from its query-aware supernet sampling mechanism.

By analogy, MaAS removes DARTS's final one-shot pruning step that determines a fixed CNN. Instead, during actual usage, it dynamically customizes each layer’s kernel size/skip connections/pooling operators, as well as the network depth, based on each input (e.g., image). This allows MaAS to retain DARTS's advantage of reducing training costs while simultaneously enhancing adaptability in agentic scenarios.

Comment 2: Possibility of supergraph

Thank you for your inspiring thoughts! We will incorporate this interesting discussion in our updated manuscript.

Comment 3: Specifying underlying LLM of baselines

Thank you! This has been specified in Line 277 and Appendix C.2. Besides, in Table 2, GPt-4o-mini is used for TapaAgent/Sibyl and GPT-4 for AutoGPT.

Comment 4: Circle size in Figure 4

Thank you! The circle size in Figure 4 is proportional to the value of y-axis.

Comment 5: Removal of textual gradient in Table 4

After carefully reviewing Table 4, we identified a typo that exaggerated the impact of removing textual gradient on inference cost. Specifically, 0.09 should be corrected to 0.90. To provide a more detailed analysis, we present Table 4 with standard deviation included for finer observation (https://anonymous.4open.science/r/maas-rbt/table4-stdev.md). We believe the cost reduction occurs because textual gradient increases the length of prompts in MaAS.

Comment 6: Learning visualization of agentic supernet

Following your insightful suggestion, we have visualized the evolution of operator sampling trends as the sampling count increases (https://anonymous.4open.science/r/maas-rbt/figure-learning.pdf). It learns to avoid overly confident early stopping and instead prioritizes testing and self-refinement in deeper layers.