MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems

Thanks for your time devoted to reviewing our paper. We are glad to see that you acknowledge that our method is novel, experimental designs are sound, and our paper is insightful. It is encouraging. We would like to address your remaining concerns in the following.

---

 

Experimental Designs Or Analyses 1: Lack of Single Agent Framework Specification: The paper does not specify the framework employed for the single-agent baseline models.

Answer: Sorry for the missing details. We employ vLLM library to deploy LLMs. For all baseline methods, we run their official open-sourced codes.

---

 

Experimental Designs Or Analyses 2: Cost Comparisons and Metric Justification. I have reservations regarding the cost comparison metric used in this experiment. To the best of my knowledge, standard practice in evaluating LLM cost involves assessing token cost and time cost to quantify both financial and temporal overheads. Therefore, I recommend that the authors augment this section by providing token cost and time cost data for MAS-GPT and all baselines across the benchmarks used. Furthermore, the methodology for performance measurement in this cost comparison section, as well as the source of the experimental data, are not clearly delineated. Please provide this essential supplementary information.

Answer: Thanks for the recommendation! Following your advice, we have reported the token consumptions in addition to the number of LLM calls in the following table. In our initial experiments, we did not record token consumptions, so we need to conduct additional experiments during rebuttal. Due to limited time, we report the following results . We will report more in our revision. From the table, we see that our MAS-GPT achieves the best performance with the least inference costs (both call times and token consumptions).

The performance measurement uses accuracy as metric, which is averaged on all benchmarks in Table 2. And the specific numbers are taken from Table 2. We will include all of these in our revision.

---

 

Experimental Designs Or Analyses 3: Scaling Effects of Data Size and Supervised Fine-tuning: The "Scaling effects of data size" analysis appears inconsistent with established supervised fine-tuning methodologies. It is widely understood that fine-tuning a 32B parameter LLM with only 100 data samples is generally insufficient to achieve meaningful adaptation. Consequently, the experimental results presented in Figure 5(a) may not adequately support the conclusions drawn in this section.

Answer: Thanks for this valuable comments. We would like to answer from two perspectives.

(1) The base model here is a 32B-Instruct model, which has coding capability at some levels. Therefore, training on 32B-Instruct models to learn to generate MAS code may have difference compared to training 32B-pretrained models to learn instruction-following capabilities.

(2) Despite that Figure 5 (a) shows that using 100 training samples could significantly reduce the error rate of execution, it does NOT indicate that using 100 samples is sufficient. Please note that in Figure 5 (b), using 100 training samples results in significantly lower perfomance compared to using 10000 samples. That is, using 100 samples may successfully teach the LLM to generate executable MAS code, but fails to teach it to generate appropriate MAS, which is essential in ensuring good performance.

We truly believe in this new paradigm. With more diverse and high-performance MAS being included, we believe that MAS-GPT will be further advanced in a way similar to the advancement of ChatGPT: with better data and training techniques, the models become better.

---

 

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

We are glad that you acknowledge that our idea is novel, method is comprehensive, and direction is aligned with a broader trend toward developing more streamlined, scalable, and adaptive architectures. Let's address your remaining concerns!

---

W1: Representing MAS as python code neglects important aspects of agent functionality...?

A: Sorry for the missing details! Our MAS-GPT actually supports functionalities mentioned by you (tool and multi-turn interactions) and they are indeed compatible with code.

(1) Code execution tool is implemented as a python function execute_code(code_text), which outputs the execution results to other agents. More tools can be implemented similarly.

(2) While single-turn LLM call is represented by a function call_llm(prompt), multi-turn interaction is represented by multi_turn(history, prompt).

Representing MAS as python code is a promising direction, as most functionalities in our AI community can be represented by python code. Along this new direction, we believe that with more data, tools, MAS, MAS-GPT could be continuingly improved as that from GPT-3.5 to GPT-4.

---

W2: Discussion on AgentPrune.

A: Thanks for recommendation! We will include these discussions and cite it.

(Methodology) The goals of our paper and AgentPrune are significantly different. We aim to optimize MAS-GPT once on diverse domains such that the MAS-GPT can generalize to diverse domains. While AgentPrune aims to optimize on one validation dataset such that it can work on the corresponding test dataset. There are also two recent papers with goals similar to AgentPrune: GPTSwarm(ICML2024 oral), AFlow(ICLR2025 oral). While these three require re-optimizing for each test set, MAS-GPT can generalize to diverse test sets without re-optimizing. Since in practice, users will not provide several examples in advance for optimization, we believe that MAS-GPT is a promising direction to truly make MAS practically applicable.

(Experiments) We run the official code of AgentPrune on HumanEval and MMLU. From the table, we see that our method performs much better with less inference cost.

---

W3: Comparison with ADAS.

A: Thanks for the advice.

(1) The reason why we compare with AFlow rather than ADAS is that AFlow is an improved follow-up of ADAS.

(2) We compare MAS-GPT with ADAS (optimized on GPQA). MAS-GPT performs significantly better than ADAS on these diverse datasets at a much lower cost, indicating that MAS-GPT is much more generalizable.

---

W4: The performance gain seems modest?

A: Thanks for the comments. We would like to respond from the following perspectives.

(1) MAS-GPT is the only method that consistently achieves better performance than SINGLE. Here, we report the percentage of benchmarks where a method outperforms a single agent.

(2) On challenging datasets such as AIME-2024 and GSM-H, our method achieves significantly better performance than SINGLE (16.67 on AIME-2024), demonstrating substantial potential.

(3) We agree that for some benchmarks, improvements are not significant. Please note that these patterns commonly exist in all methods. The reasons why we still report them are twofold. First, these benchmarks are commonly used in MAS literature and we follow their setups. Second, we want to show that our method is generalizable on diverse datasets.

---

Q1: Are there cases where the MAS incorporates if-else structures...?

A: Yes, we have observed interesting workflows. For example, we see MAS with if-else structures in several cases. (1) Agents evaluate the current status and determine whether to stop or continue solving. (2) If code executor shows that the code runs correctly, then the system ends; else the system continues.

---

Q2: "code test agent"?

A: Yes. We verify the generated code by actually executing it. We implement two related python functions: one takes code piece as input and one takes code piece together with test cases as inputs.

---

Q3: Number of query-MAS pairs? A: Thanks for the advice. The selection process will filter out those queries that all MAS fail to correctly answer. The refinement process essentially replace those original query-MAS pairs(Line 256-258).

---

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

def execute_code(code):
    if not code:
        return "Empty code. No output."

    temp_dir = tempfile.mkdtemp()

    output_dict = {"output": None, "stdout": None, "error": None}

    def run_code():
        try:
            global_vars = {}
            local_vars = {}

            # Write the code to a temporary file
            with open(os.path.join(temp_dir, "script.py"), "w", encoding="utf-8") as f:
                f.write(code)

            # Capture standard output
            stdout_capture = io.StringIO()
            with contextlib.redirect_stdout(stdout_capture):
                exec(code, global_vars, local_vars)  # Execute code

            output_dict["stdout"] = stdout_capture.getvalue().strip()  # Capture print() output
            output_dict["output"] = local_vars.get("output", "None")  # 'output' variable

        except Exception as e:
            output_dict["error"] = traceback.format_exc()

    if output_dict["error"]:
        return f"Error:\n{output_dict['error']}"

    return f"Final output: {output_dict['output']}\nPrint during execution:\n{output_dict['stdout']}"

- `LLM(model_list)`: a class that represents an LLM with the given model list, with two available functions: call_llm(self, prompt) and multi_turn(self, history, prompt).
- `execute_code(code)`: a function that executes the given code and returns the output.
- `test_code_get_feedback(code, test_cases)`: a function that tests the given code with the test cases and returns the feedback.
- `get_function_signature(llm, taskInfo)`: a function that returns the generated function signature for the given task.
- `get_test_cases(llm, taskInfo, function_signature)`: a function that returns the generated test cases for the given task and function signature.
- `extract_code_solution(solution)`: a function that returns the code by extracting (wrapped within <Code Solution> and </Code Solution>) from the given solution.
- `generate_and_extract_code(llm, prompt, temperature=None)`: a function that returns the generated response and the extracted code from the response.

We are glad to see that you agree with our motivation and that our approach is novel. We notice that your main concern is on the experimental setups, of which we believe is caused by some confusion. (Since the comments in Claims are overlapped with Weaknesses, so we focus on the latter)

---

 

W1: The authors trained MAS-GPT and evaluated it within the same domain?

Answer: Sorry for the confusion.

(1) We would like to highlight that we do NOT 'train MAS-GPT and evaluate it within the same domain'. In contrast, we train MAS-GPT on diverse domains simultaneously and evaluate them on diverse domains without re-training. For example, during evaluation, GPQA (graduate-level QA), SciBench (college-level scientific problems), AIME-2024 (mathematical competition) are all out-of-domain benchmarks and much harder than the training data.

(2) Following your advice, we report performance results in the medical domain, MedQA (there is no medical dataset in training). From the table, we see that MAS-GPT is indeed generalizable.

(3) Compared to many optimization-based methods, MAS-GPT is indeed more generalizable. GPTSwarm (ICML 2024 oral), AFlow (ICLR 2025 oral), and AgentPrune (ICLR 2025) are all benchmark-dependent methods. That is, to test their methods on one benchmark, they need to firstly optimize on a subset (with GT labels) of that benchmark. In contrast, our MAS-GPT is optimized on diverse training data and then generalize to many benchmarks without modification.

---

 

W2: The performance difference between "Single" and proposed method is not obvious?

Answer: Sorry for the missing details. We run twice. We would like to emphasize from three perspectives.

(1) MAS-GPT is the only method that consistently achieves better performance than SINGLE. Please refer to the following table, where we report the percentage of benchmarks where a method outperforms a single agent.

(2) On particularly challenging datasets such as AIME-2024 and GSM-H, our method achieves significantly better performance than SINGLE (e.g., 16.67 on AIME-2024), demonstrating substantial potential.

(3) We agree that for some benchmarks, improvements are not significant. Please note that these patterns commonly exist in all methods. The reasons why we still report them are twofold. First, these benchmarks are commonly used in MAS literature and we follow their setups. Second, we want to show that our method is generalizable on diverse datasets.

---

 

W3: Improvements originate from manually designed MAS?

Answer: Thanks for the comments. We would like to address your concerns from two perspectives.

(1) Following your advice, we compute the number of unique MAS during test time (GSM-Hard, MATH, and SciBench) compared to those in training data. This strongly verify that MAS-GPT does NOT essentially memorize the training data.

(2) It is NOT a bad thing if MAS-GPT sometimes generates MAS that exists in the training data. The key of training MAS-GPT is making it learn to generate appropriate MAS based on each specific query. The keyword here is appropriate but not new. This is actually similar to the training of LLMs, LLMs could generate identical sentences seen during training and also new sentences as long as they are appropriate.

---

 

Q1: How many runs?

Answer: We performed 2 runs. See W2.

---

 

Q2: Why did the authors train and test MAS-GPT within the same domain? When claiming that MAS-GPT is efficient, did the authors account for the development and training time of MAS-GPT itself?

Answer: We did NOT train and test MAS-GPT within the same domain. Please refer to responses in Weakness 1.

Please allow us to emphasize our focus on this paper. Our ultimate goal is to make MAS-itegrated application effective and efficient during inference. To achieve this, we consider a brand-new paradigm: training an LLM. This LLM (i.e., MAS-GPT) is trained on diverse data and can generalize to diverse scenarios, making it a step closer to this goal. Training is one-time while inference could be endless (just like OpenAI trains GPT-4 for one time and serves the world countless times). Training-time efficiency is not the focus of this paper, which could be a potential future direction.

As an analogy, in efficient LLM research, their goal is to ensure the trained model enables LLMs to be efficient in applications (inference), rather than focusing on training efficiency. Our work shares a similar objective.

---

 

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

We are glad to see that you acknowledge that our approach is novel, methodologically sound, flexible, and scalable. We are sorry that we leave you some concerns and let's clarify.

---

W1&Q2: The generalization capability of MAS-GPT across significantly different or novel task domains remains unclear...

Answer 1: Our MAS-GPT can generalize well to new tasks and we have put efforts to verify this point in our paper. Sorry for potentially missing details.

(1) SciQ and SciBench are significantly different, where SciQ is a knowledge-based benchmark (multiple-choice, https://huggingface.co/datasets/allenai/sciq) while SciBench is a reasoning-based benchmark (non-choice, https://huggingface.co/datasets/xw27/scibench). Meanwhile, SciQ is a dataset for 4th grade exams while SciBench are all college-level scientific problems, which are much challenging than those in training data.

(2) Our paper also include results on GPQA (graduate-level QA) and AIME-2024 (mathematical competition), which are all out-of-domain and much harder than training data. Here, we additionally test on medical domain, MedQA (there is no medical dataset in training). From the table, we see that MAS-GPT is indeed generalizable.

(3) We kindly remind the reviewer that our MAS-GPT has achieved a better generalization capability compared to existing optimization-based methods: MAS-GPT does not require re-optimizing when being applied to different benchmarks. For methods such as GPTSwarm (ICML 2024 oral) and AFlow (ICLR 2025 oral), given a benchmark (e.g., MATH), these methods first optimize on a subset and then could only infer on the corresponding test set.

---

 

W2: The generated MAS topologies presented are relatively straightforward...

A: Thanks for the comments.

(1) The reason why we develop the MAS topologies at the current complexity is straightforward: the current complexity level is somewhat sufficient to achieve a pleasant performance on most of the benchmarks. All benchmarks are commonly used by MAS community and we follow their setups. If we continue to increase the complexity, it might be hard to achieve a good cost-performnace balance.

(2) Our approach to training MAS-GPT is scalable and methodologically capable of supporting sophisticated topologies. To achieve this, one only needs to design more sophisticated MAS and include them into the MAS pool. Based on this, the trained MAS-GPT would be able to generate sophisticated MAS for complex queries. Meanwhile, MAS-GPT already supports using tools such as code executor, indicating its potential of scalability to include more tools to handle complex tasks.

(3) The main contribution of this paper is pointing out a new direction for the MAS community. Like the training of LLMs (e.g., GPT-4), there are always cases that the trained MAS-GPT cannot solve no matter how well the current MAS-GPT is trained. That is, the current MAS-GPT is not the end. Similar to the continuing advancement of LLMs, our MAS-GPT could be continuously improved as the community designing more sophisticated (or better) MAS, tasks, and data samples.

---

Q1: If two MAS share an identical topology but differ in the instructions to the agent, are they counted as separate MAS instances?

Answer: Sorry for the confusion. Yes, they are counted as separate MAS instances. The reasons for such criteria is that it is hard to automatically and merely distinguish topologies.

---

Q3: What are the costs for generating the training data in terms of GPU hours and costs for calling APIs?

A: Collecting the training data roughly requires 245k calls of open-source LLMs. The training process takes roughly 32 GPU hours for training 32B-sized models. The refinement process roughly costs 143 US dollars for calling GPT-4o. Please note that MAS-GPT (32B) is only required to be trained once and applied to handle diverse queries without re-training both MAS-GPT and the LLMs that drive the MAS.

Meanwhile, we will open-source all of the data at every step (e.g., before and after filtering) and models to facilitate future research.

---

Q4: Did you see new topologies?

Answer: We have manually checked some of the generated topologies and found that MAS-GPT could generate novel topologies. Please refer to our case study of Case 3 (Section A.3). Meanwhile, please note that even if the generated topologies are the same as existing topologies, MAS-GPT would assign appropriate prompts (instructions) to the agents within the topologies, making them query-specific and appropriate MAS. For example, during the inference of MAS-GPT on GSM-Hard, 925 out of 1000 generated MAS are unseen from the training data, indicating that MAS-GPT is generating query-specific appropriate MAS.

---

 

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

Thanks for your appreciation of our idea and motivation. We would like to address your remaining concerns in the following.

---

 

Methods: The motivation for choosing these 40 MAS pools is unclear.

A: Sorry for the confusion. Let's clarify.

(1) The key motivation for choosing these 40 MAS in the initial pool is for teaching the LLM the basic format for representing a MAS. These 40 MAS cover basic elements such as chain-of-thought prompts, role-playing prompts, LLM-calling functions, and functions to get code execution results. As the base LLM does not know our representation of MAS (verified by poor performance in Figure 5 b, N=0), including these basic elements is critical to teach the LLM our MAS representation.

(2) MAS-GPT is not simply selecting MAS. For example, during the inference of MAS-GPT on GSM-Hard, 925 out of 1000 generated MAS are unseen from the training data, indicating that MAS-GPT is not merely selecting MAS but generating query-specific appropriate MAS.

(3) These 40 MAS constitute only a subset of the full training data. Actually, these 40 MAS only serve as seed MAS. They will be evolved during the query-MAS pair refinement process, which indeed enlarges the pool with some random systems.

We will include these in the revision.

---

 

Exp1: The authors compare the cost by computing the number of the inferences ... better to include token cost.

A: Thanks for the suggestion. Due to limited time, we report the following results and report more in our revision, where we see that MAS-GPT achieves the best with the least token cost.

---

 

Exp2: It is fairer to compare the performance of the cross-domain or zero-shot capability.

A: Thanks for the suggestion. Acturally, we have compared the performance of the cross-domain or zero-shot capability on datasets such as HumanEval, HumanEval+, GPQA, SciBench, and AIME-2024.

Please note that all of these datasets are only used for evaluation, but not training. We would like to highlight that GPQA (graduate-level QA), SciBench (college-level scientific problems), AIME-2024 (mathematical competition) are all much harder than the training data. We also additionally conduct experiments on MedQA (medical domain). We can see that our method achieves the best performance in these cross-domain setups. These experiments also verify the generality of our MAS-GPT: it could generalize to domains unseen in training data and to much harder queries than those in training.

---

 

Exp3: the baselines are largely included in the system. For instance, the CoT is inside the 40 MAS pool. ... It would be better not to include the baseline in the pool to see the filter-out performance...

A: Thanks for the comments. We would like to answer from three perspectives.

(1) Firstly, our initial MAS pool only includes a few baselines with simple operation such as Co; but does not inlcude complicated baselines such as DyLAN. We include these simple elements to teach the LLMs to generate MAS in our desired format (also see response to methods).

(2) Secondly, we would like to kindly inform the reviewer that including existing baselines into our initial MAS pool does not conflict with the rationale and motivation of MAS-GPT. One exciting and promising advantage of MAS-GPT is that it could stand on the shoulders of giants. Ideally, if we could include all existing high-performance MAS methods into the MAS pool, in real-world applications, we only need to deploy MAS-GPT to solve diverse user queries, rather than deploying multiple MAS and designing complicated rules to select them.

(3) Following your suggestions, we exclude CoT from the MAS pool and re-run the experiments. We report the results in the following. From the table, we see that excluding CoT performs comparably with the current version. This result is reasonable because the performance of CoT is normal such that during the process of pair evaluation and selection, CoT is less likely to be paired with queries, resulting in few CoT samples in the training data.

We truly believe in this new paradigm. With more diverse and high-performance MAS being included, we believe that MAS-GPT will be further advanced in a way similar to the advancement of ChatGPT: with better data and training techniques, the models become better.

---

 

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.