Self-Evolving Multi-Agent Collaboration Networks for Software Development

We sincerely appreciate your thoughtful comments. Below, we respond to each point in detail. If our responses adequately address your concerns, we would be truly grateful if you would consider raising your score. We also remain open to any further discussions that could contribute to enhancing the quality of our paper.

[Weakness 1] I truly believe that the mathematical explanation of the problem (mainly between lines 169-184) is unnecessary, even though I see that this somehow facilitates some later explanations. Although it provides some kind of generalization, the approach still relies on LLMs that are probabilistic by nature and then are not mathematically generalizable (i.e., the function is essentially a message sent to an LLM, so it does not behave exactly as a mathematical generic function as the authors may want to demonstrate). I advise the removal of the mathematical explanation. It can help sometimes but does not add value to the paper.

Answer: Sorry for the confusion. The mathematical formulation is incorporated as an objective, drawing an analogy to neural network optimization. This addition aims to help readers better understand the target proxy and its role in motivating the subsequent textual backpropagation-based self-evolving solution. To ensure greater precision and clarity, we will revise this mathematical section to highlight this analogy feature and avoid claiming a mathematical generic function.

[Weakness 2] The paper lacks an actual example/use case in the opposite (or complementary, if the authors decide to keep it) of the mathematical explanation. The authors use this type of explanation in lines 270-271. This kind of example can be used as a running example throughout the paper.

Answer: Thanks for the suggestions. We provide a detailed example to clarify each symbol and operation in Tab.4 in the updated appendix.

[Weakness 3] Typo and visualization suggestions.

Answer: Thanks for the suggestions. We will fix the typos and improve the visualization!

[Question 1] When explaining their approach (section 3.1), the authors did not mention problems regarding the context window of most LLMs. Since requirements can contain quite a large amount of textual data, is the approach capable of dealing with it without extra techniques (e.g., RAG)? If not, even though the authors did not highlight it as a problem, the context window limitations should be mentioned.

Answer: EvoMac can manage extensive requirements without relying on additional techniques. Unlike a single agent, EvoMAC is much less affected by the context window limitations.

1. EvoMAC achieves this through the multi-agent collaboration approach. Multi-agent collaboration effectively mitigates context window limitation issues by breaking down complex and lengthy requirements into smaller, more manageable subtasks. These subtasks fit within the context window of individual agents, enabling them to address specific aspects of the task. Gradually, the collective efforts of multiple agents allow EvoMac to fulfill the overall extensive task requirements.
2. Figure 7 shows that a single agent fails when requirements become too lengthy, due to the limited context window. In contrast, EvoMAC (powered by the same language model) maintains superior performance even as requirements lengthen. This demonstrates that the context window limitations do not severely impact EvoMAC, allowing it to effectively address longer requirements.

Reviewer sExs: if possible, can you reply to the rebuttal?

We sincerely appreciate your thoughtful comments. Below, we respond to each point in detail. If our responses adequately address your concerns, we would be truly grateful if you would consider raising your score. We also remain open to any further discussions that could contribute to enhancing the quality of our paper.

[Weakness 1] The benchmark proposed in this paper lacks data analysis and some basic statistical information, such as prompt length, the number of final generated files/functions, etc.

Answer: Thank you for the suggestions. We provide more detailed data analysis and statistical information in the following parts.

1. The data analysis of the benchmark is in the supplementary material: Table 1 and Figure 1. Note that the Supplementary Material is submitted as a separate PDF, which can be downloaded by clicking the down-arrow button labeled "Supplementary Material" (below the abstract). For convenience, we also provide a copy of Table 1 below.

   • Table 1 presents the number of software samples, test cases, and the average/maximum lengths of requirement prompts for both websites and games. We see that: i) our software-level benchmark is diverse, encompassing two common types of software (Websites and Games) as well as two levels of difficulty (Basic and Advanced); and ii) our software-level benchmark is notably challenging, with an average prompt length exceeding 500 on Game and 1000 on Website, making it almost 5 and 10 times longer than the function-level HumanEval benchmark separately.

   • Figure 1 illustrates the distribution of test case types across both categories. We see that our software-level benchmark includes 11 distinct test case categories and a total of 616 test cases, representing a scale 6 times larger than the function-level HumanEval benchmark. This extensive set enables a more focused and thorough evaluation of code generation capabilities.

Table 1: Basic statistics for website and game domains, including the amount of samples, the task prompt length (Average/Max), and number of test cases at both Basic and Advanced levels.

2. We provide more detailed statistical information in the table below, including code length, file count, and function count. We see that the generated code consists of over six functions across multiple files, showcasing advanced capabilities in generating complex, realistic code that requires function coordination and synchronization across files. This suggests that: i) our software-level benchmark closely reflects the advanced coding skills needed for real-world coding tasks, and ii) the proposed multi-agent collaboration system effectively supports the generation of more complex code.

[Weakness 2] The benchmark proposed in this paper is relatively easy, with the EvoMAC method already achieving around 90% accuracy.

Answer: To address the reviewer's concern, we would like to clarify that the benchmark is challenging from two aspects.

1. Our benchmark includes two difficulty levels: Basic and Advanced. While the Basic level achieves around 90% accuracy, the Advanced level proves challenging, with accuracy around 50%.

2. The Advanced level reflects more complex software functionalities, such as game logic rules and dynamic web content management. Passing each advanced test case is challenging and demands a fundamental improvement in coding capabilities. As meeting a single advanced requirement involves synchronized implementation and function calls across multiple files. For instance, fulfilling the advanced requirement of "eating a mushroom and earning points in Mario" requires checking the locations of the mushroom and Mario, adjusting the game score accordingly, and updating the visualization. It requires implementing multiple interdependent functions, demanding advanced coding capabilities to avoid conflicts.

[Question 1] Are there any issues with the citation format in the paper?

Answer: Thanks for pointing out. We will fix it.

[Question 2] Does the paper lack an appendix?

Answer: We apologize for any confusion regarding the appendix. The appendix has been submitted as a separate PDF and can be accessed by clicking the down-arrow button labeled "Supplementary Material" below the abstract.

Reviewer kPYh: if possible, can you reply to the rebuttal?

We sincerely appreciate your thoughtful comments. Below, we respond to each point in detail. If our responses adequately address your concerns, we would be truly grateful if you would consider raising your score. We also remain open to any further discussions that could contribute to enhancing the quality of our paper.

[Weakness 1] Existing studies, such as [1-4] have also explored automation in LLM-based multi-agent collaboration. Please compare the differences between EvoMAC and these works.

Answer: Thanks for the valuable suggestions. Following previous works[1,2,3,4], we model multi-agent collaboration as a network/graph and provide a detailed comparison in the table below. Compared to previous approaches, EvoMAC offers three distinct advantages.

1. EvoMAC jointly optimizes both nodes and edges, while previous approaches either rely on predefined, non-optimizable solutions [1,3,4] or support only separate optimization of nodes or edges[2]. Predefined collaboration structures are limited by human design, as a single structure cannot adapt to all scenarios. Separate optimization of nodes or edges is also suboptimal because both agent roles (nodes) and their connections (edges) are crucial to task completion. For example, in a coding task, each agent's role and the dependencies among agents are essential; if agents (nodes) are not properly optimized, subtasks may be incorrectly implemented, and if connections (edges) are not optimized, agents may conflict by modifying the same code, resulting in reduced performance. Our joint node and edge optimization offers greater flexibility, enabling more effective multi-agent collaboration.

2. EvoMAC uses external tools to provide informative, objective textual feedback. In contrast, existing methods [1, 2, 3] primarily rely on scalar feedback or offer no feedback at all [4], as textual feedback is often unavailable for most tasks. Textual feedback, however, not only validates the effectiveness of the system like scalar feedback but also identifies specific errors within the collaboration system and offers actionable guidance to improve multi-agent collaboration.

3. EvoMAC proposes a novel textual backpropagation method for system optimization while existing approaches rely primarily on heuristic design or scalar-based reinforcement learning (RL) techniques. Heuristic design lacks flexibility, leading to suboptimal performance. RL techniques rely on a single numerical value to optimize the entire complex multi-agent system, making optimization extremely challenging and demonstrating inferior optimization quality. Our novel textual backpropagation method leverages detailed textual feedback and strategically categorizes errors and modification suggestions for each node and edge, making optimization more attainable and demonstrating superior performance.

[Weakness 2] EvoMAC's updating process includes removing agents that have completed their tasks. Can the entire agentic workflow be replayed once a task is finished, or is the removed agent permanently excluded from further iterations?

Answer: Sorry for the confusion. We will provide more details in the revision. Specifically, the removed agent remains part of the agentic workflow. However, if its subtask is marked as completed, the agent will not be executed in that iteration. This allows the final agentic workflow to be replayed by executing all agents in the workflow as needed.

Reviewer aNRL: if possible, can you reply to the rebuttal?

[Weakness 3] RSD-Bench focuses on website and game software types, which may not comprehensively represent the diversity of real-world software development tasks. Expanding the evaluation to include other domains, such as enterprise applications or data processing software, would enhance the generalizability of the results.

Answer: To address the reviewer's concerns about RSD-Bench's diversity and EvoMAC's generalizability, we offer clarification from the following three perspectives.

1. Games and websites represent two typical software types, and the testing covers 11 distinct categories. These categories expand beyond previous benchmarks focused on function completion and bug fixing to provide a more comprehensive evaluation of coding capabilities. We are extending the benchmark to include additional software types.

2. EvoMAC is validated across three distinct software development tasks—function completion, website, and game, where it outperforms prior works, demonstrating its effectiveness and generalizability.

3. We extend the evaluation of EvoMAC to include data processing tasks by utilizing the InfiAgent-DABench[1] dataset (hard). The table below shows that EvoMAC enhances the performance in handling complex data analysis tasks, showcasing its effectiveness and generalizability to diverse software types.

[1] Hu, Xueyu, et al. "InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks.", ICML 2024

[Question 1] How sensitive is EvoMAC to the quality and specificity of feedback from unit tests? If the unit tests are incomplete or overly general, would EvoMAC still produce reliable code, or would it require stricter validation criteria?

Answer:

To address the reviewer's concerns, we provide clarification from the following three perspectives.

1. EvoMAC demonstrates relative robustness to unit tests. We randomly sampled 100 generated test cases and manually evaluated their accuracy, as shown in the table below. The results indicate that even when testing accuracy is around 50%, the feedback remains valid and contributes to measurable performance gains.

2. We manually evaluate testing performance within RSD-Bench, with the overall accuracy ranging from approximately 70% to 80%. These test cases effectively verify the completeness of requirements. Although they do not perfectly cover all code and requirements, they play a significant role in the evolution process, resulting in notable performance gains of 23.28%, 27.93%, 9.44%, and 9.67% on Web-Basic, Web-Advanced, Game-Basic, and Game-Advanced tasks, respectively.

3. To enable automatic verification of testing performance, we are developing a more comprehensive benchmark that includes ground-truth code. Incorporating ground-truth code will allow for automatic assessment of test case accuracy, as properly implemented code should pass all valid test cases unless the test cases themselves are flawed. This approach will facilitate a more thorough evaluation of testing capabilities and enable adaptive validation criteria, providing more accurate and complete feedback.

[Question 3] Can the self-evolving mechanism be applied to other domains outside software development? If yes, how?

Answer: Yes, our self-evolving mechanism can be applied to domains where informative feedback is available. For instance, in the auto-research domain [1,2], which focuses on developing AI agents for automated machine learning tasks, such as dataset creation, algorithm development, and model training. Here, the validation set serves as the target proxy, with feedback provided by validation performance. Analyzing results on the validation set offers valuable insights into the algorithm's effectiveness and can guide further modifications. The self-evolving mechnism iteratively adjusts algorithm generation, collects feedback, and analyzes results, promising advanced automated machine learning performance.

[1] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, OpenAI 2024 [2] MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

We sincerely appreciate your thoughtful comments. Below, we respond to each point in detail. If our responses adequately address your concerns, we would be truly grateful if you would consider raising your score. We also remain open to any further discussions that could contribute to enhancing the quality of our paper.

[Weakness 1] The EvoMAC framework introduces significant complexity with its multi-agent setup, dynamic adjustments, and textual backpropagation mechanism. This complexity may limit the framework's accessibility and implementation ease for real-world adoption outside of specialized research contexts.

Answer: Thanks for the time in reviewing. To address the reviewer's concerns about EvoMAC's applicability in real-world scenarios, we offer clarification from the following four perspectives.

1. In real-world scenarios, software development may require hundreds of engineers working collaboratively over months or even years to achieve completion. The primary focus within the software development community is on improving the effectiveness of the development process, advancing from function-level requirements to more extensive software-level requirements.
2. Multi-agent collaboration is a common practice in automatic software development [1,2], often involving tens or even hundreds of agents to enhance performance. EvoMAC's demonstrated effectiveness makes it well-suited to handling the complexities of real-world software development.
3. EvoMAC's approach, which includes a multi-agent setup, dynamic adjustments, and a textual backpropagation mechanism, is fully automated. It eliminates the need for human intervention or specific heuristic designs, ensuring it is highly adaptable to real-world scenarios.
4. EvoMAC performs effectively even when powered by smaller 14B models, as shown in the table in Weakness 2 and Question 2. This makes it feasible for deployment in resource-constrained real-world environments, offering an efficient solution without compromising performance.

[1] Zhuge M, Wang W, Kirsch L, et al. GPTSwarm: Language Agents as Optimizable Graphs[C]//Forty-first International Conference on Machine Learning. [2] Qian C, Xie Z, Wang Y, et al. Scaling Large-Language-Model-based Multi-Agent Collaboration[J]. arXiv preprint arXiv:2406.07155, 2024.

[Weakness 2 and Question 2] Although EvoMAC performs well with large models like GPT-4o-Mini, its performance with smaller or less capable models is unclear. This reliance may restrict its applicability, particularly in environments with limited computational resources. Can EvoMAC work effectively with models of different sizes, or does it rely on the power of high-capacity LLMs? Would it perform satisfactorily with smaller models that might be more efficient in constrained environments?

Answer: Thank you for your time and valuable suggestions. To address the reviewer's concerns regarding its applicability in constrained scenarios, we equip EvoMAC with smaller models, including Qwen2.5-Coder-7B-Instruct and Qwen2.5-Coder-14B-Instruct, and the performance results are shown in the table below.

1. EvoMAC performs effectively with models of varying sizes, ranging from smaller models like 7B and 14B to GPT-4o-mini. Regardless of the model used, EvoMAC consistently outperforms single-agent systems, highlighting its generalizability and effectiveness.

2. EvoMAC's effectiveness does not demand very high-capacity LLMs. For instance, the 14B model achieves performance gains of 28.36% and 29.94% on Website Basic and Advanced tasks, respectively, comparable to GPT-4o-mini's gains of 26.48% and 20.26%. However, the LLM capacity must not be too limited; the 7B model with performance 24.32% and 7.57% shows more modest improvements.

Reviewer 32VP: if possible, can you reply to the rebuttal?