MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution

Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., drawing on the software engineering idea well, drawing on the cooperation method of the human team, and the improved performance). We are sorry for the confusion and unclear expression in the previous version. We have addressed each of the comments and suggestions. Please refer to our responses below for details.

Q1: The paper uses 11 prompts in total, and the specific prompt content cannot be found in the main text, which makes people worry whether the experimental results will strongly depend on the quality of the designed prompts. After replacing other models, the effect may be uncontrollable.

Thanks for your comments.

• Prompt content: We will provide the full set of prompt content in the revision (part of it is shown below due to the length limit).

# Prompt <em>P</em> (Line 5 in Algorithm 2)
system_prompt = ("You are a software development manager."
                 "Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. "
                 "This guidance should be based on the details provided in the issue description and the existing content of the code file.")
user_prompt = ("Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\n"
              f"# Issue Description:\n{issue_description}\n# Code File:\n{file_content}\n"
               "Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.") 

• More base LLMs: Moreover, we replaced GPT-4 with other two LLMs (i.e., DeepSeek [1*] and Llama-3.1-405B [2*]), and the experimental results on SWE-bench Lite are shown below. Please note that all prompts are identical to those we experimented with on GPT-4. In the "Directly Use" setting, the prompts are sourced from SWE-bench, while prompts for other settings are designed by us.

The above table shows that our method has achieved a 38-fold performance improvement (DeepSeek) and a 12-fold improvement (Llama 3.1) compared to directly using these base LLMs. The performance improvement validates that our method is general and can still unlock the potential of other LLMs in solving GitHub issues.

---

Q2: The framework is relatively complex and requires a large number of interactions with LLMs. In the end, compared with other baseline scores, whether these additional computing resources are necessary is not further analyzed in the paper.

Thanks for your comments. Compared with direct usage, LLM-based multi-agent systems [3-5*] including ours can be more complex and need many interactions. However, these additional computing resources are considered to be necessary and worthwhile because (1) The issue-resolving task is not a task that needs immediate resolution (in contrast with coding tasks such as IDE code completion) (2) The additional computing enables the model to resolve issues that were previously unresolvable (with the resolved rate increasing from 1.74% to 13.94%, as shown in Table 2). As direct usage of LLM can resolve few GitHub issues, this paper focuses on how to better use LLM to resolve issues and improve performance.

---

Q3: SWE-bench clearly states that the hints field should not be used in the submission list, but from this paper, it is found that hints seem to be the key to MAGIS improvement. Is this unfair to other methods?

Thanks for your comments. We reported the score (10.28%) under the setting without hints in Table 2. This score shows that our framework performs approximately six times better than the base LLM, GPT-4, thus validating the effectiveness of our method even in the absence of hints. The fairness of comparisons with other methods depends on whether these methods can access external information. For instance, in the design of OpenDevin [3*], there is a web browser (Browsing Agent), which allows it to obtain content from the web and these hints are available in GitHub before the issue resolution process begins.

---

Q4: I noticed that there are a lot of new submissions on SWE-bench (Lite) recently. Do you need to do a new comparison? (Of course, I think it is reasonable not to compare, after all, they are working in the same period.)

Thanks for your comments and understanding! we are aware of the recent submissions on SWE-bench (Lite) and we compared the methods (Swe-Agent [4*], Autocoderover [5*]) that submitted their results before the submission deadline of NeurIPS 24 in Appendix D. The experimental results are shown in Table 4 on Page 18. The main difference between our method from others is that MAGIS does not use external tools such as web browsers [3*]. Our paper focuses on how to unlock the potential of LLMs for GitHub issue resolution and we conduct many empirical studies on analyzing the limitations of LLMs. Moreover, thanks for your references, we will incorporate them [3,5-7*] into discussion.

---

References
[1*] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.
[2*] Dubey, et al. The Llama 3 Herd of Models. 2024.
[3*] Wang, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. 2024.
[4*] Yang, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. 2024.
[5*] Zhang, et al. Autocoderover: Autonomous program improvement. 2024.
[6*] Chen, et al. CodeR: Issue Resolving with Multi-Agent and Task Graphs. 2024.
[7*] Ma, et al. How to Understand Whole Software Repository?. 2024.

[1] Zhang, et al. Autocoderover: Autonomous program improvement. 2024.
[2] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint (2024).

Dear Reviewer,

Thank you very much for your follow-up feedback. We would like to address your concerns:

• Contemporaneous Work: The two papers you mentioned are contemporaneous works with our submission, in line with the conference guidelines (details shown below). Both papers were published on arXiv (8 Apr and 6 May) within two months of the NeurIPS submission deadline (22 May). As such, they fall under the category of contemporaneous work, which is why we did not include them as baselines in the main body of our paper. Meanwhile, we have already cited (AutoCodeRover[61] and SWE-Agent[59]) and discussed the comparison with these works in Appendix D (Line 718-729). Given your concerns, we will further expand our discussion in the Appendix to provide a more detailed comparison (part of it is shown below).

For the purpose of the reviewing process, papers that appeared online within two months of a submission will generally be considered "contemporaneous" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work. Authors are still expected to cite and discuss contemporaneous work and perform empirical comparisons to the degree feasible. Any paper that influenced the submission is considered prior work and must be cited and discussed as such. Submissions that are very similar to contemporaneous work will undergo additional scrutiny to prevent cases of plagiarism and missing credit to prior work.
Reference: https://neurips.cc/Conferences/2024/CallForPapers

• Performance and Cost Comparison: As of the conference submission deadline, the latest score reported by AutoCoderRover was 16.11%, as noted on page 7, second-to-last paragraph of their paper (https://arxiv.org/pdf/2404.05427v2). This score is not higher than the 16.67% we reported. Regarding SWE-Agent, it indeed reported an 18.00% score, which is higher than our ablation version (w/o hints). Thanks for your great suggestions, we reviewed the cost data on SWE-bench Lite and calculated that the average cost per instance for our method is approximately $0.41, which is significantly lower than the $1.67 reported by SWE-Agent. Considering the trade-off between effectiveness (a 1.33% difference in score) and cost (a fourfold reduction), we believe that our approach remains competitive. We acknowledge the contributions of SWE-Agent and AutoCoderRover, as they, along with our work, collectively advance the state of the art in this task.

Thank you once again for your valuable time.

Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., intuitive and effective method, massive experiments, and insights for the research community). We are sorry for the confusion and unclear expression in the previous version. We have addressed each of the comments and suggestions. Please refer to our responses below for details.

W1: The reproducibility of experiments for baseline models like GPT-4 and Claude is unclear. The code and data are temporarily unavailable, hindering follow-up and comparison by other researchers.

Thanks for your comments. We will make the code and data publicly available.

---

W2: Typos: There are minor typos, such as "multi-agennt" instead of "multi-agent" on page 2, line 63, and some missing "%" signs in the resolved ratio on page 7.

Thank you for carefully spotting the typo. All these typos will be corrected in the revision.

---

Q1: In Section 2, I'm a bit confused about the calculation of coverage ratio. How is $[s_i, e_i]$ defined for pure 'deleting' and 'adding' operations? If many code adjustments occur in a single file, does it ensure that each adjustment contributes independently to the coverage? I mean, if the model generated code is different from the reference code, the position behind is not aligned.)

We are sorry for the unclear presentation. ${[s_i, e_i]}$ indicates the range of lines in the file that have been modified, and the line numbers within this range are based on the file before the modifications are made. Therefore, ${[s_i, e_i]}$ should represent the whole file (i.e., ${[1, LastLineNumber]}$) for pure 'deleting' while it should be none (i.e., $\emptyset$) for pure 'adding'. For cases where a single file has many modifications, the line numbers are aligned based on Git, which uses the Myers diff algorithm [1*] as default. The empirical study [2*] demonstrates that the Myers diff algorithm can ensure that each adjustment contributes independently to the coverage in most cases.

---

Q2: In Section 4, can you kindly provide more key implementing details, especially for the model comparison experiment in Table 2? Some helpful information may be the core instruction/input for each model, the number of interaction rounds, and whether code interpreter is allowed. Those are all related to the performance of the model.

Thanks for your suggestions. We will add the specific prompt content and make the implementation clearer in the revision (part of it is shown below due to the context limit).

Specifically in Table 2, the prompts for directly using the LLMs are sourced from SWE-bench[3*] while other prompts for MAGIS in different settings are designed by us. For each GitHub issue, our method interacts with the issue only once, which means the framework only generates one final result for an issue. Moreover, the code interpreter is not allowed to be used in our method.

# Prompt <em>P</em> (Line 5 in Algorithm 2)
system_prompt = ("You are a software development manager."
                "Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. "
                "This guidance should be based on the details provided in the issue description and the existing content of the code file.")
user_prompt = ("Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\n"
            f"# Issue Description:\n{issue_description}\n# Code File:\n{file_content}\n"
            "Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.") 

---

Q3: Do you submit (or plan to submit) the results to SWE-Bench official leaderboard? How does this method differ from other agent-based methods?

Thanks for your comments. We plan to submit the results after the anonymous review period ends. The main difference between our method from others is that MAGIS does not use external tools such as web browsers [4*]. Our paper focuses on how to unlock the potential of LLMs for GitHub issue resolution and we conduct many empirical studies on analyzing the limitations of LLMs.

---

References
[1*] Myers, Eugene W. An O (ND) difference algorithm and its variations. Algorithmica 1.1 (1986).
[2*] Nugroho, Yusuf Sulistyo, et al. How different are different diff algorithms in Git?. EMSE 25 (2020).
[3*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint (2024).
[4*] Wang, Xingyao, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint (2024).

Dear Reviewer,

 

Thank you for your follow-up feedback and for recognizing the value of our work. We appreciate your support and the "weak accept" recommendation.

We would like to clarify that our proposed method does not rely on any external tools (such as code interpreters or web browsers) and the external tools are not used in the evaluation in our paper. To address your concerns about the ease of building upon our work, we will release all relevant code, and detailed implementation instructions after the paper is accepted. This will ensure that future researchers can easily reproduce and extend our empirical findings, fostering further developments in this area.

Once again, thank you for your constructive feedback.

 

Best regards,

The authors.

Thank you very much for taking the valuable time to review our manuscript and thanks for your positive comments (i.e., the simplicity and effectiveness of our method, and the effectiveness in various settings). We are sorry for the confusion and unclear expression in the previous version. We have addressed each of the comments and suggestions. Please refer to our responses below for details.

W1: The main experiments are limited, as only GPT-4 is measured

Thanks for your comments. To further validate the effectiveness of our method, we conducted more experiments on other base LLMs. The new experiments use DeepSeek (DeepSeek-V2-0628) [1*] and Llama 3.1 (405B) [2*] in addition to GPT-4 as the base model. The corresponding results are shown below,

Please note that all prompts are identical to those we experimented with on GPT-4. In the "Directly Use" setting, the prompts are sourced from the SWE-bench, while prompts for other settings are designed by us. The above table shows that our method has achieved a 38-fold performance improvement (to DeepSeek) and a 12-fold improvement (to Llama 3.1) compared to directly using these base LLMs. The improvement in effectiveness validates that our method is general and can still unlock the potential of other LLMs in solving GitHub issues.

---

W2: Baselines such as SWE-agent are missing

Thanks for your comments. The comparison with SWE-agent [3*] is discussed in Appendix D as SWE-agent is a contemporaneous work (the paper is publicly available at arXiv in May 2024). As shown in Table 4, our method achieved the resolved ratio of 25.33% on SWE-bench Lite, which is higher than the 18.00% reported by SWE-agent.

---

Q1: SWE-bench Lite results (appendix D) should be in the body of the paper

Thanks for your advice. We will move the Lite results to the body in the revision.

---

Q2: Is ablation of the kickoff meeting possible? This is an interesting mechanism but it's hard to understand whether it is effective or not

Thanks for your comments. The kickoff meeting serves as a transitional link. The minutes of this meeting are converted into a code representing the order of work (sequential or parallel) for each Developer agent moving forward. Therefore, the ablation of the kickoff meeting is not possible in our method. To make the mechanism clearer, one detailed example is provided in Figure 7 (Appendix B) on Page 17 and Figure 14 (Appendix H) on Page 23.

---

Q3: Can you explain/show more about the prompts and exemplars (if exemplars are used) for each role?

Thanks for your advice. We will add the specific prompt content in the revision (one of the prompt templates for the Manager agent as shown below due to the context limit).

# Prompt <em>P</em> (Line 5 in Algorithm 2)
system_prompt = ("You are a software development manager."
                "Your responsibility is to provide clear guidance and instructions to a developer regarding modifications or improvements needed in a specific code file. "
                "This guidance should be based on the details provided in the issue description and the existing content of the code file.")
user_prompt = ("Review the issue description and the content of the code file, then provide specific instructions for the developer on the actions they need to take to address the issue with these files.\n"
              f"# Issue Description:\n{issue_description}\n# Code File:\n{file_content}\n"
               "Respond concisely and clearly, focusing on key actions to resolve the issue. Limit your answer to no more than 100 tokens.") 

---

References
[1*] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint (2024).
[2*] Dubey, Abhimanyu, et al. The Llama 3 Herd of Models. arXiv preprint (2024).
[3*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint (2024).

Thanks a lot for reviewing our manuscript and thanks for your positive comments (i.e., the in-depth empirical analysis, rigorous methodology, and the well-designed visualizations). We are sorry for the confusion and unclear expression in the previous version. We have addressed each of the comments and suggestions below.

Q1: Your study focuses on the SWE-bench dataset, which is limited to 12 Python repositories. Have you considered how MAGIS might perform on a more diverse set of programming languages and project types? What steps could be taken to validate the framework's generalizability beyond Python projects?

Thanks for your comments. We acknowledge that the SWE-bench has limited diversity, which we discuss in Appendix K. While we recognize the need to evaluate MAGIS on a broader range of programming languages (PLs) and project types, we found that there are currently no available datasets in other programming languages. To evaluate our framework's generalizability beyond Python projects, we plan to construct a dataset with various programming languages and projects, then validate MAGIS by collecting pull requests, issues, and test cases from popular repositories, setting up the testing environments, and executing MAGIS to assess issue resolution. This process aims to demonstrate MAGIS's effectiveness across diverse programming languages and project types.

---

Q2: The paper doesn't discuss the computational requirements of MAGIS. Can you provide information on the runtime and resource requirements of MAGIS compared to simpler approaches? How does this impact its practical applicability in real-world software development environments?

Thanks for your comments. We will include the computational requirements of MAGIS in the revision. Specifically, our framework resolves each issue in approximately 3 minutes, with an average processing time of under 5 minutes per instance (as noted in Appendix E). Our method utilizes GPT-4, so any machine capable of accessing the OpenAI API can run the experiments. While LLM-based multi-agent systems [1-3*], including ours, require more computational resources and time compared to simpler approaches, this investment is justified. Our method significantly increases the issue resolution rate from 1.74% to 13.94% (as shown in Table 2), allowing us to tackle problems that were previously unresolvable. In summary, this paper emphasizes how to leverage LLMs effectively to enhance issue resolution and improve overall performance.

---

Q3: How have you ensured that MAGIS is not overfitting to the specific characteristics of the SWE-bench dataset? Have you tested the framework on any GitHub issues outside of this dataset to validate its real-world applicability?

Thanks for your comments. First, our method operates as an LLM-based multi-agent system, which means it does not require training on a specific dataset. This characteristic helps prevent overfitting to the SWE-bench. While we have not yet validated our method on other GitHub issues, we acknowledge the importance of this step and plan to address it in future work. We understand that broader testing will enhance the generalizability of our results. Finally, the SWE-bench dataset comprises various repositories and real-world GitHub issues, making it a reasonable basis for evaluating our method.

---

Q4: While you describe the roles of different agents, there's limited analysis of how their interactions contribute to overall performance. Have you conducted any ablation studies on different interaction patterns between agents? This could provide insights into which aspects of the multi-agent approach are most crucial for success.

Thanks for your comments. We recognize the importance of analyzing the contributions of different agents in our system. In Section 4.2, we conducted an ablation study for the QA agent, and the corresponding results are shown in Table 2. While we have not yet performed ablation studies specifically on different interaction patterns between agents, we analyze the contributions of each phase—planning and coding—in Sections 4.3 and 4.4. These sections provide insights into how these phases impact overall performance. We appreciate your feedback and will explore different interaction patterns in future work.

---

Q5: While you mention using prompts, there's limited discussion on prompt design. Could you provide more details on your prompt engineering strategies and how they impact the framework's performance?

We appreciate your interest in our prompt design. In the revision, we will include a detailed discussion of our prompt engineering strategies and provide specific examples of the prompts used. Our prompt engineering strategies adhere to established best practices [4*]. Specifically, we construct our prompt templates in markdown format, ensuring that each prompt covers all relevant information required for the communication.

---

References
[1*] Wang, Xingyao, et al. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint (2024).
[2*] Yang, John, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint (2024).
[3*] Zhang, Yuntong, et al. Autocoderover: Autonomous program improvement. arXiv preprint (2024).
[4*] Jessica Shieh. Best practices for prompt engineering with openai api. OpenAI, https://help.openai.com/en/articles/6654000 (2023).