SWE-bench: Can Language Models Resolve Real-world Github Issues?

Thanks for your thorough review of the paper and suggestions - the feedback has been helpful to clarify several details.

Please refer to the general response for a summary of our updates, which are also reflected in the updated paper. We also answer your specific concerns as follows:

1. Temporal comparisons to determine if knowledge cutoff provides unfair advantage: We agree that comparing different models trained on data from different points in time could lead to unfair advantages, which is a shared concern among all LLM research. To fully investigate this, we added an extended temporal analysis on our results (Appendix C.3). We calculate model performance on task for each calendar year from 2018-2023.

Our original observation holds:

• Earlier years’ problems are not easier
• Models trained on datasets w/ later cutoff dates != Better performance on more recent tasks (i.e. GPT-3.5 vs. GPT-4).

The fact that different models are trained on data from different dates is an important issue for anyone dealing with evaluating LMs, and we believe that our analysis efforts show that our results are not tainted by this issue. SWE-bench’s ease of creating new issues via automatic collection also mitigate this concern for future users of this benchmark.

2. SWE-Llama is not a significant contribution: SWE-Llama is not the main contribution (full list in General Response). SWE-Llama is important because it establishes a lower bound for future attempts at the benchmark. It also shows the promise of fine-tuning on SWE-bench-train, which would likely improve performance for models like GPT-4.

3. No comparison between CodeLlama and SWE-Llama: As discussed in Section 4, we found that open source models like CodeLlama are completely unable to generate well formatted patches. CodeLlama’s performance is 0% in any retrieval setting, and patch generations consistently fail to apply. This prevents any meaningful evaluation of CodeLlama, and is the primary motivation for fine-tuning with SWE-bench-train. We thank you for bringing this up and have clarified this in the new version of our paper.

4. Characterizing patches regardless of whether they applied or not: Thanks for pointing this out. Based on your feedback, we recalculate the metrics (i.e. Total Lines, Added, Removed) for an average patch generation across all patches (not just successfully applied patches). Across all metrics + models, patch generations are much closer in size to gold edits. From this table, it’s clear that models struggle with generating longer patches that are correctly formatted. Further inspection of these flawed generations, as shown in Appendix F, indicate that hallucinations, abiding to existing code style/structure, and referencing long range dependencies correctly are common formatting errors that surface more frequently in longer generations.

Thanks so much for your time and effort towards this review, and for pointing out that SWE-bench is a good, real-world benchmark.

We provide a list of updates in the general response comment above. Regarding your specific comments and suggestions:

1. Lack of novel solutions to the task in the benchmark: Past precedents such as the GLUE benchmark for NLU, WMT benchmarks for machine translation, and WikiText103 for LM perplexity, have shown the importance of simply presenting a tough benchmark for driving progress. We believe that presenting work which focuses primarily on introducing a challenging, useful benchmark can be very useful for the ML and NLP communities. Through experiments and analysis we also present actionable insights for future improvement (See Section 5, Appendix C/D/F).

2. Reorganization of the tables/figures to be closer with text: In the latest upload of our paper, we have incorporated this suggestion and put figures closer to text and made sure tables are mentioned in order. This should improve the readability. Thanks for this suggestion.

3. Unclear sentence discussing retrieval: Yes, your understanding is correct. To make the paper clearer we’ll be replacing the mentioned sentence with the following: "We compare the BM25 retrieval results with those of the 'oracle' retrieval setting, as shown in Table 3. We observe that in approximately 40% of instances, BM25 retrieves a superset of the oracle files for the 27000-token context limit. However, in almost half of the instances with the 27000-token limit, it retrieves none of the files from the 'oracle' context."

Thank you again for your feedback and comments. If you have any remaining questions or concerns, we would be happy to address them.

Thank you for your encouraging comments and suggestions.

Please refer to the general response for a summary of our updates. Regarding your specific concerns and questions:

1. Patch generation is out of domain: We agree that LMs trained on code may seem out-of-domain for generating patch files. We ultimately think this is why open source models, including CodeLlama instruct and Llama 2 chat, are currently incapable of solving any issue in SWE-bench; they’re simply not on-par with proprietary models at instruction following yet.

   We considered this problem as well in an ablation in Section 5, where we find for instance, when evaluated on the shortest half of inputs in the Oracle retrieval setting, Claude achieves 3.9% resolution rate when generating whole files compared to 7.8% when generating a patch.

   We therefore find that patch generation is both a practical and efficient modeling decision for our baselines, and that is why we decided to use it across the board.

2. Pretraining code doesn’t indicate the file names or paths: You’re correct that pre-training data typically omits things like filenames, directory structures, etc. In all of our experiments, when representing code, we always wrap files’ contents with a start tag like [start of src/utils/readers.py] and an end tag [end of src/utils/readers.py], as well as prepend each line with its line number to help models create well formatted patch files. We’ve made this clearer in the description of our representations in the paper.

We greatly appreciate your comments and suggestions. If you have any additional questions or concerns, please let us know. Thank you for your time and consideration.

Thank you so much for your review.

Please refer to the general response for a summary of our updates. To address your specific concerns and questions:

1. More in-depth and insightful analysis: Given the number and diversity of problems in SWE-bench, insightful analysis of generated solutions is certainly a challenge. To provide better insight, we’ve expanded our qualitative analysis in the paper to analyze 11 full generations from both Claude and SWE-Llama, showing that: Models tend to propose shortcut solutions which do not consider all possible inputs to the code. Models have a simplistic coding style that ignores the style and utilities of the overall codebase, leading to less accurate and portable solutions. We provide more insights and possible solutions, e.g., prompting with documentation, learning from interaction, and test generation, for solving these problems in the paper.

2. Our exact contributions — We restate our exact contributions in the general response comment and clarify them in the paper. Please refer to the general response for more information on this concern.

3. Results are hard to interpret/distinguish between models: We find that many of the challenges to solving long-context code problems are common to all models evaluated. While our results show that some models have better average success, overall performance on SWE-bench is so low across the board that it can be difficult to differentiate between models’ behavior. We think that SWE-bench can provide an important benchmark in guiding LM progress on the continual refinement of long context understanding, high and low-level reasoning, and grounded generation.

4. Is there a need for a benchmark even simpler than SWE-bench?: LM progress moves fast and evaluation benchmarks are quickly outdated. We believe that setting a bar as high as SWE-bench is of great interest to the community since it could serve as a more persistent aspirational achievement for LMs. Achieving high accuracy on SWE-bench would be both incredibly useful to actual software engineers and also indicate substantial technical progress in the development of LMs. Additionally, provided the structure of SWE-bench, it is possible for users to evaluate their models in slightly simpler settings (e.g. with the Oracle and Oracle-collapsed baseline), which can provide researchers some improved insights in the development of their own models.

Thank you again for your feedback; we’ve uploaded a new version of the paper which addresses your concerns. If you have any remaining concerns or questions, please let us know.