SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

We sincerely thank the reviewer for their feedback on our work.

W1: Criteria for project and issue selections are unclear Q1: This work extended SWE-Bench, while major challenges encountered during data collection may be similar to those in SWE-Bench. Could authors clarify what unique challenges were faced in data collection for this work compared to SWE-Bench?

Please refer to Section 2.2, where we address exactly this question and thoroughly review the project and issue selection criteria step by step. The primary challenges were namely adjusting task collection to work for JavaScript repositories (e.g. more heuristics that target image assets), designing entirely new execution environments from scratch, and performing extensive human validation (absent from SWE-bench).

W2: generalizability of the JS-based multimodal issues is limited.

Our interpretation of this weakness is that JavaScript-based multimodal issues do not reflect all of Software Engineering. If this is correct, we agree, but the discussion around generalizability is less in reference to the coverage of this benchmark, and more in reference to how existing agent systems are designed. We make no mention of generalizability in Section 2, and only bring up generalizability in the context of AI system design in Section 3. In short, we are not claiming SWE-bench Multimodal represents all of software engineering, but the fact that there is such a significant performance gap between SWE-bench M and SWE-bench systems is very interesting because it calls into question generalizability.

W3: there already exists a rich research body on multimodal bugs/issues---mobile bugs/issues Q2: There is an established research body on reproducing/fixing multimodal bugs in mobile apps, while this work focuses on JavaScript bugs. What is the motivation behind focusing specifically on JavaScript-based multimodal bugs, given there already exist mobile app bug datasets?

We review prior works in the Related Works section. SWE-bench Multimodal is filling a clear research gap - the evaluation of AI systems on software engineering issues with visual components. There are several existing works on GUI navigation and multimodal HumanEval-style function completion, but to the best of our knowledge, there is no work that addresses editing a repository given visual specifications.

If the reviewer can refer us to the mobile app bug datasets like SWE-bench Multimodal, we’d be happy to incorporate it into our Related Work section and Table 1. We are aware of works such as [1] [2] and [3], but these works are significantly different from SWE-bench Multimodal or repository-level code generation/editing, as they focus on automating user testing for mobile apps. These works evaluate how well a system can recognize or reproduce bugs in a mobile app. Code completion is a part of such works. It does not evaluate how an app’s codebase can be updated to resolve an issue.

[1] Wendland, Tyler, et al. "Andror2: A dataset of manually-reproduced bug reports for android apps." 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021.

[2] Xiong, Yiheng, et al. "An empirical study of functional bugs in android apps." Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2023.

[3] Fazzini, Mattia, et al. "Enhancing mobile app bug reporting via real-time understanding of reproduction steps." IEEE Transactions on Software Engineering 49.3 (2022): 1246-1272.

W4: the adaptions for existing models are mainly on the language level, it's unclear how to handle images/videos

We address these points directly in the main paper, although if the reviewer has more specific qualms, that would be helpful. Please refer to Section 3.1, where we discuss specifically how (1) existing systems were adjusted such that images + videos can be fed to the underlying LM, and (2) when possible, new tools were created to help with viewing, creating, and manipulating visual assets in a codebase or via a browser.

Q6: The impact of visual data (images/videos) on the bug-fixing process is unclear; the performance decline without images/videos is around 3%. Does this imply that images/videos are unimportant for these tasks?

No, this does not imply images/videos are unimportant. Our analysis attributes the 3% improvement to images/videos. But the fact that the gain is only 3% suggests, rather, that existing systems do not utilize visual components effectively. Sections 2.2 and 2.3 emphasize, with human effort, that images/videos/visuals in an issue are necessary to solving the problem. Section 4.1 is our analysis point dedicated to showing that improvements on SWE-bench Multimodal performance correspond to improved abilities in image understanding.

W5: the comparison between these approaches on multimodal issues and text issues is missing

This is discussed in our report (Sections 2.2, 2.3, 3.1 + Table 5), although we would appreciate more clarification regarding what additional analyses the reviewer might be looking for.

Sections 2.2 and 2.3 discuss the different challenges in collection strategy (simply, that SWE-bench Multimodal’s collection strategy is much more extensive than SWE-bench’s). Section 3.1 details how AI systems were adapted to incorporate visual feedback, and Table 5 + results illustrate how the inclusion of visual components in SWE-bench Multimodal makes for a more challenging benchmark. Our human annotation procedure (Appendix D) shows that SWE-bench M is generally more difficult based on estimated time to complete a task. Appendix A includes multiple breakdowns illustrating quantitative differences between SWE-bench Multimodal issues compared to SWE-bench issues (Figure 5).

W6: how important the visual data (images/videos) in the process of bug fixing is unknown

We address this extensively. In Section 2.3, Appendix D.(1,2,3), we discuss how our manual validation process with human annotators strongly suggests that visual components provided in a naturally occurring GitHub issue are indispensable to solving the problem (83.5% of task instances were marked as unsolvable without its associated image(s)). These challenges are also echoed empirically in Section 4.1, where evaluation metrics and our analysis of trajectories continue to reflect these points.

Q3: The data collection process inlcudes five steps, but the criteria for each step are unclear. For example, in step 1, the authors selected 17 repositories. What was the project sample size after applying the filters (e.g., 5,000 or more stars and 500 or more pull requests)? Step 5 includes human validation, how many authors involved? How to solve the conflict during the process?

• What was the project sample size after applying filters? - For specific metrics on how many candidate task instances were filtered out at each collection stage, please refer to Table 13. As you write, 135k PRs is filtered down to 1478 candidate task instances.
• How many authors involved - 10 authors. We will update the paper to say this explicitly.
• How to solve conflicts during the process? - In Appendix D, the explicit annotation + label selection processes are provided per question. In general, for easy questions (e.g. label image category), just one annotator was used per question. For harder questions (e.g. estimating task difficulty), four annotators provided labels for each question + task instance, and the final answer was selected via majority vote.

Q4: "From 135k PRs, this new filtering criteria yields 1,478 candidate task instances." Does this indicate that the identified bugs are relatively uncommon or corner cases? How important is the importance of these collected multimodal bugs compared to text-only bugs?

The candidate task instances meet a number of criteria, including (1) the PR has 1+ issues (2) the PR has 1+ tests (3) either the test(s) or issue(s) have images in them. In SWE-bench, which put forth the first two criteria, usually just 10% of PRs passed this filter. In the case of SWE-bench Multimodal, criterion (3) filters out an additional 90% of SWE-bench style candidates, which is high yield (compared to nearly 99% being filtered out for Python, a number we determined in a minor experiment early on in the project).

The implication that these filters may lead to a distribution shift that yields uncommon, non-representative task instances is a non-issue. Our extensive manual human validation of task instances, which involved the authors reading every single issue description, suggest that task instances are meaningful and visuals are necessary. From Figure 5, it’s also apparent that quantitatively, SWE-bench M instances have [problem statement length, files/functions/lines edited, # of tests distributions] that are similar to SWE-bench task instances.

Q5: Do the baseline models perform significantly differently on multimodal issues than text-only issues within these JavaScript projects?

We agree this is an interesting question. We believe that we address this with our analysis that indicates how understanding visual components is one of two main reasons for why SWE-bench M is harder than SWE-bench (the other being JavaScript). To answer the reviewer’s version of this question, we would have to perform additional effort of curating a SWE-bench style dataset for JavaScript task instances, which is non-trivial. Curating SWE-bench “JS” is also tangential to the focus of our paper.

Q1: Have the authors tried applying SWE-agent JS or SWE-agent M on the original SWE-bench tasks?

We have not tried this. SWE-agent JS/M’s Agent Computer Interface (ACI) is generally a superset of the original capabilities (e.g. tools, prompts, history processing) of SWE-agent. We could run this analysis, but given that all of SWE-agent’s original capabilities are maintained in SWE-agent JS/M, we’d expect performance on the original SWE-bench task to be very similar.

Q2: How are human annotators chosen?

Thanks for pointing this out, we will provide more details in the updated version. The human annotators were 10 of the authors. For easier annotation tasks (e.g. D.1 image classification), the per-image labeling was divided evenly among all annotators. For harder annotation tasks (e.g. D.4), 3/10 annotators were randomly assigned to each task instance, and the final label was assigned as majority vote.

We sincerely thank the reviewer for their response (and for the friendly categorization of feedback).

W1: More analysis on the difficulty of the dataset

The goal with Section 2.3 (Features) and Section 4.1 (Analyses) is to communicate (1) that SWE-bench Multimodal is more difficult for AI systems, and (2) this difficulty can be directly attributed to the necessity of visual understanding that existing VLMs struggle with. Our human annotation efforts are also meant to suggest how SWE-bench Multimodal problems are tractable for humans, but difficult for models because of the introduction of visual components.

With respect to the reviewer’s feedback, we can understand how it might seem like we are suggesting that for humans, SWE-bench Multimodal problems are inherently harder than SWE-bench problems. This is not our goal, and we will update the paper such that it is clear we are not claiming this. In fact, we aim to show that SWE-bench Multimodal problems are very tractable for human developers. It would be prohibitively expensive for us to have human task workers attempt every instance, so following the work of SWE-bench Verified, we use human annotations (mainly Q.2 - Q.4) to demonstrate that (1) the tasks are do-able (anything considered impossible was removed), and (2) the difficulties and key to solving SWE-bench Multimodal lie in visual understanding.

W2: Improvements to Testing

We agree with the reviewer’s point entirely - Test augmentation for SWE-bench Multimodal / SWE-bench would (1) greatly improve the quality of the benchmark, but (2) it is also difficult enough that it can constitute an entirely separate research contribution. (See SWT-Bench [1], which is similarly a derivative of SWE-bench that focuses exclusively on test generation).

The main challenge of such an effort is that codebase unit tests are nowhere near as self-contained as traditional code generation (e.g. HumanEval) unit tests. In codebases, tests are often supported by an entire setup procedure of mocks, stubs, and testing entities that require understanding in order to write meaningful unit tests. These challenges are only complicated by the variance in testing infrastructures across repositories and time.

As with the original SWE-bench work, we maintain faith in the scraped tests because they are written by a human maintainer, suggesting that the test is extremely faithful to the intended behavior. In its current form, SWE-bench Multimodal is already an effective evaluation for the same reasons as SWE-bench when it comes to testing (if not better due to the more extensive testing facilities that come with JavaScript + visual components as discussed in Section 2.3). Improving coverage like EvalPlus [2] did for HumanEval is great, but non trivial.

[1] Mündler, Niels, et al. "Code Agents are State of the Art Software Testers." arXiv preprint arXiv:2406.12952 (2024).

[2] Liu, Jiawei, et al. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in Neural Information Processing Systems 36 (2024).

W3: More metrics than just pass or fail

We generally agree that introducing more metrics to repository-level coding tasks would be interesting. In our analyses (Section 4, Appendix C.3), we provide several further metrics (% resolved by repo/year/annotation label, the rates of different categories of actions issued by SWE-agent JS/M) that present resolve rates more granularly. As an immediate response, we can also include file localization (F1) scores for SWE-agent JS/M in the appendix as well.

With regards to establishing new intermediate metrics, we think that such contributions might be more appropriate for methods that build upon this benchmark. For instance, with SWE-bench, different approaches (e.g. procedural vs. agentic) tend to use different signals as a way of measuring progress, and these signals are not necessarily transferable across methods. While one method might rely on test-reproduction frequently (e.g. SWE-agent), another does not use it at all (e.g. Agentless), so the utility of measuring issue reproduction success rates is dependent on the method. Also, as SWE-bench Multimodal intentionally inherits the strong characteristics of SWE-bench, we do not invest as much in new metrics in order to better illustrate that the benchmarks have the same interface for evaluation, making it easy for practitioners to develop on both simultaneously.

We thank the reviewer for the detailed and supportive comments. We restate your main points and address each of your concerns below.

W1: SWE-bench M should emphasize the visual components of JS to front-end development rather than the limitations of Python.

We appreciate your insight here and on realizing the value of SWE-bench M in evaluating multimodal systems. We think that our core contributions are not necessarily limited to expanding SWE-bench to visual programming problems, but the introduction of a second major programming language family is a major contribution in itself. Especially since we believe that a critical failure point of the evaluated baselines is not directly related to their image processing abilities, but their dependence of Python-specific attributes. We believe our results, analysis and discussion provide substantial insight into the visual components of our evaluation, but we’ve additionally updated our introduction and some language to further highlight the visual testing components of SWE-bench M.

W2: Insufficient effort made to adapt baselines - cannot make claims of generalization.

Your response highlights an important aspect of our contribution that we would like to reiterate. In particular, our strongest claims about generalizability refer to particular systems, not necessarily to methods. However, based on our results, we also suggest to readers an important finding, which is that highly structured systems, like Agentless, may be prone to language-specific design choices that make them inherently less generalizable. Our approach deliberately preserved the core architectural assumptions of baseline systems while making only necessary adjustments, allowing us to examine their inherent capacity for generalization. While you’re correct that, perhaps, an extensively re-engineered adaptation of systems like Agentless and Moatless could provide stronger performance on SWE-bench M, the very necessity of very major adaptations demonstrates the lack of flexibility of these systems as proposed. Our experiments show that lighter weight, agent-centered systems, such as SWE-agent, simply work out of the box with essentially no language specific adaptation; where even language-adapted variants of Agentless and Moatless performed poorly.

Q1: Can we provide more details on adaptations made for Agentless / moatless?

Our approach to system adaptation was guided by a principle of minimal intervention - making only those changes necessary to enable the intended functionality while preserving core system design. For Agentless, this meant implementing JavaScript-specific equivalents to their Python-specific parsers and tools, while maintaining the system's built-in assumptions about code structure and modification.

Q2: Have we considered other visually-rich domains, like game dev and mobile apps?

Because the construction of SWE-bench style benchmarks is extremely expertise and labor intensive, we take a similar approach to SWE-bench when constructing SWE-bench M. This means focusing on repositories that can provide the greatest number of problems with the natural SWE-bench Issue-PR-Test structure. The resulting distribution of tasks and problems are the natural result of this task acquisition process. While other visually-rich domains such as game dev and mobile apps could further increase the diversity of problems to evaluate, we’ll leave the acquisition of new datasets to future work.

Q3: How do we envision SWE-bench M evolving as multimodal models improve?

We see the contribution of SWE-bench M as helping guide the current and next generations of software engineering agents and LMs to focus on a greater diversity of problems, including multiple languages, front-end, and visual programming. We think that the future of the SWE and multimodal evaluations will require expanding to more fundamentally difficult problems and domains, such as long-playing game development, which could require very different solutions and models. We see SWE-bench M as being a major contribution to provide meaningful evaluation and comparison of future systems. The current performance of systems of SWE-bench M suggests that there is still a lot of improvement for these systems to make before tackling these harder domains.

---

Thank you again for the thoughtful review and suggestions. We hope we’ve thoroughly addressed your concerns.

Thank you so much for your insightful comments and feedback. Below we summarize your main points and try to address each of them in turn.

W1: Does measuring generalization require extending to more languages than just javascript?

This is a great insight, and we certainly share your sentiment on the benefit of increasing the diversity and number of languages LMs / agents are evaluated on. However, we believe that a greater number of languages is not currently necessary for evaluating and encouraging more general AI programming systems. For instance, even though our benchmark emphasizes Javascript, Typescript, and related languages, we show that existing systems already fail to generalize to this domain. As such, this shows that evaluating systems on SWE-bench and SWE-bench M already will put significant pressure on researchers to develop systems that are more language agnostic and thus more likely to generalize to different domains, languages, and paradigms. Ultimately, while we agree that future evaluations can benefit from more languages diversity, we think that SWE-bench M, as is, makes the first, and most significant, step in that direction, and can have great evaluation power; especially when combined with the original SWE-bench.

W2: Is SWE-bench M is a meaningful innovation?

While SWE-Bench provides a strong foundation, SWE-Bench M addresses a critical gap between SWE-Bench and real-world software engineering challenges by introducing new language and multimodal settings. With SWE-bench M we show that systems can easily overfit to SWE-bench, and Python in particular, which makes evaluation of these systems for real world scenarios harder. We demonstrate that evaluating on SWE-bench and SWE-bench M together is a more effective evaluation, that tests approaches that are more transferable to the real-world than Python-tailored systems.

W3: Is multimodal too hard? Is increasing difficulty an effective evaluation strategy?

While we agree that existing systems may not be immediately effective on SWE-bench M, we believe that this very challenge makes SWE-bench M essential for several reasons: First, the integration of visual and textual understanding reflects the inherently multimodal nature of software engineering in practice. Our analysis reveals that 83.5% of image-containing issues require visual comprehension for resolution - these aren't artificial difficulties we've introduced, but rather authentic challenges that real developers navigate daily. Besides filtering for issues with images, we perform no adversarial or other filtering to increase the difficulty of these problems. To shy away from such tasks would be to mischaracterize the nature of software engineering and provide a less than realistic evaluation of evaluated AI systems. Second, while current performance is indeed modest (12.2% success rate), this should not be interpreted as a failure of SWE-bench M, but rather as a more accurate reflection of the current capabilities and limitations of our systems. Just as early NLP benchmarks helped drive progress despite initially low performance, we believe SWE-bench M serves a crucial role in highlighting the gap between current AI capabilities and the full spectrum of real-world software engineering challenges.

Q1: Why are many patches and test patches empty in the authors' benchmark, and does this affect benchmark quality?

At this time, we’ve intentionally avoided releasing full solutions and tests for the test split of SWE-bench M in order to more strongly discourage overfitting to the test split. Of course, we’ve released these details for the development split of SWE-bench M for active system development. In collaboration with our sponsors, SWE-bench M will provide an evaluation API for submitting and reporting performance on the test split.

Q2: Is pixel-level visual testing a valid way to evaluate front-end patches' effectiveness?

Our pixel level testing primarily applies to instances from Chart.js, a library for chart generation that typically involves generating data visualization charts such as bar plots, scatter plots, etc., and openlayers, a library for geographic visualization and embedding maps in webpages. Visual testing component then apply to pixel-level comparisons for these types of charts and maps, which have low variation and ambiguity.

Q3: Can you demonstrate successful and failed repair cases to prove system can handle complex fixes and explain failure reasons?

Based on your suggestion, we’re adding 2 failed and successful dev instance trajectories from SWE-agent to better demonstrate the nature of these problems and agent solutions. These are shown in the updated appendix.

Q4: Are we missing DevEval, CrossCodeEval, RepoExec?

Based on your suggestion, we’re also updating Table 1 to include the additional benchmarks mentioned.