Breakpoint: Stress-testing systems-level reasoning in LLM agents

Evaluation uses only one agent

Our experiments evaluate five different state-of-the-art models: GPT-4o, o4-mini, Claude-3.7-Sonnet, o3-mini, and Deepseek V3. These represent diverse model families and training approaches (reasoning vs non-reasoning models).

Beyond evaluating different models, we conduct extensive ablations on agent configurations:

1. Tool budget scaling: We vary tool-use iterations (4, 8, 16, 32) and submission attempts (1, 2, 4, 8) to understand how increased computational resources affect performance (see linked figure).
2. Tool preference analysis: We analyze how different models use available tools, finding that successful models prefer high-level tools like read_file over more granular read_function, and gather significantly more information before attempting repairs (see linked figure).

Missed Commit0 related work

We thank the reviewer for bringing COMMIT0 to our attention. We note that this paper is not in the citation period, but we are happy to add this related work to our paper.

While also focused on the same domain of LLM agents solving inverse problems, the cited paper is different in focus than ours in two main ways.

1. System comprehension vs. greenfield development: COMMIT0 evaluates agents' ability to build libraries from scratch given specifications. Breakpoint evaluates agents' ability to understand and repair existing complex systems they didn't create. This distinction is crucial—real-world software engineering often involves debugging and modifying unfamiliar codebases with intricate dependencies.
2. Diagnostic vs. generative challenges: Breakpoint's discovery mode specifically tests fault localization—a critical skill where agents must navigate large codebases to identify problems before fixing them. Our finding that diagnosis is often the primary bottleneck (Section 4.1.3) provides insights not capturable in greenfield development tasks.

In light of these clarifications and improvements, would you consider increasing your score for our paper? If not, could you let us know any additional changes you would like to see in order for this work to be accepted?

Thank you for your feedback. Please see our responses to the helpful points raised in your review below:

“unrealistic” — synthetic vs real bugs

We believe our tasks are representative of important skills in software engineering.

In remove mode, the model is given a function and needs to implement it. This is very similar to classic SWE workflows where given some large codebase a developer needs to implement a well scoped out functionality.

For our discovery mode, this is similar to a bug or backwards compatibility being introduced in a codebase, and then a developer having to diagnose where it lies and fix it. In fact, our evaluation shows much of the difficulty in remove mode lies in finding where the bug is.

Our results remain interesting beyond the scope of evaluating SWE ability. We want to understand the ability of AI agents to dynamically explore and control a complex system, of which codebases are a first class example. This general ability is very broad and important for real world integration, in and outside of software engineering.

Our methodology also allows us to select and create difficult tasks that can scale in difficulty beyond human level. In our qualitative human testing we found that our hardest sets were very challenging for experienced programmers, suggesting our evaluation is able to sample and select for very hard problems.

We link a figure reflecting how having synthetic bugs allows us to scale difficulty in this way. In our n-corruption mode where we corrupt related function components, we are able to create a hard set where the model can solve 0% of the problems. We also select on centrality/complexity to filter tasks in remove mode, creating a much harder set for current models (right). This task creation and selection on difficulty is possible because our problems are synthetic.

"Limited novelty — akin to mutation testing"

We appreciate the reviewer's attention to this important distinction. While Breakpoint shares surface similarities with mutation testing, it fundamentally differs in purpose, methodology, and insights gained. We respectfully disagree about the claim on the limited novelty of the paper.

Our goal in this paper is not to develop better mutation testing tools for evaluating software artifacts; instead, it's to apply automated code transformation methods (including, but not limited to, mutation) to understand if and how an LLM‑powered agent can reason about and repair a fault whose effects cascade through a real, multi‑module codebase.

Our benchmarking methodology is innovative for the following reasons:

• Graph‑aware corruption. We select targets by the joint distribution of complexity and centrality, then introduce semantic edits (multi‑line changes or deletions) that propagate across module boundaries—faults that ordinary mutation pipelines rarely create. As far as we know, this is a novel way of sourcing hard problems, that allows us to explicitly target systems level reasoning.
• Creation of discovery and targeted modes. By separating localisation (discovery) from patch synthesis (targeted), Breakpoint tests for a variety of the subskills involved in SWE tasks.
• Predictive metrics for scalable generation of difficult tasks. We show that two simple graph‑derived scores—complexity and centrality—jointly explain success probability with Mcfadden's $R^2 = 22.1\%$. This confirms that our synthetic tasks capture an interpretable difficulty gradient, which we can then use to scale difficulty smoothly. This novelty is unavailable without combining synthetic mutation generation with LLM evaluation.

Our analysis on our generated SWE data gives us the following insights:

• Resource‑scaling diagnostics. Using the above difficulty axes, we can understand the effect of different resource settings as we scale interaction budget, reasoning gains, etc.. Our statistical tests find that reasoning models are differentially improving at complex functions rather than central ones, and that an increased interaction budget helps with both. The novelty of our problem framing and synthetic testing here allows us to precisely quantify and describe the effect of different resources for LLMs.
• Agent behavior analysis: We add tool usage pattern analysis, revealing that successful models gather significantly more information before attempting repairs and use targeted search effectively. This behavioral analysis provides insights into what makes agents effective at system-level debugging (see linked figure).

In short, Breakpoint is reasoning‑diagnostic rather than coverage‑diagnostic: it creates controllable, labelled challenges that let us ask “which resources help where, and why?” This question lies outside the scope of traditional mutation testing and therefore complements rather than competes with it.

Continuing response in next comment.

Human calibration

Response.

We thank the reviewer for this suggestion. We added the following preliminary evaluation appendix section to our paper:

To better contextualize the difficulty of Breakpoint tasks, we conducted a preliminary human evaluation with the authors solving a small set of benchmark tasks. Specifically, four problems from each of the \textsc{remove} and discovery modes were manually attempted. All eight tasks were successfully solved, with completion times ranging from 4 to 30 minutes per task. These results suggest that Breakpoint tasks—at least at lower corruption complexities—are within reasonable human solvability.

We also qualitatively assessed the difficulty of the hardest Breakpoint problems by sampling one hard-set problem for both remove and discovery. Based on our human evaluation, we found that the tasks are quite challenging and would require several hours of human time to understand and fix the codebase, although they are solvable. This supports the human calibration of our claims on difficulty, as our designated hard sets do seem indeed much harder.

To facilitate future systematic comparisons, our benchmark release includes explicit support for human evaluations, providing users with interfaces to efficiently explore, diagnose, and repair tasks directly within the codebase environment.

In light of these clarifications and improvements, would you consider increasing your score for our paper?

Thank you for your overall supportive review and detailed feedback, which we believe will improve our final manuscript. Please see our responses to the helpful points raised in your review below:

“Only Python limits immediate application to other domains"

We believe the claims we make are fairly general and should not be substantially different for repositories in other languages.

Our toolkit is easily adaptable to other languages and coding environments, as the only Python specific functionality is in searching for candidate functions by parsing the files. Our code takes in a test command to run tests before/after corruptions and this command could just be changed for different language/testing environments.

"In real-world engineering, developers rely on richer signals (e.g., commit messages, stack traces, historical fixes) which are not modeled here."

We do give the model stack traces as test feedback on the state of the codebase, and the model sees the results of the stack trace when it submits modifications to the code.

We think that giving the models richer commit and history signals would indeed be an interesting extension, and we could give our SWE agent access to git to enable that functionality. Given the nature of breakpoint corruptions, giving the model access to a previous "clean" state of the code would trivialize the task. One way to get around this is to introduce multiple edits at once, some of which introduce bugs and some of which don't. We agree that these kinds of extensions would be valuable to explore, but we wanted to keep the scope of the paper focused on ways to generate tasks by modifying a codebase at one point in time.

However our results remain interesting beyond the scope of trying to mirror SWE workflows. We wanted to develop a method that can straightforwardly generate tasks of nearly arbitrary difficulty. In this case, we think that depriving models of signals that human SWEs have is a good way to make harder problems than mirroring alone will allow.

Little qualitative failure analysis

We thank the reviewer for this feedback. We've added additional analysis on failure modes. For instance, we examine the proportion of the time the model finds the right function in discovery mode (see figure). In doing so, we find that Claude-3.7-sonnet and o4-mini, which have very similar performance in discovery mode, actually have very different solve profiles -- Claude finds more correct functions, and o4-mini has a much higher solve rate once it finds the right function. In general, the majority of the reason at least for failure of reasoning models in discovery is in the task of finding what function was corrupted.

We also look at how much better models get on their additional submission attempts. Here, we find that models rarely improve past their second submission attempt (see linked figures for discovery and remove).

Analysis of generalizability across diverse repos / test coverage

Breakpoint's methodology is inherently adaptable to diverse repositories, regardless of size, coding style, or domain. To ensure our findings generalize, we selected repositories representing a wide variety of software domains, including large parsing libraries, machine learning tooling, database systems, and others.

Moreover, in the revised manuscript, we analyzed whether repository-specific factors influenced task difficulty, including: repository size, test coverage, domain (ML/parsing/databases), and coding style conventions. We found that, while average performance varied substantially between repositories, we found no statistically significant correlations between these repository characteristics and agent performance.

We sincerely thank the reviewer for their comments which have been very helpful in improving the paper. We are also grateful for your recognition that our work "tackles a critical gap in evaluation" and provides a benchmark that "can be re-created or targeted to specific software suites, without being restricted to a particular domain or repository." We are also encouraged by your acknowledgment that we "propose a general framework for evaluation in the code repair task".

"The task difficulties are not entirely convincing. Could you elaborate on why the proposed heuristics sufficiently capture and explain task difficulty?"

We thank the reviewer for this question. We answer the following subquestions.

1. Why did we expect that these axis capture task difficulty a priori?

Our goal was to differentiate two capabilities:

• Localized reasoning: fixing a self-contained bug or implementing a single, clearly-defined feature.
• System-level reasoning: diagnosing and coordinating changes across complex, interconnected codebases.

As a proxy for the first capability, we measure complexity of the corrupted function itself, to quantify the difficulty inherent to the code the model will have to generate.

As a proxy for the second capability, we measure call graph centrality, to quantify the difficulty that is inherent to properties of the codebase outside the function, through the dependencies that have to be understood to fix the original function.

2. To what degree were we able to explain task difficulty with our axes?

We find that these two axes explain difficulty fairly well. In remove mode, for instance, the two static measurements of complexity are both correlated with more difficult problems (18.3% and 21.6% McFadden's $R^2$), and jointly they explain problem difficulty better than they do individually by AIC. We find that, by Mcfadden's $R^2$, the joint regression explains $22.1\%$ of the variance, which for McFadden's indicates excellent fit (McFadden 1974).

In our additional figures, we show the results of regressing difficulty onto function complexity and code-line count, and both together. Follow the links for the relevant figures.

We also find that problems with high function complexity and centrality are particularly difficult for the model: for problems in the 90th percentile of both these axes, we find that model performances are between 46% and 31% worse percent on remove questions (see attached figure for detailed result).

We have reworked the results section substantially to highlight these lines of analysis. We agree that these proxies cannot not fully explain the difficulties via this static decomposition, and that is very unlikely that any particular static decomposition can fully explain task difficult. We mainly claim that these axis have some explanatory power, and that they are sufficient for scaling problem difficulty.

“The claim regarding data contamination is unconvincing. If a repository has already been seen during pretraining, generating repair tasks from it does not inherently make the evaluation free from contamination.”

We have revised our paper's claims on data contamination: we meant to claim that the specific problem instances contained in Breakpoint are novel, but we agree that the base repositories themselves may have been seen before. In our revision, we highlight the following additional points:

1. Our methodology can be directly applied to private code repositories, which means that it can be updated to minimize contamination as time goes on. We will more clearly differentiate this "future-proofing" from claims about contamination.
2. Even though the questions are plausibly trained on, frontier models like o4-mini still struggle to solve them.
3. This issue is also less significant for the discovery part of our benchmark because we create fully novel corruptions using an LLM. This means the model has to find the source of a previously unseen bug in the code.

"Test-suite dependence is underexplored"

We agree test suites alone may not cover all failure modes. We do make an effort to filter for high quality corruptions and questions:

1. we only evaluate on problems where the corruption causes at least 5 tests to fail, which makes it harder for the model to get the tests to pass without addressing the core functionality
2. The repositories we select are popular, with at least 1000 github stars. This means that it's quite likely that the testing is correct, if not complete.

Even if the test suite for a given corruption might not fully capture the function behavior, we believe that this repair task is still a valuable signal of model capability, signaling its ability to diagnose and fix issues specified only by stack traces.

"Systems-level reasoning" term is ambiguous

We thank the reviewer for highlighting this ambiguity. By 'system-level reasoning,' we refer specifically to the ability to understand and manipulate interconnected components within complex systems, where modifications to one component can cascade through dependencies to affect distant parts of the codebase. This concept draws from systems thinking (Bertalanffy, 1968; Meadows, 2008), which emphasizes understanding wholes through their interconnections rather than isolated parts. We distinguish this from 'System 2 reasoning' (Kahneman, 2011), which refers to deliberate versus intuitive thinking modes—an orthogonal concept, as one could apply System 2 thinking to either local or system-level problems. In our work, we operationalize system-level reasoning through call-graph centrality metrics and multi-function corruption tasks that require understanding how components interact, rather than analyzing them in isolation. We will clarify this distinction by making this definition explicit in the revised manuscript to avoid confusion.

Toolkit release

We are currently open sourcing our toolkit, and will add the link to the paper in the camera-ready. For anonymization purposes we won't share the link, but could share a link to an anonymous zipfile if it would be useful.

In light of these clarifications and improvements, would you consider increasing your score for our paper? If not, could you let us know any additional changes you would like to see in order to improve this paper further?