R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents

We thank the reviewer for their positive feedback and are pleased that they found our breaking new ground for test-time scaling and with rigorous experimentation, including those with SOTA models.

I am not totally convinced of the claim that the open-source approach is comparable to the best proprietary models. Figure 1c seems to make that case, but Table 3 (which is never referenced in the paper) paints a different picture. Here the best result from the opensource approach is comparable to middling proprietary models, but not the best ones.

Thanks. Figure 1c compares our system, comprising a 32B model, against strong closed-source model offerings from OpenAI (o1 models) and Anthropic (Claude-3.6-Sonnet). Notably, Claude-3.6-Sonnet was SOTA on the SWE-Bench leaderboard for 4 months starting November until February-end (1 month before COLM submission) when Claude-3.7-Sonnet was released.

We only claim comparable performance to the Claude-3.6-based systems and provide a comparison to Claude-3.7-based systems in Table 3 for completeness. We will modify the draft to make this clearer. We would, however, like to point out that Claude-3.6 and OpenAI o1 models are notably strong closed model systems, and our work makes progress towards achieving parity with them.

$\newline$

I'd like to hear an explanation for how to reconcile these two things. I'd also like to see it acknowledged that though the results of this paper seem to indicate value in integrating execution-based and execution-free metrics, that past studies cast doubt of the validity of execution-free metrics. Here's a reference to corroborate that:

Xie et al. CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks (on arxiv)

Thanks for the question.

Indeed, in Sec. 4.2, we also observe limitations of execution-free verifiers. We perform various analyses to understand the reason for their limitations, and one of our key findings is that execution-free verifiers can be unreliable as they may be biased by other heuristics (e.g., agent thoughts) over the final output patch. So, for example, as depicted in Fig. 8, even for an incorrect trajectory, the verifiers can get biased by agent thoughts (showing confidence) and predict SUCCESS for an incorrect trajectory. But even though execution-free verifiers are imperfect, they can still be useful and provide some signal for comparing correct and incorrect programs, as also found in various previous works such as SWE-Gym [5], Neural Code Rerankers [6].

The reason that execution-free verifiers provide value in combination with execution-based verifiers is that, unlike CodeBenchGen [4], our execution-based verifier (or testing agent) uses self-generated tests and does not have access to ground truth information including the sourcecode or tests. Thus our generated tests can differ from oracle tests that are found in the repository and can be imperfect. We again study the limitations of these tests and highlight two key limitations – lack of distinguishability (i.e., lacking coverage to distinguish correct patches from incorrect patches) and generating toxic tests (i.e., tests that pass incorrect patches but fail correct patches). Through this analysis (Sec. 4), we demonstrate that combining the two weak verifiers can result in improved performance of the overall system.

• For example, distinguishability - in practice, the agent-generated tests do not cover all edge or challenging test cases; thereby leading to ties during verification. The use of an execution-free verifier can help provide better distinguishability in such cases; given the agent trajectory and output patch, the execution-free verifier is able to easily verify agent trajectories where the agent tried to handle all edge cases beyond just the problem statement.

$\newline$

Can you shed some light on how it's possible that integrating a low-validity metric (execution-free) improves performance? What kinds of cases does it help with over-and-above just the execution-based metric?

Great question! As noted above, we find that integrating execution-free and execution-based verifiers is useful when execution-free verifier scores provide distinguishability in case of a tie with execution-based test scores.

To elaborate:

• Ideally, given a very good test-generation agent, execution-based verification should alone be sufficient to rerank the generated patches.
• However, in practice, the testing agent-generated tests suffer from low distinguishability and do not cover all edge or challenging test cases; thereby leading to ties during verification.
• The use of execution-free verifiers can help provide better distinguishability in such cases; as given the agent trajectory and output patch, the execution-free verifier is able to easily verify agent trajectories where the agent tried to handle all edge cases beyond just the problem statement.

We thank the reviewer for their positive feedback and are excited that they found our work excellent and much-needed with novel and interesting findings.

This is an excellent and much-needed work! I don't see any strong reason to reject this paper. A couple of minor points:

Given LLMs are used to generate the data. It'd be good to have a sense of the quality and solvability of these tasks. E.g., to what extent can humans solve these auto-generated tasks? I understand, though, this is for training, so the bar is not necessarily as high, and downstream performance improvement already is a good signal of utility.

Please note that while LLMs are used to generate data, the problems are generated directly from commits (instead of human PRs). Thus, each problem should be solvable given two conditions:

1. The ground truth patch (commit patch) is valid, i.e., some tests fail before application of the ground truth patch, and all tests pass after application of the ground truth patch.
2. The generated natural language problem statement is non-ambiguous / solvable: i.e., the repository and the natural language problem statement provide complete information required for fixing the issue.

Regarding (1), we adopt our test validation strategy (discussed in Sec. 2 and Appendix A), ensuring that each ground-truth patch is validated in the training dataset. Regarding (2), to ensure non-ambiguity in the natural language problem statement, we use the backtranslation approach (refer Sec. 2) for generating Github issues grounded in the execution behavior (execution outputs) of the program.

Further evaluation: While exhaustive human annotations were not feasible due to large sample size and resource constraints, we also performed LLM as Judge annotations to evaluate the quality of the curated problems. In particular, we follow the annotation criteria from [1], where for each problem, we collect annotations on the two axes for problem quality. We use a total of 500 problems for both synthetic and real Github tasks. o4-mini was used for LLM as judge experiments (note that o3-mini was used as the back-translation model).

• Are the tasks well specified?: Since SWE-agents are required to generate an output patch given a problem statement and the codebase, we wish to have non-ambiguous problem statements to ensure the solvability of the task. To this end, following [1], we label the problem statement with these 4 possible labels:
  • 0: The issue is well-specified and it is clear what is required for a successful solution.
  • 1: There are some blanks to fill in about the issue, but there is a sensible interpretation of what is required for a successful solution.
  • 2: The issue is vague and there is room for ambiguity. It is unclear what a successful solution would look like.
  • 3: It is almost impossible to understand what you are being asked to do without further information.
• How valid are the evaluation criteria (i.e. tests)? To further evaluate the quality of the generated tests, we label the tests with these 4 possible labels:
  • 0: The tests perfectly cover all possible solutions.
  • 1: The tests cover the majority of correct solutions, however some unusual solutions may be missed.
  • 2: The tests work but some perfectly reasonable solutions may be missed by the tests.
  • 3: The tests are too narrow/broad or they look for something different than what the issue is about.

Results are shown below. As seen, we observe that synthetic problems not only perform similarly w.r.t. real github issues after downstream training (Sec. 3.1) but can also help control / reduce the ambiguity in problem statements when creating new SWE tasks.

We will include this evaluation in the updated version of our paper

$\newline$

It is limited to only Python. But the entire pipeline can, in the future, perhaps be extended to other languages.

Thanks! While the current work mostly focuses on synthetic data curation for Python repositories, the proposed testing and back-translation framework for synthetic SWE-task generation is generalizable to other languages.

References:

[1] Introducing SWE-bench Verified, Chowdhury et al., OpenAI, 2024

As the end of the response period is approaching, we would like to check in with the reviewer to ensure that our clarifications adequately address your questions. Please let us know if you need more information for making your final decisions!

We thank the reviewer for their positive feedback and are excited that they found are work well-written and enjoyable to read, very timely and super useful, including a great choice of experiments and articulation of findings with valuable insights and strong performance.

(Important but not a reason to reject) Please note ``AgentGym'' already exists as a related but apparently different resource for agents. See https://agentgym.github.io/ and https://arxiv.org/abs/2406.04151

Thanks! We will update the paper title to avoid any naming conflict.

As the end of the response period is approaching, we would like to check in with the reviewer to ensure that our clarifications adequately address your questions. Please let us know if you need more information for making your final decisions!

We thank the reviewer for their positive feedback and are pleased that they found our work well-written, a significant contribution, and supported by extensive, compelling empirical results.

I'd be curious to learn how to evaluate the quality of the curated problems--a qualitative evaluation of the curated benchmark would be helpful.

Quantitative Results

• Downstream Evaluation. In Section 3.1 (L143-150) we study the quality of the curated problems by evaluating their downstream effect on training. Specifically, we compare models trained on real GitHub issues with models trained on our synthetic problems, using the same number of trajectories (400) for each. Models trained on synthetic data achieve nearly identical performance (27.8 % vs. 28.0 %), demonstrating that our procedurally generated environments match the training value of real-world examples.

• LLM-as-Judge Annotations. While exhaustive human annotations were not feasible due large sample size and resource constraints, we performed a LLM-as-Judge protocol to asses the problem quality. In particular, we follow the annotation criteria from [1], where for each problem, we collect annotations on the following axis for problem quality. Following [1], we annotate 500 synthetic and 500 real GitHub tasks along two axes:

  • Are the tasks well specified? Since SWE-agents require to generate output patch given problem statement and the codebase, we wish to have non-ambigous problem statements to ensure solvability of the task. To this end, following [1], we label the problem statement with 4 possible labels:

    • 0: The issue is well-specified and it is clear what is required for a successful solution.

    • 1: There are some blanks to fill in about the issue, but there is a sensible interpretation of what is required for a successful solution.

    • 2: The issue is vague, and there is room for ambiguity. It is unclear what a successful solution would look like.

    • 3: It is almost impossible to understand what you are being asked to do without further information.

  • How valid are the evaluation criteria (i.e., tests)? To further evaluate the quality of the generated tests, we label the tests with these 4 possible labels:

    • 0: The tests perfectly cover all possible solutions.
    • 1: The tests cover the majority of correct solutions, however, some unusual solutions may be missed.
    • 2: The tests work but some perfectly reasonable solutions may be missed by the tests.
    • 3: The tests are too narrow/broad or they look for something different than what the issue is about.

Results are shown below. As seen, we observe that synthetic problems not only perform similarly w.r.t. real github issues after downstream training (Sec. 3.1) but can also help control / reduce the ambiguity in problem statements when creating new SWE tasks.

As the end of the response period is approaching, we would like to check in with the reviewer to ensure that our clarifications adequately address your questions. Please let us know if you need more information for making your final decisions!