Training Software Engineering Agents and Verifiers with SWE-Gym

Could you clarify why a larger dataset (SWE-Gym) with executable environments is necessary, given that SWE-Bench already offers multiple, human-verified versions?

We’d like to clarify that SWE-Bench doesn’t include executable environments or unit tests for its training split. This makes it impossible to use for learning algorithms that train models through real-world action execution and observation. And training on the test split will introduce data contamination and invalidate our results.

SWE-Gym addresses this fundamental gap by providing a separate, complementary dataset with fully executable environments that allows us to train agents on real-world software engineering tasks while maintaining the integrity of SWE-Bench as a clean evaluation benchmark. This separation is crucial for accurately measuring progress of software engineering agents.

Can you provide comparisons against existing methods evaluated on SWE-Bench to isolate the unique contributions of your training approach?

Because our training approaches are focused on training models in real-world software engineering tasks, this requires execution environments and test cases for training task instances. SWE-Bench only includes environments and test cases for its test set, not its training set. Thus, we did not conduct these ablation studies as training on any subset of SWE-Bench’s test set would be problematic.

How do you justify the chosen evaluation criteria in light of the human verification already present in SWE-Bench, and what additional benefits does SWE-Gym provide?

We want to clarify that throughout our paper, we exclusively use SWE-Bench (both Verified and Lite versions) as our evaluation framework. This choice is deliberate as SWE-Bench provides human-verified test cases that serve as a reliable, standardized benchmark for measuring agent performance. SWE-Gym complements this by uniquely providing an effective training dataset with executable environments that enable end-to-end agent training without contaminating our evaluation data.

Essential References Not Discussed and Comparisons against existing methods evaluated on SWE-Bench We appreciate the reviewer pointing out these references. We would like to clarify that we have discussed both references in our paper: SWE-agent [1] in line 102 and Agentless [2] in line 093.

However, we acknowledge that our comparison with these works could be more comprehensive. In the next version of our paper, we will expand our analysis to include a more detailed comparison of our results with these frameworks. Essentially, although these approaches demonstrate that better prompts and agent scaffolds can enhance performance, our work shows that horizontally, end-to-end training—without relying on manual prompt design—can yield even greater improvements. We will also present their performance on SWE-Bench to better contextualize our contributions.

Additionally, for concurrent works on SWE Agent training, we include a detailed comparison in appendix section A.

computational budget constraints limited the number of training trajectories to 491, which may affect the generalizability of some findings.

We would like to emphasize that our results represent the state-of-the-art open-model results at the time of submission, and used a substantial compute budget exceeding $30K. The challenge of collecting more training trajectories stems from the inherent computational intensity of agent training research, particularly when working with real-world software tasks that require full execution environments.

Importantly, our scaling experiments in Sections 5.1 and 5.2 demonstrate consistent log-linear scaling behavior, which provides strong evidence for the generalizability of our findings beyond the current dataset size. Our experiments in Section 5.2 suggest that the task diversity of SWE-Gym is not a limiting factor in further model improvements.We believe these results offer valuable insights that will motivate and guide future open research in this direction.

Additionally, we note that these 491 trajectories are positive ones used for agent training. In fact, we used over 1000 trajectories (both positive and negative) for training the verifiers.

Limited exploration of alternative fine-tuning methods

We appreciate the reviewer's insightful suggestion regarding alternative fine-tuning methods. We would like to clarify that the primary contribution of our work is the SWE-Gym dataset itself, which addresses a critical gap in the field by providing executable environments for real-world software engineering tasks. While we demonstrate the dataset's effectiveness through recently-proposed approaches for model improvement via agent training and test-time scaling, these implementations serve as solid baselines rather than exhaustive explorations of optimal training techniques.

As one of the first papers to study SWE agent training on real-world tasks, we deliberately established strong baseline models using well-understood training approaches (Zelikman 2022, Singh 2023, Pan 2024) to provide clear evidence of SWE-Gym's effectiveness. Our results show substantial performance improvements (up to 19% absolute gains in resolve rate) using these straightforward methods, which we believe validates SWE-Gym's value as a training resource.

We agree that exploring more sophisticated training paradigms represents a promising direction for future research. By releasing our dataset, trained models, and agent trajectories publicly, we aim to facilitate such explorations by the broader research community.

How SWE-Gym could be extended to other programming languages beyond Python in future work.

We thank the reviewer for this valuable suggestion. In our future work section, we will add a comprehensive discussion on ways to extend SWE-Gym beyond Python. We envision two promising approaches: (1) establishing a collaborative community effort to systematically collect and curate datasets across multiple programming languages; or (2) developing specialized environment-setup language model agents that can automatically analyze repositories, identify dependencies, and construct executable environments for diverse programming languages.

I see two addition as compared to boarded literature i) Addition of verifiers in fine-tuning of agent and ii) adding unit tests for part of training data

We thank the reviewer for pointing out these contributions. We will update the related works section to include a more detailed comparison of our results with these literature.

We thank Reviewer aRo7 for their insightful feedback. Below, we address the primary concerns and suggestions provided.

I think Novelty is limited given that swe-bench training data already provide training data

To clarify, SWE-Bench doesn’t include unit tests or executable environments for their training data. Its training data is insufficient for effective agent training because it lacks both executable environments and test case for those instances. Thus, we put in the work to create a suitable training environment. We decided to focus on our own set of repositories, rather than the ones included in SWE-bench, to avoid contamination concerns.

Our work is one of the first to study training SWE agents on real-world software engineering tasks, which is a significant departure from existing work that focuses on evaluation or prompting-based agents. We also achieve state-of-the-art open-model results on SWE-Bench, with log-linear training and test-time scaling results.

These distinctions make SWE-Gym a novel and valuable contribution to the field of software engineering agents.

Precision and Recall Analysis:

We appreciate the suggestion to include precision and recall metrics. While our evaluation follows established protocols from prior work (Cobbe et al., 2021), we agree that precision-recall curves would provide more comprehensive insights into our verifiers' performance. We are currently working on these curves and will soon follow up with the updated results.

Regarding intermediate step evaluation, developing effective process rewards for software engineering agents is still an open problem. We would welcome the reviewer's insights on potential approaches to this problem, and would love to incorporate these in the future work.

Clarification Regarding pass@k:

We clarify that pass@k (k≥2) solely serves as a reference point to compare our learned verifiers against an oracle verifier that always selects the optimal solution. This metric was not used for our primary results or for comparisons with other methods.

Is this training data verified or not, i.e., does it contain sufficient information for the issues to be resolved? It may be the case that LLM learns to hallucinate if trained on this since there may be training data which doesn't have sufficient information

Our data is as verified as in the original SWE-Bench paper. As described in Lines 152-186, following SWE-Bench’s task construction pipeline, we only keep around issues that have a gold-standard PR, which indicates that the issue was able to be resolved by a human programmer based on the information provided. Also, as shown in Table 9 in Appendix, claude 3.5 sonnet without any SWE-Gym specific training achieves the reasonable performance of 29.1% on SWE-Gym Lite within 50 turns, suggesting that a significant proportion of the task in SWE-Gym is solvable. Furthermore, our ultimate evaluation of our trained model is on SWE-Bench, not SWE-Gym, which is quite out of distribution from SWE-Gym, so any repo-specific hallucination learned through SWE-Gym wouldn't explain our improved SWE-Bench performance. We’d be happy to add follow-up experiments if the reviewer has any thoughts on validating this hypothesis.

In Table 3, how many runs are used for showing the confidence intervals? Would be good if confidence intervals could be added to other results or comment on them; given the stochasticity of LLMs, its difficult to conclude from a single-digit number.

To mitigate the stochasticity of LLMs, we apply a consistent random seed and sampling temperature of 0 across all experiments in Table 3. We use a bootstrap test to estimate the standard deviation by sampling 1000 different subsets of the evaluation result. Regardless of confidence estimation, we believe the performance gap before/after fine-tuning is significant. For experiments with higher stochasticity, we plot error regions in Figures 3 and 4 to represent confidence intervals, as described in lines 357-365. Following Lightman et al. (2023), we estimate the uncertainty as detailed in Appendix B.2.

In the abstract, can you add the model name and the number of parameters along with improvement numbers?

We thank the reviewer for the suggestion. We will update the abstract to include the model name and the number of parameters.

only~2.5K training samples

We kindly request the reviewer to refer to our first response to Reviewer BE56, where we discuss how our dataset is already effective for model training and achieves state-of-the-art results. Importantly, our experiments in Section 5.2 directly show that the task diversity of SWE-Gym is not a limiting factor in further model improvements.