RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing

Thanks for your feedback and suggestions!

 

Q1: Performance & Error analysis on Repo-level Code Generation

We sampled 10 failure examples from RepoEval and manually checked the causes of error:

• Instruction not clear: 2/10 (1/10 does not have docstrings)
• package version mismatch (e.g., transformers): 1/10
• problem with boundary cases: 4/10 (3/10 are not specified clearly)
• correct functionality but wrong implementation: 1/10
• wrong functionality: 1/10
• format error (e.g., does not generate a function): 1/10

Among the 10 failure cases, 6/10 are actually due to unclear instructions or unclear package versions, only 4/10 are actually incorrect. In other words, these 6/10 examples cannot even be solved by human. This is a known issue with the RepoEval dataset, where the intention of the functions cannot be fully entailed from the context.

Thus, performance is calculated over the total set of examples, including many that are inherently unsolvable or ambiguous. We believe adjusting for this by considering only clearly solvable cases would highlight a more substantial gain from our method.

We observe similar issues with our RepoST-Eval datasets and present a potential solution: using model-generated docstrings to improve clarity (see Sec. 5 for details).

 

Q2: Repo-level context

It is totally possible to use other types of context without even modifying the RepoST-Eval / RepoST-train datasets. Note that we provide the .zip file containing all the repos, we only need to mask the target function, retrieve relevant context for each function, and perform retrieval-augmented code generation. We listed it as a promising future work in the “ Conclusion and Future Works” section.

 

Q3: Cost

The cost of the RepoST-Eval dataset is 0.14 per example with GPT-4o. Note that the cost can be further reduced with cheaper and stronger models such as o3-mini.

Thank you for your thoughtful and constructive feedback! We would like to address your concerns as follows:

 

Q1: Generalize RepoST to tasks involving multiple components

This is a good question. We did list it as one of the future directions in the “Conclusion and Future Works” section.

In future work, we can generalize RepoST to multi-component feature implementation as follows: (1) In inference, we mask all components directly relevant to a specific feature. Then we apply models to generate code based on the partially masked repository context. (2) When generating the evaluation script, since we already aggregate all dependent components to the evaluation script, no changes are required in the sandboxing stage. In the post-checking stage, we need to check whether the behavior of all the components to generate remains the same. Also, in the test generation step, we need to prompt the LLM to generate tests for the entire feature, not just a single function. Similarly to our current design, the GT modules must pass all the generated test cases. (3) In evaluation, we copy all the generated components to the evaluation script, including the previously masked modules and all standalone functions and classes.

Similarly, we can also generalize RepoST to issue-solving tasks as follows: (1) We extract all modified modules for fixing the bug (e.g., by extracting from an issue-solving PR) (2) Then we sandbox and generate tests for each module separately (3) In evaluation, we check whether the models’ generated patch passes the tests for all the modules

The above extensions show that RepoST can be adapted beyond single-function generation tasks.

 

Q2: Reliance on closed-source AI

While we used GPT-4 to construct our initial dataset, RepoST is model-agnostic and can easily incorporate open-source LLMs (e.g., deepseek-ai/DeepSeek-V3-0324). Our recent experiments on SWE-Bench show that deepseek-v3 is even stronger than GPT-4o on issue solving, a specific repo-level coding task, making it a reasonable alternative model.

 

Q3: The tests are not perfect, which may affect training data quality

You’re absolutely right that LLM-generated tests may occasionally contain flaws. Our human study (Sec. 3.1) found that 1 out of 10 tests labeled "valid" by GPT-4o was actually incorrect.

However, we want to highlight that even with imperfect tests, RFT training still improves model performance. For instance, even lenient tests can ensure that the target output function can compile, run, and produce outputs without errors. In other words, the model will learn to produce code with no runtime/compilation errors from those training examples.

 

Q4: Cases where the LLM-generated tests were incomplete or too lenient and how to improve it

As noted, in our human study (Sec. 3.1), 1 out of 10 tests labeled "valid" by GPT-4o was actually incorrect. In the only failure case, the goal of the target function is to edit some class attributes. Our failure case involves a test function that doesn’t reset the class attributes before calling the model-implemented function. Despite this, this test still filters out generations with compilation/runtime errors.

To improve test quality, one way is to finetune an evaluator model to judge test correctness. Specifically, the training data can be collected using SWT-Bench’s method: (1) collect issue-solving PRs, (2) use an LLM to rollout candidate tests, and (3) label the generated test as correct/incorrect by whether the modified function in the PR can pass the test but its original implementation cannot.

 

Q5: HumanEval is contaminated

We agree that HumanEval is saturated and potentially contaminated in modern LLM training. However, we would like to emphasize that RepoEval is our primary benchmark. It is a repo-level dataset specifically tailored to the repo-to-function generation task. To our knowledge, RepoEval is not as saturated as HumanEval, and better reflects the complexities of real-world development.

Thanks for your thoughtful feedback. We appreciate the time you took to review our paper in detail. Below, we address each of your concerns:

 

Q1: Sandboxing on complex cases and more analysis of API mocking

We agree that the complexity of generated examples is inherently limited by the capabilities of current models. This limitation applies to all automatic environment construction methods, including R2E.

Furthermore, RepoST is still able to sandbox complex functions with complex dependencies. For instance, our most complex evaluation script is 1143 lines long and aggregates context from 8 different files.

To better balance simple and complex functions during training, a possible direction is to generate a large pool of examples and apply downsampling. We leave a comprehensive analysis of this strategy to future work.

 

Q2: The scale of human study is too small

We acknowledge that the human study is limited in size due to budget constraints. Despite the small sample, the results were consistent and supported by strong inter-rater agreement. Specifically, Cohen's Kappa was 0.9179 for functionality checks and 0.7750 for test checks, indicating high reliability.

 

Q3: Novelty compared to R2E & SWE-Gym

The major difference between our method and SWE-Gym is that we present a fully automatic framework, while SWE-Gym requires huge human effort to set up the coding environments.

Compared to R2E, which is also automatic, the major novelty is the design of “sandbox testing”. As introduced in the introduction section, R2E leverages integration testing, which requires setting up the repository environment. Setting up the repository environment is a difficult task, and suffers from low success rate when automated by an LLM. In comparison, we present sandbox testing that tests the target function in a separate script. With sandbox testing, the LLM only needs to install the external packages for the target function and its necessary dependencies (which is much fewer than the entire repository), and can also debug the evaluation script without breaking the original repository.

With an automated framework with sandbox testing, our method achieves significantly better scalability than both SWE-Gym and R2E. Please refer to Table 1 for comparative statistics.

 

Q3: Analysis of failure cases of API mocking

We evaluated the success rate of functions with and without external APIs on RepoST-Eval:

• With external APIs: 34/234 (14.5%)
• Without external APIs: 262/809 (32.4%) The results are expected because in the former case, the model will need to construct the mock API classes from scratch.

We manually checked the examples involving API mocking and discovered three common failure types: (1) the model does not create a mock class, or directly call the original external API in the evaluation script, (2) the mock class has compilation/runtime errors, and (3) the mock class has a different interface as the real external one, so the functionality of the sandboxed function has altered.

It's worth noting that existing approaches like R2E avoid this problem entirely by filtering out such examples at the first step. To our knowledge, RepoST is the first method that actively explores evaluation script construction involving API mocking.

 

Q4: the dataset seems to lack a critical field specifying the exact "necessary packages" (and their versions)

We appreciate your effort in carefully reviewing the datasets. While we do not explicitly list required packages per example, all examples in our dataset can be executed within a common Python environment. We have released Docker images for both RepoST-train and RepoST-eval. Users can reproduce our results by pulling these images.

The exact package versions can be checked by (1) building a docker container based on our image, and (2) inside the container, run pip freeze > requirements.txt to check the python and package environments.

We believe this setup provides a practical and reproducible way to manage dependency information.