Commit0: Library Generation from Scratch

We thank the reviewer for the thoughtful review!

W0: Solving commit0 is too costly.

• This is a valid concern. For this reason, we make the lite split of the benchmark.
• We expect the costs of running LLMs to continue decreasing significantly. As evidence: in 2022, the leading model, text-davinci-003, was 100 times more expensive to run than GPT-4o mini in 2024, despite GPT-4o mini being a much stronger model.
• The following table shows how much it costs to run SDE-I on lite and all.

Table 1: costs to run SDE-I on commit0 lite.

Table 2: costs to run SDE-I on commit0 all.

W1: Commit0 doesn’t consider producing specifications and unit tests.

• We note that there are indeed many scenarios in software development where a spec exists and the coder has to implement it, such as implementing a new programming, database backend, or a web browser. This is often referred to as test-driven development (TDD). Therefore, an agent that performs well on commit0 demonstrates its utility in TDD contexts.
• We also agree that a current limitation of Commit0 is that the specification and unit tests are given. This is a design choice we explicitly made in order to ensure that evaluation is unambiguous, repeatable, and robust. Note that our benchmark is still very difficult, requiring code generation at a much larger scale than previous benchmarks.
• In future work, we will investigate relaxing the assumption that specifications and tests are fully observed. Robust evaluation given incomplete specifications is an open problem and we are very excited about it. Indeed, methods like Reflexion -- where models iteratively improve their outputs through self-assessment -- are valuable for enabling agents to generate and refine their own specifications and tests. These methods can help models not only generate code but also reason about its correctness, completeness, and alignment with the intended functionality.

W2: Fair comparisons of baselines require setting fixed budgets / need more extensive study on test-time scaling approaches for commit0

• Thank you for highlighting the importance of controlling computational resources when comparing different SDE-I agents. We agree that to ensure fair comparisons, it's crucial to account for the amount of compute associated with each approach.
• While measuring compute by calculating the exact number of FLOPs would be ideal, it's not feasible due to the proprietary nature of some models. We thus address this by approximating compute usage through the dollar cost incurred during model evaluations. We will plot the average pass rate against the total cost for each approach on our leaderboard. This allows us to visualize the trade-offs between performance and computational expense, effectively identifying methods that lie on the Pareto frontier. By doing so, we can highlight approaches that offer the best performance for a given cost.
• Thank you for suggesting test-time scaling approaches for commit0. We note that researchers are also exploring varying levels of test-time computation for SWE-Bench. Considering the difficulty of solving commit0, it would be exciting to see advancements using these test-time scaling methods. We encourage researchers to explore these strategies, and on our leaderboard, we will recognize approaches where unit test pass rates effectively scale with increased test-time computation.
• As the reviewer suggested, we plot (1) independently generating each module 1, 3, 10 times and picking the best implementation based on pass rates, and (2) iterating 1, 3, 10 times to fix errors. We plot these six runs in here: https://drive.google.com/file/d/1xjXgreigXSzmL5F7EZePf4MWjIJKWYPT/view?usp=sharing

Table 3: Best of $n$ independent generation.

Table 4: Iterative refinement based on test feedback.

I read the updated paper and the comments from the authors answering the many questions. I think the paper has improved and have increased my score accordingly, I hope to see it accepted to the conference. I think this benchmark sets up a framework to push the LLM community forward in solving more difficult and realistic coding tasks and provides example baselines for folks to easily build on with their own ideas.

W1: Current generation unit might be sub-optimal.

• Thank you for your feedback regarding our choice of using entire modules as the unit of generation. We understand your concern that modules in popular repositories can be quite large, potentially making them complicated for models to generate and possibly affecting the results of our ordering experiments. We would like to clarify a few points:
  • Module Size and Context Length: The ablation study is conducted on the lite split, where the modules and their imported dependencies are relatively short. When generating code in random order and topological order, we include content from imported modules. In both cases, generating entire modules, with their imports included in the context, does not exceed the model's context length limitations. The only difference lies in whether these included modules have their implementations filled in at that point in the generation process. This only results in minor length differences.
  • We have also shown that including content from imported modules is helpful. See Figure 4, excluding content from imported modules results in fewer unit tests passed.
• Generation unit is not the smaller the better: We have observed that selecting the appropriate generation unit is crucial for performance. We experimented with filling in code at the function level, but this approach produced significantly worse results than filling in at the module level. When generating code by function, models focus solely on the current function and often overlook interactions with other functions. Conversely, generating code by module enables models to develop a holistic understanding of all functions within a file. We will provide a qualitative analysis in the Appendix for further explanations.

Table: Comparison between filling in by module and filling in by function using Claude Sonnet 3.5 in stage 1.

Q0: Whether spec is included in stage 1 experiment?

• We apologize for the confusion and will clarify this in the paper. In the stage 1 experiment, we do not include the spec docs; we only provide models with the module to be filled in. The prompt is

Here is your task: You need to complete the implementations for all functions (i.e., those with pass statements) and pass the unit tests. Do not change the names of existing functions or classes, as they may be referenced from other code like unit tests, etc. When you generate code, you must maintain the original formatting of the function stubs (such as whitespaces), otherwise we will not able to search/replace blocks for code modifications, and therefore you will receive a score of 0 for your generated code.

One important detail is that, in the module, there are function stubs and docstring descriptions (for example: https://anonymous.4open.science/r/minitorch-3E5F/minitorch/autodiff.py). Therefore, the spec docs are not necessary for understanding the spec of the current functions but can be a useful resource for obtaining a holistic understanding of the entire library.

We thank the reviewer for the thoughtful review!

W0: Missing actionable insights: what are the new insights commit0 can bring in addition to previous benchmarks?

• Actionable direction 1: Modular Code Generation Strategies. Commit0 presents a significant increase in task complexity compared to benchmarks like HumanEval, SWEBench, and R2E. While those benchmarks focus on generating one or a few functions -- manageable via static one-shot code generation -- commit0 requires generating an entire codebase consisting of numerous interdependent functions. This complexity introduces challenges that cannot be addressed by existing methods.
  • Challenge: Generating all functions in one shot is impractical due to context window limitations and the intricate dependencies between functions.
  • Primitive experiments: We conducted preliminary experiments where we treated individual files as the unit of generation and generated these files in topological order based on their dependencies.
  • Findings: Generating code at the file level allowed some unit tests to pass, indicating progress over attempting to generate everything at once. While ordering files topologically didn't yield immediate improvements, it highlights a promising direction for optimizing generation sequences.
  • Future Direction: Research into optimal generation units (e.g., classes, modules) and order could lead to more efficient code generation strategies.
• Actionable direction 2: Advancement of Long-Context Models. An alternative approach to generating large codebases is to utilize long-context models. commit0 can thus also function as a benchmark for long-context capabilities.
  • Challenge: The current models have an effective maximum generation length of approximately 4k to 16k tokens, which is considerably shorter than that of a real-world codebase.
  • Future Direction: An alternative approach to generating large codebases is to utilize long-context models. commit0 can thus also function as a benchmark for long-context capabilities.
• Actionable direction 3: Enhanced Error Handling Techniques. In benchmarks like SWEBench, debugging is typically confined to minor issues in mostly correct code (given that a correct codebase already exists), allowing models to focus on errors near the top of the call stack. In contrast, commit0 starts from an empty implementation, meaning errors can originate from any level within deep call stacks. This complexity requires models to adopt more sophisticated debugging strategies.
  • Challenge: With errors potentially existing at any level, models need to efficiently navigate and rectify issues throughout the entire codebase.
  • Primitive experiments: We experimented with providing models with type errors from linters before presenting them with actual execution error traces.
  • Findings: This approach improved model performance by helping them identify problematic areas early on.
  • Future Direction: Investigating how models can better plan and prioritize when faced with long and complex error traces could lead to advancements in effective debugging.

Thank you for the detailed response.

While the insights you have detailed are correct, I still think it is quite limited to (1) managing context or (2) handling errors -- which overlap with several other code benchmarks. The generation unit aspect is interesting, but the current results with module-level generation don't reveal much as of now.

We experimented with filling in code at the function level, but this approach produced significantly worse results than filling in at the module level.

This is interesting but could be result of how whether or not an invariant during generation is maintained. For instance if a function foo(a: T1, b: T2) with argument types T1 and T2 was generated first, the model needs to maintain foo as an invariant while generating a function bar that calls foo. The right experiment would provide the model enough context of the previously generated code as generation progresses.

Overall, I appreciate the detailed responses, but would like to maintain my score.

We thank the reviewer for the thoughtful review!

W0: Limited results on the all split of commit0.

• Obtaining results for all stages of SDE-I using current state-of-the-art models is prohibitively expensive. Running o1-preview for the first and the third stages on the lite split alone cost $913, and we are limited by our budget. However, to do the best we can to test out all stages of SDE-I on the all split, we test Deepseek V2.5 and GPT-4o-mini.

Table 1: Results on commit0 all split using DeepSeek V2.5 and GPT-4o-mini.

W1: Limited evaluation on other agents.

• We would like to note that none of the existing agents can automatically work for library generation without modification.
• To include an evaluation of another agent, we selected the top agent on the SWE-bench leaderboard as of today. The leading agent is OpenHands + CodeAct v2.1 (claude-3.5-sonnet-20241022), released on October 25th. We collaborated with the OpenHands team to ensure the best performance of their agent. The results are provided below.

Table 2: New results on OpenHands.

W2 & Q1: Missing commit0 examples and trajectories.

• Commit0 example: https://anonymous.4open.science/r/minitorch-3E5F/README.md. A specific processed module can be seen here: https://anonymous.4open.science/r/minitorch-3E5F/minitorch/autodiff.py
• Example trajectory on all stages of tinyDB with Sonnet: stage 1 (https://drive.google.com/file/d/1DVpK3yVXvG0zSnMFzOIB-B6wvhC47BJd/view?usp=sharing), stage 2 (https://drive.google.com/file/d/15G56gA5xLnSwC3O7PRuJDwVP_Niq1gJ0/view?usp=sharing), stage 3 (https://drive.google.com/file/d/1AgKcz8_YGIFy3zeoSX3J3HepicKTIx5P/view?usp=sharing)
• We will update the appendix to include links to all examples and all trajectories.

W3: Missing costs of SDE-I agents.

• Thanks for the great suggestion. We will add the following table to the paper.

Table 3: Costs for running the SDE-I agent across different stages.

W4 & Q0: Missing description for how to prevent agents from accessing source code.

• We automatically verify whether an agent has retrieved information from GitHub by detecting the presence of 'github.com' or 'raw.githubusercontent.com' in the retrieval URLs. If these URLs are detected, the corresponding generation will be voided. Thus far, the agent hasn’t retrieved code from GitHub. We will clarify this in the paper.

W5: Feeding the first 10k tokens of spec docs to agents might be sub-optimal.

• Thanks for the great suggestion. We have added a retrieval experiment where we first tokenize the specification, and break it into chunks of 1000 tokens. When generating a module, we retrieve the top 10 relevant chunks and include them as part of the context. Using BM 25 to retrieve the most relevant 10k tokens does lead to an improvement.

Table 4: Feeding specification documents with a retrieval component.

Thank you for your response.

We would like to note that none of the existing agents can automatically work for library generation without modification.

This is not true, as can be shown by the high results of OpenHands on Commit0 Lite.

We selected the top agent on the SWE-bench leaderboard as of today.

It seems that OpenHands has higher performance on both Commit0 Lite and All compared to SDE-I. I have several questions:

• What's the cost of OpenHands on Commit0.
• Given that SDE-I is specifically tuned for library generation, what could be the reason for OpenHands achieving much higher performance than SDE-I.

Your responses to W0, W2, W3, W4, and W5 addressed my concerns. However, it would be good to submit a revised version of your paper for the rebuttal, with the promised additions.