RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems

Dear Reviewer mVr1,

We appreciate the time you invested in reviewing our paper. Your comments have provided valuable perspectives that we have carefully considered in this response.

The novel research contributions of the paper are a bit limited, especially for an AI conference.

We would like to highlight the significance of our work:

1. We hope that RepoBench can address the evaluation of code generation in LLMs at current stage. This is a key area in the field of AI, as understanding and improving code generation capabilities are vital.
2. RepoBench-C can be used to facilitate tasks involving long-context generation. The customizable token number feature of RepoBench-C allows the construction of 2k, 8k, 16k and even 128k versions, making it a benchmark for evaluating the efficacy of LLMs in handling long context length. This flexibility is particularly important in the current landscape of AI research, where handling longer contexts is becoming increasingly crucial.
3. We are encouraged by the interest RepoBench has already garnered in the LLM community. The StarCoder team has noticed the performance of StarCoder here, leading to a collaboration study focused on repo-level model training for next version. You can check the preliminary ablation study we already have shown in the global response.

Weak baselines selection of RepoBench-R.

First, we would like to emphasize the relevance of Jaccard similarity as a baseline. It has been revealed through reverse engineering studies online that GitHub Copilot uses Jaccard similarity backend for retrieval.

In our initial experiments, we evaluated both encoder-only and decoder models, such as CodeGPT and CodeGen. While encoder-only models demonstrated superior performance for retrieval tasks, we initially omitted these results due to page constraints. In response to your feedback, we plan to include these findings to present a more comprehensive evaluation.

We are also excited to share that we have conducted additional experiments with more recent encoding models, specifically CodeT5+ and Intruct-XL. The results from these experiments have shown Instruct-XL demonstrates similar performance to UnixCoder.

Table 4: Results of CodeGPT, CodeGen, CodeT5+ and Instruct-XL as retriever on RepoBench-R.

Dear Reviewer zCY2,

Thank you for your appreciation and constructive feedback on our paper.

Opt-Out Possibility and Living Benchmark

We are grateful for your suggestion about the opt-out mechanism and the idea of a living benchmark. Interestingly, we are already moving in this direction and the living benchmark is exactly what we want to present since we have the pipeline for construction and we can always crawl the newest data for benchmarking.

Specifically, we are collaborating with the StarCoder team, and we will present the newest version of RepoBench shortly with all the repositories are created in September, 2023, aligning with the next release of "The Stack." As for the opt-out mechanism, the StarCoder team has implemented a rigorous deduplication process on their new training data, which will also be open-sourced soon.

Alternatively, is there an intention to check and disqualify models that have trained on test set data?

Thanks for pointing out this. RepoBench as a living benchmark as you expected, it allows us the flexibility to continuously update and release new versions (e.g., September version, October version, November version, etc.). So, if a model has been trained and released, we can simply validate it against the latest version of RepoBench. This method ensures an effective check against any potential training on test set data, as models will be evaluated on newly updated benchmarks that they have not encountered before, ensuring the integrity and fairness of the evaluation process.

Thank you for your thoughtful review and valuable comments on our paper. We appreciate the opportunity to clarify and elaborate on the aspects you've highlighted.

It is not clear what new insights RepoBench and experiments on it contribute to the field. Were the results previously unknown or unexpected?

We understand your concern regarding the novel insights provided by RepoBench. Our primary contribution through RepoBench is in addressing the need for benchmarks that evaluate repository-level code auto-completion, a gap that exists in current benchmarks focused mostly on single-file tasks. This focus is particularly relevant given that real-world applications of code models often require a comprehensive repository-level approach for optimal functionality.

Additionally, RepoBench has the ability to construct generation tasks with customizable context lengths (e.g., 2k, 8k, 16k, 64k, etc.). This flexibility allows for an in-depth analysis of model performance under varying long-context scenarios

During our experiments, we did encounter several intriguing findings that were unexpected and might be insightful:

1. Multilingual Code Generation: While current models show promising results in Python, matching Codex in some cases, we observed a noticeable performance gap in Java code generation
2. Long Context Generation: On repobench-c with 8k max length, StarCoder is still trailing behind the capabilities of Codex, highlighting a crucial area for future development.
3. Repo-Level performance: Our initial assumption was that repo-level code generation would be generalizable. However, we found that models like StarCoder did not perform well at this level, suggesting they might need to be trained with repository-level data. This insight led to a collaborative effort with the StarCoder team and our subsequent contribution to the model improvement. We show some of the preliminary ablation study in a global response, which demonstrates that repository-level training can improve the performance dramatically.
4. Retrieve as much as possible: Merely retrieving the gold code snippet is not always the most effective approach. Instead, filling the context with as much 'relevant' code as possible from the repository tends to improve performance.
5. Positioning of Cross-File Context: Unlike long context generation in textual documents where models pay more attention to the beginning of the prompt, in repository-level code generation we found that placing more relevant cross-file snippets towards the end of the prompt (closer to the in-file context) can improve performance.

It concerns me a bit that random retrieval is close to or even outperforms some non-random retrieval methods.

The observation regarding random retrieval occasionally outperforming strategized retrieval, particularly in Java, is indeed intriguing. We speculate that this phenomenon is influenced by the inherent characteristics of the language. In Java, cross-file references often involve class definitions (e.g., public class DerivedClass extends BaseClass {). In this scenario, previous lines might not provide contextual clues (meaningful information) for retrieval. While in Python, cross-file references may often involve more direct contextual clues such as function calls with parameters, making strategized retrieval more effective. For example, if you try to use calculate(left, right), you may define left=? and right=? in the previous lines. Thus, our findings suggest that effective retrieval strategies may vary between different programming languages.

What is exactly "the first appearance of a cross-file line within a file"? Is this the import line? Is this the first line that uses cross-file function?

We acknowledge for the ambiguity in our description. The "first appearance of a cross-file line within a file" refers to the first instance where a cross-file function/class is used in the code, not the import line. So strictly speaking, it should be the second appearance (the first one will always be the import line).

Table 3: Results on RepoBench-C (8k).