Q1: Evaluation metrics.

A1: Thanks for your positive feedback on our paper. We have additionally provided more evaluation results on General Response (G.Q2).

Q2: Correlation between CodeBLEU scores and the more traditional EM and ES metrics.

A2: Firstly, the definitions of EM and ES are as follows: Exact Match (EM) evaluates the percentage of cases where the prediction exactly matches the reference, providing a strict measure of how often LLMs produce correct code without deviations. Edit Similarity (ES) measures the similarity between the prediction and the reference based on the number of edits required to transform one into the other, typically using metrics like Levenshtein distance.

However, EM and ES metrics only assess the textural similarities and ignore semantic structure similarities among predictions and ground truth. Specifically, the limitations of EM and ES are as follows: (1) The perfect accuracy of EM is too strict, and underestimates different outputs with the same semantic logic. (2) ES score does not take into account the grammatical and logical correctness, resulting in favoring candidates with high ES accuracy and serious logical errors.

To address these limitations, the CodeBLEU is proposed, which considers information from the shallow (n-gram) match, the syntactic match, and the semantic match. More specifically, the n-gram match assigns different weights for different n-grams, the syntactic match considers the abstract syntax tree (AST) information in the evaluation score by matching the sub-trees, and the semantic match uses data-flow structure to measure the semantic similarity.

To analyze the correlation between CodeBLEU scores and the more traditional EM and ES metrics, we analyze the EM/ES/CodeBLEU scores for six models using retrieval in the following table, where we define completion on 1 line, completion on 2-3 lines and completion on 4-5 lines as easy, middle, and hard settings, respectively.

The benchmark results (EM/ES/CodeBLEU) are as follows:

Then, we compute Spearman's Rank correlation coefficient as the ranking relationships between CodeBLEU and EM/ES scores, where a larger value denotes greater correlation. The Spearman's Rank correlation coefficient results are as follows.

We observe that Spearman's Rank correlation coefficient is lower when increasing the difficulty (completion line), which is also reasonable. The possible reason is as follows.

The completion span with larger completion lines usually includes more complex code syntax and semantic structure information, which is considered in the CodeBLEU metric. However, the EM/ES does not consider this structure information.

Q3: Proprietary model comparison.

A3: We have provided a detailed comparison with recent powerful proprietary models in General Response (G.Q1). Please refer to General Response (G.Q1) for more details.

Q4: Qualitative error analysis.

A4: Thanks for your insightful advice. We have provided a detailed qualitative error analysis in Appendix A.7 in our new version.

Q5: Figure Clarity.

A5: Thanks for your suggestions. For Figure 4 in the original version, we have replaced the figure with a table for better illustration. For Figure 14 in the original version, we have greatly increased the font size and improved the illustration in Figure 12, Figure 13, and Figure 14.

Q1: Data leakage concerns.

A1: Please See General Response (G.Q1).

Q2: Repository-level coding.

A2: In Lines 209-210 of Section 3.2, we discard test samples that could be exactly predicted by DeepSeekCoder-1.3B without cross-file contexts. Meanwhile, to discuss more clearly, we also use three strong code LLMs (i.e., DeepSeekCoder-6.7B, StarCoder-7B, and DeepSeekCoder-33B) to analyze the ratios of evaluation cases with or without using repository-level contexts. Specifically, we prompt DeepSeekCoder-6.7B, StarCoder-7B, and DeepSeekCoder-33B using the in file contexts of each sample and obtain three predictions. Then, if one prediction exactly matches the ground truth, this sample is considered to be predicted without requiring repository-level contexts. Finally, we observe that 71% of samples cannot be well predicted only using in file contexts, which indicates that our dataset is challenging. We have added this detail in our new version in the Appendix A.6.

Q3: Evaluation metrics.

A3: Thanks for your suggestions. Please See General Response (G.Q2).

We have cited these provided works [1] [2] and updated the above discussion in our new version.

[1] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374

[2] Jain, Nihal, et al. "On Mitigating Code LLM Hallucinations with API Documentation."arXiv:2407.09726

Q4: Analyses in lines 416-428 are quite obvious.

A4: Thanks for your suggestions. We have rewritten the analyses by removing useless analyses in our new version.

Q5: Typos.

A5: We have revised these typos in our new version.

Q6: Evaluation of large models.

A6: Please See General Response (G.Q1).

Q7: More insights from multi-lingual transfer experiments.

A7: We have provided the detailed results of different languages in the new version (Fig. 22 and Fig. 23). Below, we also provide the results of relative improvements after using Python-only tuning, and the observations are as follows. (1) We observe that the improvement of HTML is the highest. For this phenomenon, we assume that HTML is a markup language with a few simple syntax rules, which is easy to learn. (2). The improvement in Python language is significant using Python-only tuning, which is straightforward. (3). Most high-resource languages (e.g., Go, C++, C#, Scala) benefit a lot based on python-only tuning. For this phenomenon, as code LLMs have good fundamental capacities for these high-resource languages due to the large pretraining quota, it may be easy to promote the emergent code completion abilities of other languages based on the strong fundamental abilities of these high-resource languages. (4). The improvements on these low-resource languages (e.g., Rust, Haskell, Lua) are usually relatively less than the improvements on high-resource languages.

Finally, it should be mentioned that the cross-lingual transfer abilities are influenced by many factors (e.g., fundamental abilities of base LLMs for different languages, syntax similarities among different languages, data qualities, and quantities of tuning stage). Therefore, we think that it is not easy to predict well ahead of time. In the future, we will continue to investigate the cross-lingual scaling law among different languages for both pretraining and tuning stages to better explain the cross-lingual transfer abilities for code intelligence.

Hi, Reviewer uHra,

We believe we have addressed your concerns carefully. If you have other questions or comments, please let us know. We are very glad to solve your concerns.

Thanks for your insightful suggestions.

Q1: Data contamination issues.

A1: Thanks for your advice. Please See General Response (G.Q1).

Q2: A more realistic approach would be to allow arbitrary completion cursor positions while ensuring the completed code forms a valid AST node, similar to previous work cited in the paper.

A2: Thanks for your comments. As mentioned in many works [1, 2], developers often expect LLMs to complete the current code into a complete snippet, such as a completed code line or loop block, instead of suggesting an incomplete code snippet. Besides, the recent Qwen2.5-Coder adopts a similar way with M2rc-Instruct to produce the instruction dataset. Specifically, in Section 4.2 of Qwen2.5-Coder, the authors mentioned that ``Inspired by programming language syntax rules and user habits in practical scenarios, we leverage the tree-sitter-languages to parse the code snippets and extract the basic logic blocks as the middle code to infill''. Meanwhile, to demonstrate the effectiveness of our M2rc-Instruct, we also constructed a test dataset called M2rc-Eval (Random) based on your comments. Specifically, for a fair comparison, based on the same repositories of M2rc-Eval, we randomly choose arbitrary completion cursor positions while ensuring the completed code forms a valid AST node, where the generated testing set is named M2rc-Eval (Random). The evaluation results are as follows:

We observe that higher performance is obtained in M2rc-Eval (Random). For this phenomenon, the possible reason is as follows. In each completion span, the completion cursor position of our default M2rc-Eval is the start position of the syntax node block, which means that no context can be used inside the syntax node block. In contrast, the completion cursor position of M2rc-Eval (Random) is the arbitrary position of the syntax node block, which means that there exist additional informative contexts inside the current syntax node block for better completion. In other words, in M2rc-Eval (Random), the additional contexts are inside the syntax node block and before the completion cursor position, which can decrease the completion difficulty and improve the completion quality. In our new version, we have included the above discussion.

[1]. Qwen2.5-Coder Technical Report

[2]. aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Completion

Q3: Justify why this particular bucket granularity (i.e., 10).

A3: We have analyzed the minimum, maximum, and average depths of these 18 programming languages, and observed that depths of AST for different languages are different greatly, where the minimum depth is from 1 to 7, and the maximum depth is from 23 to 51. Therefore, we cannot use the node's layer number in the AST as the fine-grained annotation well. Besides, we observe that the average depth of different languages is from 9.7 to 20.2, and the overall average depth is 14.6. In our implementation, to explain clearly, we just chose 10 as the bucket number. Note that we will provide the tree depth number for each completion sample, and users can select the corresponding bucket number freely.

Q4: Discussion of language-specific insights.

A4: We have classified 18 programming languages in M2rc-Eval into 5 programming paradigms and 9 application scenarios as follows:

Based on the above programming classification structure, we also report the EM results as follows, and have the following observations:

(1). For different programming paradigms, we observe that the markup language paradigm has the lowest performance, and the functional paradigm has the best performance. Besides, after tuning, the performance of the markup language paradigm improves greatly. We assume that the syntax rules for markup language are easy, and these code LLMs can quickly obtain these rules after tuning.

(2). For different application scenarios, the performance varies significantly. Specifically, the Web Frontend and Scientific Computing have relatively low performance, which needs to be improved for existing code LLMs.

(3). We observe these LLMs share some common strengths and weaknesses and we will continue to investigate more language-specific insights to better improve the code completion abilities of existing code LLMs.

Q5: Elaborate on repo splits (line 204) across eval and instruct.

A5: In Line 204, we only provide the number of repositories for testing split, and the numbers of repositories for train, validation, and test sets are 37439, 1635, and 5993, respectively. We have updated this detail in our new version.

Q6: Degradation issue on ES for StarCoder-7B, Code LLama-7B, DeepSeekCoder-6.7B.

A6: As shown in the General Response (G.Q1), we observe that the recent strong general LLMs (LLama3.1-70B, Qwen2.5-72B, GPT-4o, Claude 3.5 Sonnet, DeepSeek-V2.5) and code LLMs (DeepSeekCoder-33B, Qwen2.5-Coder-7B, Qwen2.5-Coder-32B) usually obtain stable and sufficient improvements on both EM and ES metrics when introducing retrieval strategy. In contrast, the capabilities of original base models (e.g., StarCoder-7B, Code LLama-7B, DeepSeekCoder-6.7B) are relatively weaker, and the improvements are not stable. Notably, we observe that StarCoder-7B and DeepSeekCoder-6.7B achieve better performance on both EM and ES metrics in M2rc-Eval-2403 and M2rc-Eval-2406 splits.

Q7: Issue of Line 445.

A7: Thanks for your pointing out this misleading issue. We have removed the corresponding description in our new version.

Q8: Better illustration of Figures 4 and 5.

A8: Thanks for your suggestions. For Figure 4 in the original version, we have replaced the figure with a table (Table 2) for better illustration. For Figure 5 in the original version, we have reduced the number of languages in Figure 4 and added a table (Table 6) for better illustration.

Dear Reviewer TS5v,

Hello! We believe we have addressed your concerns carefully. If you have other questions or comments, please let us know. We are very glad to solve your concerns.

Thanks for your insightful suggestions.

Thanks for your feedback.  For the new questions, the responses are as follows: 

(1). As mentioned in the previous response, generating unit test cases and providing execution sandboxes for repository-level code completion are very challenging. Moreover, existing open-sourced sandboxes usually focus on file-level or function-level code execution and there are no execution sandboxes, which support repository-level execution. In our implementation of these 17 languages, we first prepare corresponding docker execution environments for the repositories for these 815 samples in G.Q2. Then, as the rebuttal phase is short, the authors of M2rc-Eval manually execute unit-test cases for these samples. 

(2). For the scalable issue of this benchmark, as the benchmark usually includes a relatively small number of samples, we have prepared the docker files or requirements for each repository, which are essential for execution. In our future work, we will continue to investigate how to execute repository-level unit-test cases automatically. Notably, if we can automatically build the sandbox environment based on the provided docker files or requirements, we can easily scale up the benchmark size. 

(3). For the coverage issue, we think that it is not necessary to compute the coverage metric for repository-level code completion. Specifically, the coverage metric is usually used to measure the proportion of executed lines of code for each function body. However, the completion code is usually a part of the function body, where the number of code completion lines is usually less than 5. In this way, in the process of collecting test cases in G.Q2, we ensure that the test cases execute all code completion lines, and we do not need to keep all lines in each function body executed.

We will add the above details in our new version.

If you still have questions, please let us know. We are glad to solve your concerns carefully.

Dear Reviewer TS5v,

As the discussion deadline will end soon, please let us know whether our responses have addressed all your new questions. Moreover, as you have mentioned that we have addressed most of your questions, could you reevaluate our work and improve your rating?

Additionally, we believe that our M2rc-Eval has improved a lot based on your insightful and constructive suggestions.

Thanks again for your valuable efforts.

Q1: Details of env setup, tooling, and sandboxing.

A1: Thanks for your suggestions. In our main paper, we mainly use the EM, ES, and CodeBLEU metrics to evaluate the performance of different LLMs, where EM and ES mainly assess the textual similarities and the CodeBLEU additionally considers the semantic structure. Besides, computing these metrics does not rely on any execution sandbox and we just follow repositories of CrossCodeEval [https://github.com/amazon-science/cceval] and CodeBLEU [https://github.com/k4black/codebleu] to obtain these metrics. We have provided these details in our new version.

Q2: The evaluation is still based on surface form metrics like exact match. Scaling execution tests can be hard.

A2: Please See General Response (G.Q2).

Q3: Evaluation of more strong recent models.

A3: Please See General Response (G.Q1). In general response, we have provided more evaluation results of different LLMs in our new version. Note that as Yi-Coder [https://huggingface.co/01-ai/Yi-Coder-9B/blob/main/tokenizer_config.json] has not supported the FIM, we have not provided the results of Yi-Coder.

Q4: How well the benchmark can be solved if the model just placed all the most pertinent files in context instead of just retrieving a few short snippets?

A4: Thanks for your suggestions. We take the Qwen2.5-Coder-32B with 32K length as an example to test the results in the validation set of M2rc-Eval with longer input as follows. Specifically, for Qwen2.5-Coder-32B (Infile), we only use the infile context for code completion. For Qwen2.5-Coder-32B (Snippet Retrieval, 4K), we follow the same retrieval setting by retrieving relevant code snippets to construct a completion prompt with 4096 tokens. For Qwen2.5-Coder-32B (File-wise Retrieval, Topological, 4K), we just use the relevant pertinent files based on the similarity scores and order these files based on the topological sort algorithm in DeepSeekCoder. For Qwen2.5-Coder-32B (File-wise Retrieval, Random, 4K), we directly randomly order the relevant files to construct the prompt with 4096 tokens.

Similarly, we also report the results of 8K, 16K, and 32K lengths and we have the following observations.

(1). First, when increasing the completion prompt length, lower performance results are obtained for all retrieval methods, which means that the Qwen2.5-Coder-32B cannot handle well in long input although the input length is less than 32K. This phenomenon is similar to the conclusion of many general long-context understanding methods (e.g., Ruler[1], LV-Eval[2], STRING[3] ), which claim that the really effective length is less than the claimed length a lot.

(2). Second, the default snippet retrieval achieves the best performance in 4K length. We assume that the file-wise retrieval will include many noisy or redundant contexts, which introduces more challenges to obtaining the informative contexts for repository-level code completion.

(3). Third, the ordering method based on the topological sort is relatively better than the random baseline in the file-level retrieval setting, which indicates that it is effective to use Topological ordering and it may be a good choice to investigate a more suitable ordering strategy for these retrieved code snippets.

[1]. RULER: What's the Real Context Size of Your Long-Context Language Models?

[2]. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K.

[3]. Why Does the Effective Context Length of LLMs Fall Short?

Q5: The cross-lingual transfer benchmarks may not be properly baseline. The paper uses StarCoder-1, a multi-PL pre-trained model that is continually pre-trained on Python code. This is a rather non-standard setting and would be fairer (across langs) if it were started from StarCoder-1-Base.

A5: Sorry for this misleading. It should be mentioned that we just use the base model of StarCoder, which is not continued pretrained on Python code. We have clarified this detail of Appendix A.3 in our new version.

Q6: Explore the effect of various code retrievers.

A6: To find the most suitable retriever, we evaluate the code match (EM/ES) performance on the validation set with Jaccard similarity, BM25, and UniXCoder in the following Table. Specifically, the UniXcoder is a neural retriever, which costs more response time for inference. In contrast, both Jaccard and BM25 are lexical retrievers, which can reduce infer time greatly when compared with UniXCoder.

In the following Table, we observe that UniXcoder has not achieved better performance when compared to lexical retrievers. Besides, the results of Jaccard and BM 25 are comparable. Moreover, for Jaccard and BM25, we also benchmark the retrieval time consumption based on Intel Xeon CPU E5-2682 v4 with a base frequency of 2.50 GHz, where the results are 2.94ms and 3.14ms, respectively. Therefore, we choose Jaccard as the default retriever in our M2rc-Eval.

Hi, thanks for your quick feedback. We have updated the results based on 17 languages in General Response (G.Q2). Note that HTML language cannot be executed, and we do not provide execution test samples for HTML.

If any concerns remain, we would be grateful for further clarification and are happy to continue the discussion in this rebuttal process.

Thank you for getting back with the experiments regarding pertinent files in context. I would have been persuaded to raise my score had this been a method paper.

However, the Java-only exec-based testing added during the review period in my view is not enough for a paper in the Datasets and Benchmarks track. This work would be ready for release if the authors were to take the effort to scale up synthetic test generation in all the languages and follow that up with human validation in a manner done by existing industry-standard benchmarks [1].

I really believe that there is a niche for a massively multilingual repo-level code generation benchmark. Especially one that is execution tested. I hope the authors will rework and resubmit what I believe can be a very strong and well adopted benchmark but for now I have decided to retain the same score.

[1] Terry Yue Zhuo, et.al.: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. CoRR abs/2406.15877 (2024)

This paper has improved during the review process. With the understanding that the authors will incorporate the 17 language collection, filtering and evaluation details into the final version, I have raised my score to 6.

Thanks the authors for the detailed response.

Could you please clarify how you were able to do execution in 17 languages? Different languages require different ways to build and test, how would you make it scalable enough as a benchmark? Further, how would you validate that these unit tests have high coverage that could ensure the reliability of the result?