Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?

The dataset curated in this work is very small and poorly reflects the stated motivation of learning from the industrial analysis in Sec. 3. The methodology has a number of problems. And the writing includes a number of invalid claims and frequently lacks clarity. These concerns are discussed in order below, followed by a number of minor issues.

The main artifact in this work is the dataset curated by Codev-Agent. This dataset spans just 10 projects, only in Python, across which the test cases cover just 55 files (Sec. 3.3.2). This is an insufficiently large and diverse dataset size to draw meaningful conclusions about model performance. To be clear, automatically generating executable test environments from real repositories is challenging, so it is understandable that the resulting dataset size is smaller than non-executable counterparts. But realistic evaluations need to involve substantially more distinct test cases, preferably spanning multiple programming languages and distinct domains.

On a related note, this limitation stands in sharp contrast to the strong claims made in Sec. 3.3.1, which should be toned down. It is unreasonable to argue that all other benchmarks listed here are not extensible and require manual effort to extend (that certainly does not apply to benchmarks like RepoBench, which scraped 25K repositories from GitHub, clearly not by hand), and the inclusion of an "Agent" column seems redundant with the extensibility one. It is especially improper to state that Codev-Bench's extensibility even applies to repositories with few to no tests in that it just requires writing some test cases (L314), which surely applies to all other benchmarks as well then. Similarly, the argument that RepoMasterEval does not "deep dive" enough into user needs to qualify seems unnecessary, especially given the concerns about the industrial analysis below.

Secondly, the connection between the dataset, at least in terms of the scenarios discussed in Sec. 4, and the industrial/business analysis in Sec. 3 is not clear. The latter analysis mainly shows that completions tend to involve non-control-flow statements, require completing an entire single line, and come with a wide variety of contexts. The actual evaluation considers completions at the block level, where functions, control-flow, and other tatements are lopped in together. Although the appendix subsequently breaks down performance by statement type, it is not clear where the observation regarding completion types factors in. The analysis in this work does not reweight aggregate scores by completion type, or otherwise adjust its conclusions based on Fig. 1.a's distribution. The observation that most completions concern a single line is not utilized at all in the rest of the paper. Nor is the idea that many completions come with just a few tokens of context (itself highly surprising in real-world programming, see the concern below).

On that note, it is quite concerning that the origin and scale of this dataset is not provided so we can evaluate whether it is, in fact, representative of realistic code completion usage. The paper does not discuss what LLM (or other model) is used as a code completion model in the original data, nor under what constraints it operates (e.g. when is it triggered, how well it handles single vs. multi-line completions). Many other details are missing too, such as what exactly a "general statement" is. It would be better to use established nomenclature from program parsers. Similarly, in Fig. 1.B, does "entire lines" mean multiple lines? And why are so many completions "empty"? Completions are normally triggered by edit actions, in which case an empty completion is never correct. In Fig. 1.c, is the "prompt" the entire file-level context at the time the completion is invoked? If so, why are so many contexts (nearly) empty? Completing the first token in a file must be a very rare occurrence. These insights also all seem quite far from "fine-grained". Many empirical studies have been conducted on general properties of code completion queries in relation to model performance. I would expect a fine-grained study to offer more specific insights.

Finally, the prompting setup raises a number of concerns, which are reflected by the remarkably poor results of most models on most tasks. In particular:

• The prompt shown in App. C makes it quite unclear how to leverage the suffix. It only shows a single example, which contains an empty suffix. The prompt should at least include sample-appropriate demonstrations for the different types of scenarios. It may also be noted that using the language of FiM is not necessarily suitable for post-trained models. A more natural format would show the complete code file with a cursor placeholder. This may well be the cause of the pattern the paper notes in which the models continue their completion beyond the target line(s). The example shown in Fig. 7 (App. D.1.) strongly suggests that the model has failed to understand the task, as it copies the input code verbatim after generating the missing line. That points at a prompting failure.
• The prompt also repeatedly tells the model to provide a (non-empty) completion, only mentioning the possibility of an empty response towards the end. That may well be the reason for models often failing to generate empty completions.
• There are a number of other odd statements in the prompt, such as the start ("If you were a code completion agent" instead of a concrete instruction) and the end (which seems to tell the model how to format the output twice, in two different ways).

All this probably place a big role in the remarkably poor performance reported for most models and tasks in Tab. 3 - 7. These numbers are far below values on other benchmarks, which strongly suggests the models are being queried with insufficient context and/or misleading prompts (and/or the evaluation harness has serious issues).

Minor Issues:

• The citation style appears to be incorrect.
• L12: "most frequent" -> "most frequently used"?
• L34: multiple typos
• The first three pages repeat the challenges from prior work (that motivate this paper) rather often (e.g. "coarse-grained tasks" is mentioned five times); consider reducing this redundancy.
• L250: how exactly does "fusing" work here?
• L287/288: "by by" -> "by one"
• Tab. 2: this table is entirely unclear. Do "Scene1-4" refer to scenarios? If so, those are not introduced as concepts until the next section, and conflict with the title, which says "Different projects" (of which there should be 10). What are these averages of?
• L509: ends abrumptly, missing text.

The 'product business data analysis' description lacks detail. How many data points were collected? From what type of people? Using which language models? Was this a standard product like GitHub co-pilot or something else? How was line completion integrated in the IDE? There are also several existing publications reporting on industrial or practical usage of line completion, using different categories. See, e.g., https://arxiv.org/abs/2402.16197

Also for Codedev-Bench, crucial details are missing, such as the actual 10 repositories used, the actual files, test cases, etc.

It is unclear to me what the dynamic and static call chain analysis actually does, and how it informs test case generation. The process described is reminiscant of mutation testing, where the code under test is mutated to explore which tests will fail. Since tests are used, it is crucial to report the quality (adequacy) of the tests in place, for example in terms of branch coverage.

The paper states:

we systematically extract test samples that match the distribution of real-world business scenarios

This makes sense, but the paper doesn't show any example or any detail that allows the reader to assess this claim, nor to reproduce this result. What were concrete 'real-world scenarios' for the 10 repositories chosen? How many of such scenarios were identified?

The paper has no discussion section offering an interpretation of the results, or an asssessment of the results' limitations.

The completion setting generated is still fairly artificial, as it is essentially fill in the blank. This assumes a correct context exists, in which just a few lines need to be added. In actual software development, the code will be incomplete, and most likely changes across different file locations or files are needed. This is not discussed.

Ultimately, the four findings are unsurprising, as they primarily indicate that code LLMs are better at coding tasks than general LLMs.

While Codedev-Agent seems impressive machinery, the resulting benchmark seems small, with just 296 code blocks. Why is this?

Lastly: were the repositories chosen part of the training data for the language models selected? Please discuss.

• Limited discussion/details on the evaluation of completed code at different levels of granularity: For in-line or single-line code completion, testing against all unit tests might be excessive. Since these completions are smaller and more context-specific, they often don’t impact the broader program behavior significantly.
• Benchmark adaptability: The focus on industrial-level repositories and high-quality data is valuable, yet the paper could discuss more on the adaptability of Codev-Bench to repositories with minimal unit testing or incomplete documentation, which are common in many open-source settings. Addressing these would improve the applicability of Codev-Bench in diverse dev environments.
• Agent v/s manual data collection: The authors argue an agent-based data collection process helps with making the benchmark more extensible to new repositories/programming languages. While that's indeed the case, a manual data collection is also often focused on code quality.

• The details regarding the construction of the benchmark with CodevAgent are somewhat unclear. For instance, while CodevAgent appears to utilize unit tests from the raw repositories for evaluation, the authors do not clearly explain how to ensure the quality of these test cases or the selection of benchmark repositories, given that raw repository tests might not be reliable. It is also unclear the selection rate, data quality, and time investment in the transition from raw repositories to well-generated test samples via CodeAgent.

• The analysis presented in this paper is somewhat high-level. While the authors provide examples of error cases, the detailed input is missing, and the presented examples rely heavily on specific context. A statistical analysis of the highlighted error cases would enhance the robustness of the findings.

• In section 3.1 Product Business Data Analysis, the authors present data analysis results to illustrate the needs of expert developers. However, there is no explicit description of how the source data was acquired, who was interviewed, or the scale of these interviews.

• There are instances of repeated sentences, such as:

  • "Automates repository crawling, constructs execution environments, extracts dynamic call..." appears in both the abstract (Line 023), introduction (Line 061) and methodology (Line 150).

  • "three common challenges: (1) ..." appears in both the 2.3 Benchmark For Code Completion (Line 136) and the Methodology (Line 150).

  To enhance clarity, it would be beneficial to either remove duplicate content or revise the phrasing.

• There are some presentation issues:

  • Line 509: We also show some. It seem to lack some additional words.

  • Figure 3 is not cited within the main body.

  • Table 2 is mentioned but lacks an explanation of the significance of the numbers presented.

  • Tables 6 and 7 are included in the appendix, while the analyses appear in the main body. It would be more coherent to move the tables and the corresponding analyses together.

  • The figures related to completion errors are not referenced in the relevant analysis sections of the appendix.