Copilot Arena: A Platform for Code LLM Evaluation in the Wild

Thanks for your helpful comments. We address your comments below:

[Weakness 1: The evaluation metric of the model is relatively simple and can only reflect user preferences.]

• There is an extensive list of existing literature that follows this paradigm (see the first paragraph of the related work section). Despite its simplicity, it has proven to be an effective way to have humans-in-the-loop to evaluate models
• However, the data we collect enables other forms of evaluation. For example, by utilizing the snapshots from a user’s code trajectory, we can analyze the long-term impact of each code completion. This is a direction we are actively pursuing as future work.

[Weakness 2: Although the benchmark is multilingual, the uneven use of programming languages ​​by users may result in inaccurate benchmarks for less common languages.]

• We agree with your concern but note that we report model performance aggregating across languages, rather than performance on an individual language.
• Your comment prompted us to do further analysis into the distribution across languages. We found that even accounting for examples with over 50+ samples, we still have 23 programming languages. This is significant compared to previous static benchmarks (e.g., those mentioned in Table 1). Please see the following table for a more thorough distribution of our programming languages:
• We propose to add this table and modify the writing. For example, in our data analysis (Section 5.1), we can explicitly discuss how the data is not evenly distributed over all 103 languages, but there is a core set of languages for which there are a substantial number of votes. We believe these changes will help reduce any misinterpretations of our results on multi-lingual programming languages.

[Question 1: Can this method include evaluations of some open-source models? How do you think you should balance user experience with incorporating more models into the evaluation?]

• Yes, EvalX can include open-source models! In fact, we reported results in our submission on 4 open-source models from multiple organizations (e.g., Llama 70b [1], Llama 405b [2], Qwen32b Coder[3], Codestral [4]). Since EvalX is an ongoing data collection effort, we continue to add open-source models to our platform.
• However, you do raise a great point that we will clarify in our revision. We don’t anticipate being able to evaluate all models using EvalX because we do not want to impact user experience negatively. One way to operationalize this is to select models that perform well on existing benchmarks, indicating they will be usable in practical settings. This is what we did when selecting models for this work.

[1] https://huggingface.co/meta-llama/Llama-3.1-70B.
[2] https://huggingface.co/meta-llama/Llama-3.1-405B
[3] https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct
[4] https://huggingface.co/mistralai/Codestral-22B-v0.1

Thank you for the helpful suggestions. We address the comments below:

[Weakness 1: While the deliverables are well-suited for the community, the main methodology appears to offer limited contribution to this ML community. Despite these concerns, I am inclined to accept the paper.]

We thank the reviewer for advocating for accepting our work. We believe there are multiple reasons why the EvalX is a timely and important contribution to the ML community. Since the final version allows an additional page, we propose to add a Section 5.3 to explicitly summarize the insights and takeaways for researchers building new coding assistants:

• Different models excel in different settings. Claude 3.5 Sonnet performs better at frontend/backend tasks, while Gemini and Deepseek excel at longer contexts. In contrast, changing models for different programming languages seems unnecessary. More generally, a routing approach based on input code context is an interesting direction for future research.
• Models should be trained and evaluated on varying code structures. We observed a variety of tasks and code structures in our platform (e.g., FiM, docstrings, and inline comments). Future benchmarks and training schemes should explicitly account for variations in code structures.
• Models should be trained on human preference for code. Given the clear gap between our leaderboard and static benchmarks, models should be trained on human preferences (of which our dataset is one option). Recent approaches have considered this, but still have significant gaps (e.g., Qwen-2.5 Coder trains on LLM-as-a-Judge preferences as a proxy for human preferences). By releasing EvalX data, our work can help to address the need for human preference data in real-world coding contexts.

[Suggestion 1: I recommend rephrasing Section 2.3….clarify whether the offline datasets are used for tuning the models?]

• As discussed in L166 (right), we do not fine-tune models, but rather only use the dataset for evaluation and tuning our prompts (L195, left).
• However, we agree that Section 2.3 can be further clarified! To this end, we will also significantly expand on existing details in Appendix A and release all corresponding code for these experiments.
• Altogether, with changes to Section 2.3 writing, additional details in Appendix A, and full release of code, we believe this will reduce any confusion on our prompting methodology.

[Suggestion 2: explain FiM task relation to "smaller models seem to outperform in other static benchmarks compared to our leaderboard".]

• In Section 5.2, our analysis indicates that FiM may not be the main contributor to model performance. While direct comparisons between models with and without FiM training would be ideal, most providers don't disclose their training paradigms. Our ablation using the Deepseek API (Appendix E, Table 6) demonstrates that FiM as an input format doesn't significantly impact performance. We'll clarify this nuance in our revised text.

[Question 1: How do you determine whether a completion is a FiM task?]

• To answer this, we conducted further analysis on whether the suffix is related to the completion. We broadly categorized our suffixes as either 1) inline (in which case it is clearly related) or 2) on a new line. Suffixes on the next line could be in scope (i.e., in the same function or loop as the last line of the prefix) or out of scope.
• As shown in the table below, the vast majority (~80%) of our suffixes are related to the completion. We will include this analysis in the Appendix.

[Question 2: How do you categorize the domain of the completion, such as frontend or backend?]

We follow a two-step process detailed in the Appendix (starting from L880), which we summarize here:

• First, we ask a model (e.g., GPT-4o-mini) to summarize all code contexts into short one-sentence descriptions.
• Next, we prompt a model (e.g., GPT-4o) to cluster all one-sentence descriptions.
• Finally, we provide the full code context and ask the model to categorize the context given the aforementioned clusters. We additionally note that two authors of the work verified that the categorizations were sensical before scaling.

[Question 3: Is it particularly challenging to further scale the collected data ... for a specific language?]

• Since we are collecting data in the wild, we cannot directly control what languages we collect. However, we strive to grow the user base of EvalX to reach broader audiences, which will lead to more data in additional programming and natural languages.

Thank you for the helpful suggestions. We address your comments below:

$Claim \+ Evidence 1: The position bias in the interface design is concerning. The preference data may reflect convenience rather than quality assessment, which may lower the data quality.$ *

• Since we randomize the ordering of the two generated responses (L149), the position bias affects all models equally. This means that the overall quality of the assessment is not impacted. However, although the quality of the assessment is not impacted, the efficiency of our platform is impacted. In short, since there’s a significant position bias, we need more votes to shrink the confidence intervals enough to draw reliable conclusions. Recall, we use logistic regression to estimate Bradley-Terry coefficients and bootstrap the samples to build confidence intervals (which allow us to be able to tell whether models are statistically significantly different). Overall, despite the efficiency reduction from positional bias, EvalX collects a sufficient number of votes to yield tight confidence intervals on model comparisons.

$Claim \+ Evidence 2: While the authors claim supporting 103 programming languages, the actual distribution is skewed.$ *

• We agree with your concern but note that we report model performance aggregating across languages, rather than performance on an individual language.
• Your comment prompted us to do further analysis of the distribution across languages. We found that even accounting for examples with over 50+ votes, we still have 23 programming languages. This is significant compared to previous static benchmarks (e.g., those mentioned in Table 1). Please see the following table for a more thorough distribution of our programming languages:
• We propose to add this table and modify the writing. For example, in our data analysis (Section 5.1), we can explicitly discuss how the data is not evenly distributed over all 103 languages, but there is a core set of languages for which there are a substantial number of votes. We believe these changes will help reduce any misinterpretations of our results on multi-lingual programming and natural languages.

$Method 1: Is there any way to quantify the potential impact on the reliability of specific model pair comparisons?$ *

• In the analysis conducted in the original submission, we took a few steps to mitigate this impact.
  • First, when computing Bradley-Terry coefficients, we use an L2 regularization term to prevent overfitting and bias towards models that have received more votes.
  • Second, we conduct a statistical bootstrap of the leaderboard so we can obtain confidence intervals around the estimated coefficients, which improves the reliability of results.
  • Finally, we ensure that we have reasonable coverage across all model pairs (e.g., >150 votes per pair), so votes are not too sparse across some pairs.
• Inspired by the author's suggestion, we did additional simulations using our data. We upsampled model pairs that had fewer comparisons and downsampled model pairs that had more comparisons so that the difference between the most and least voted pair is within 10%. We find that the rankings remain identical to those we report in the original submission. This provides additional confidence that we collected sufficient votes across all pairs.
• Overall, we appreciate this suggestion and will highlight this more prominently as a point of consideration for future evaluation platforms.

Finally, thank you for the helpful pointers to additional evaluation benchmarks! We agree it would be challenging to include in our comparisons, but we will definitely mention them in the related work and opportunities for extending EvalX.

Thank you for the detailed review and appreciation of our work. We aim to address your comments below:

$Claim \+ Evidence 1: focus on code completions$ *

• We agree that the paper already provides value to the research community and will be more precise about the scope of our work early on (e.g., we will revise L37 to state “... coding assistants, specifically focusing on their ability to generate code completions”).

• While there are many ways to interact with AI programming assistants, code completions are one of the most frequent use cases, as found by https://dl.acm.org/doi/10.1145/3597503.3608128.

• Further, EvalX naturally leads to many interesting directions for research for the broader AI and software engineering community, which we discuss in Section 7. For example, EvalX can be extended to include more interaction modes beyond code completion. Since our submission, we have already added a prompt-based editing feature. The reviewer’s suggestion of long-term impact is another direction that can be studied using data collected from EvalX.

$Claim \+ Evidence 2: preferences can be biased, compared to static benchmarks which can be nuanced$ *

• We agree, and believe our work is complementary to existing approaches and does not seek to replace them—both are necessary for a holistic view of LLM code evaluation. We discuss this in Related Work (L417), but will update the Introduction to be clearer.

• We also speculate that building evaluations in the IDE may mitigate the reviewer’s concern for biases towards readability. Since our platform is embedded in a user’s real development environment, users are likely to prefer “useful” generations (readability being only one aspect of usefulness) as they would when writing code with Github Copilot.

$Experimental design 1: Does the model distribution (skewed towards latency) impact the quality of data? Impact of$\tau$?$ *

• In the analysis conducted in the original submission, we took a few steps to mitigate this impact.

  • First, when computing Bradley-Terry coefficients, we use an L2 regularization term to prevent overfitting and bias towards models that have received more votes.
  • Second, we conduct a statistical bootstrap of the leaderboard so we can obtain confidence intervals around the estimated coefficients, improving reliability of our results.
  • Finally, we ensure that we have reasonable coverage across all model pairs (e.g., $>150$ votes per pair), so votes are not too sparse across some pairs.

• Inspired by the author's suggestion, we did additional simulations using our data. We upsampled and downsampled model pairs that had fewer and more comparisons respectively so that the difference between the most and least voted pair is within 10%. We find that the rankings remain identical to those we report in the original submission. This provides additional confidence that we collected sufficient votes across all pairs.

• As $\tau$ increases, our coverage becomes more evenly distributed (uniform at $\tau \rightarrow \infty$).

$Suggestion 1: More detail about other benchmarks$ *

While we briefly cover metrics for each benchmark in Table 1, we will additionally provide a short summary of each benchmark in Section 4.2.

$Suggestion 2: “insights/takeaways for model training or prompting$ *

Since the final version allows an additional page, we will add a new Section 5.3, which will explicitly summarize the insights and takeaways for researchers building new coding assistants. Due to character limits in the response, please see our first response to Reviewer 3 (1UGk) for additional details. Additionally, if the reviewer has any suggestions, we would be more than happy to incorporate them!

Questions (not repeated due to space):

• We believe there are three main motivations: 1) free generations, 2) overall interest in participating in ML research, and 3) GitHub Copilot or alternatives may be unavailable in some countries.
• Models are randomly selected (as discussed in L149).
• Fidelity refers to the quality of the response, largely in terms of formatting. We will clarify the caption.
• We observe this in our own data! Figure 6 shows that 65% of votes are on FiM tasks.
• l is the maximum latency between models $i$ and $j$ at one point. $F\_{max}(l; i,j)$ is the CDF of these latencies. F\_{max}(l;i,j) \= P(max(L\_i,L\_j) ≤ l) where $L\_i$ and $L\_j$ are random variables for the latency of $i$ and $j$. We agree that $l\_{max}$ is a better notation and will fix this.
• The total cost is approximately $5k USD.
• Correct, we always log model comparisons. Users are required to accept this before using the platform.
• Each element of the matrix is the win-rate between two models over all their battles. The win-rate would be $\in$ 0, 1 $$. We will clarify this in the paper by saying $W ∈ ℝ^{(M×M)}$ with $W\_{ij} ∈$ 0,1 $\forall i,j$