LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Novelty: The benchmark's construction primarily aims to create a newer and harder version of existing benchmarks. However, a key issue with LLM benchmarks for code is that Olympiad in Informatics (OI) programs are not typically representative of real-world software engineering and programming languages.

Contributions. Our work has two-fold contributions.

• First, we carefully curated diverse problems with necessary difficulty distribution and tests. We used these problems to construct evaluation scenarios that allow measuring programming agent capabilities in isolation.
• We rigorously analyze model performances on our evaluation set and provide empirical findings on contamination, overfitting in HumanEval, comparison across tasks, and various model comparisons. Notably, in contrast with prior work, we identify over-fitting from post-training datasets as opposed to previously studied pre-training corporas.

Comparison to software engineering benchmarks. We believe that competition and interview style problems provide a complementary and challenging evaluation set to software engineering benchmarks.

• Different but complementary research investigations. While primary research investigations on SWEBench have focused on designing optimal agent computer interfaces [1], research in competition programming problems is attempting to train models with such behaviours.

• Availability of High-Quality Problems APPS, CodeContests, and TACO provide good quality and easy-to-use training datasets for competition problems with execution environments and test cases. Achieving such environments is possible but considerably more challenging for software engineering setup. Such easy-to-use training environments are beneficial for researchers and have allowed the exploration of various reinforcement learning-based solutions.

Given such community interest, we believe LiveCodeBench provides a high-quality benchmark for evaluation competition programming capabilities of language models.

The tasks of code execution and test case output prediction appear to be quite similar.

Yes, we plot the correlation between the tasks in Figure 13. As the reviewer observes, we find that code generation is more correlated with self-repair and test output prediction is more correlated with code execution scenarios.

At the same time, we want to highlight the two scenarios that measure different skills. In particular, the natural language problem description does not provide algorithmic details for the solution. Thus, the model needs to jointly come up with a satisfying solution and “simulate” it while predicting tests.

Test case generation typically is designed as predicting the entire test case from a given function implementation [3,4],

Thank you for the question! The test output prediction scenario is designed to evaluate the capabilities of code LLMs to create tests to filter their own incorrect solutions. Various “agentic” solutions like CodeT and AlphaCodium have incorporated test generation and demonstrated its advantages. We design the test output prediction task to evaluate this capability in isolation.

1. Disentangling test inputs for test generation. As the reviewer mentioned, test generation comprises predicting both test inputs and corresponding outputs. Evaluating both aspects jointly is challenging and prior works have resorted to evaluating both coverage and test validity. Even then, since different test generation systems can generate independent inputs, the test validity scores may not be directly comparable. We believe that normalizing the test inputs allows for comparing test validity more rigorously – predicting test outputs on the same inputs.

2. Using more Inference time-compute to predict test outputs. Competition programming problems often involve challenging algorithms and math-heavy puzzle questions. Constructing test outputs for such problems is also non-trivial, and very close to MATH problems. Thus, as observed in MATH problems, we find that allowing the model to use more inference time using strategies like chain-of-thought, self-consistency considerably improves the test validity accuracies. For example, GPT-4O-Mini test output prediction accuracy is 61% which drops to 46% without chain-of-thought.

Using competitive programming problems for LLM evaluation has been well adopted. Additionally, the evaluation scenarios in this paper are also not new but borrowed or adapted from prior work

Our work has two-fold contributions.

• As highlighted by the reviewer, first we carefully curated diverse but balanced difficulty problems. We used these problems to construct evaluation scenarios that allow measuring programming agent capabilities in isolation.
• We rigorously analyze model performances on our evaluation set and provide empirical findings on contamination, overfitting in HumanEval, comparison across tasks, and various model comparisons. Notably, in contrast with prior work, we identify over-fitting from post-training datasets as opposed to previously studied pre-training corporas.

The relatively small number of problems (~40 LeetCode problems every two months) raises concern about the reliability of the results. I wonder if it’s possible to estimate the variance of the pass@1 performance and the statistical significance of the comparisons.

Thank you for raising the important concern regarding reliability of our evaluations. We agree that using a sufficiently large and representative sample is crucial for statistical validity. In our study:

• Extensive Dataset Collection: While Figure 1 illustrates performance over a two-month window using approximately 40 LeetCode problems to highlight contamination patterns, our overall evaluation is based on a much larger dataset. Specifically, since May 2023, we have collected a total of 713 problems across multiple platforms, with 387 problems gathered since the beginning of this year.

• Variance and Statistical Significance Estimation: In Appendix G, we perform a detailed bootstrapped analysis to estimate the variance of the pass@1 performance. Our findings indicate that with 338 problems, the error margins when comparing models are within 1-2%. This margin decreases further with the full benchmark, enhancing the statistical significance of our comparisons.

Finally, note that Figure 1 uses 40 LeetCode problems just to signify contamination. In general, LiveCodeBench evaluations average model performances on problems collected over multiple months from different platforms for reliable evaluations.

It seems that the drop in performance occurs only for the LeetCode problems and is smooth for other platforms like AtCoder. Do you have any explanation for this interesting phenomenon?

Thank you for highlighting this intriguing observation. While we do not have direct access to the models' training datasets, we can offer some plausible explanations based on available information:

• Relative Popularity and Data Availability of LeetCode: LeetCode is a widely used platform globally, particularly for technical interview preparation, with over 500,000 registered users. Consequently, there is an abundance of LeetCode problem solutions publicly available on platforms like GitHub. This widespread availability increases the likelihood that LeetCode problems and their solutions are included in the training data of LLMs, leading to potential overfitting or data contamination.

• Limited Exposure to AtCoder Problems: In contrast, AtCoder is less prevalent internationally and is primarily utilized by a niche community of competitive programmers. As a result, AtCoder problems are less likely to be included in publicly available datasets, reducing the chance of contamination and overfitting.

• Training Data Selection Practices: Previous research [1] indicates that early Code LLMs may have prioritized data from popular platforms like LeetCode during supervised fine-tuning. This selection bias could contribute to the observed performance discrepancies.

[1] Evaluating Large Language Models Trained on Code

While these contributions are highly relevant and practical for the Code LLM community, it's important to note that dynamic benchmarking is a technique that has been utilized in various prior works [1,2,3,4] as early as 2021 [1].

Thank you for bringing these references to our attention. We have updated the manuscript to appropriately contextualize prior work on dynamic benchmarking. Our work extends dynamic benchmarking to the Code LLM setting (emphasized highly relevant and practical by the reviewer). Note that beyond code generation, we also study (or introduce) other scenarios like repair, test generation, and code execution -- allowing measuring programming agent capabilities in isolation.

Additionally, the issue of data contamination has been systematically examined in earlier studies [5,6], limiting the novel insights that the paper can offer to the broader ICRL community.

We appreciate your observation regarding prior studies on data contamination in code LLMs [5,6]. Our work builds and extends findings by shifting the focus from contamination in pre-training corpora to overfitting issues arising from post-training datasets. Specifically, we demonstrate that fine-tuned models, such as DeepSeekCoder, while achieving state-of-the-art performance on benchmarks like HumanEval, show significantly reduced performance even on LiveCodeBench-Easy where most closed models perform similarly to HumanEval.

Indeed, prior works mentioned by the reviewer have studied the contamination in code LLMs.

However, these works primarily focused on detecting contamination from samples in pre-training corpora. We expand those findings and in contrast, emphasize over-fitting from post-training datasets. In particular, while various fine-tuned open models achieve SOTA performance on HumanEval (e.g. DeepSeekCoder) they performed considerably worse even on LiveCodeBench Easy problems. We further identified that this gap widens after fine-tuning – i.e. base models are well correlated on HumanEval and LiveCodeBench-Easy but start to overfit on HumanEval after post-training (see “Comparing Base Models” and “Role of Post Training” paragraphs in Section 5).

Beyond contamination, we also provide insights on model performance trends, contamination, and programming agent scenarios through a rigorous analysis of over 50 models in Section 5.

introduce new methods to alleviate more intricate contamination issues such as re-phrasing [6]. Have you considered addressing more intricate data contamination issues like re-phrasing within the scope of this work?

Thank you for the suggestion. Unfortunately, we operate under full black-box access to underlying models, i.e. we do not have access to training (pre-training or post-training) datasets, model weights, or even log-probs. Hence, we can only identify contamination by measuring model performances on different sets of problems (time-segmented evaluations studied in this work). Since rephrasing contamination detection would require access to the training corpora, it is not immediately clear how to apply it in our scenario. Please let us know if we misunderstood your point.

Unproven data representative: Besides contamination, we also need to know how many code problems we need and how representative they are, to conduct a comprehensive evaluation. However, they were not answered in this paper. Are there mechanisms to guarantee or prove these properties, in particular for new models?

• Large and Growing Dataset: LiveCodeBench currently hosts over 700 problems collected primarily from LeetCode and AtCoder platforms. Our dataset is continually expanding as we automate the collection of new problems.

• Statistical Significance: In Appendix G, we present a bootstrapping analysis to measure evaluation noise. We found that using a subset of 338 problems yields a low variance of 1-2% in model comparisons. Utilizing the full suite of 713 problems further reduces this variance, enhancing the reliability of our evaluations.

• Diverse and Real-World Problems: Our benchmark includes a wide range of problem types, such as algorithms, data structures, mathematical computations, and real-world scenarios. This diversity ensures that models are evaluated on various aspects of programming skills. Additionally, since these problems are sourced from popular platforms hosting such problems. Given humans regularly participate in such contests, these problems reflect how models would perform in such a 'real-world' setting.

Unevaluated workflow reliability: While the workflow of benchmark construction is completely automated, this workflow's reliability was not evaluated. For example, the accuracy of the HTML extractor.

We followed existing best practices while extracting problem statements from HTML pages. In particular, we extracted equations, normalized markdown, removed duplicated problems, and filtered problems with images. When constructing our extractors, we also used LLMs to provide feedback by using them to evaluate any mistakes in parsing however did not find it very reliable.

We manually scanned through 30 random problems in our collected benchmark and compared them with the rendered HTML webpage problems and did not find any errors in our collected problems.

Unproven test case completeness: Averaged 18 of the test case count basically is far fewer than test cases used inside the programming task websites; the completeness or sufficiency of these test cases was not analyzed in this paper.

Thank you for the question. First, note that while we only provide 18 tests, we collect large and complex tests, often with inputs sized larger than $1e5$ elements using our generator-based input generation approach. Thus even single instances of such inputs can evaluate multiple diverse program behaviors.

Next, we observed that the number of tests presented a tradeoff in terms of benchmark usability and evaluation precision. In particular, initially, LiveCodeBench provided up to 100 tests per problem and suffered from considerably longer evaluation times (over 10 minutes for computing pass@1 for 400 problems.

Based on user feedback, we studied pass@1 performance with different counts of the sampled number of tests and found 20 tests to hit a sweet spot in terms of evaluation reliability and providing fast evaluation signals. We have added this analysis to Appendix A.4 and Figure 9.

Multi-step track needed: This work's designs of four tracks were motivated by AlphaCodium. Including AlphaCodium, state-of-the-art competition-level code generation works nowadays broadly applied multi-step workflow. However, besides one-step revision, multi-step code generation was not evaluated in this work.

Thank you for the question. Indeed, multi-step repair is employed in current works. However, performing multi-turn repairs can be expensive, and introduces various design choices (e.g. which solutions to repair as studied in [1], type of revision feedback [2]). In this work, we therefore evaluate the revision capabilities of models using a baseline single-step repair setup. While a thorough analysis of various design choices is outside the scope of our work, our benchmark and evaluation setup can be easily extended to performing multi-turn evaluations.

[1] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff [2] What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

The contribution of this work to the community depends largely on your continued maintenance efforts. Do you have a long-term plan?

LiveCodeBench problem curation pipeline is completely automated (scraping, test generation) and can be easily run to collect new problems. Currently, the pipeline is run every 1-2 months, based on the sufficiency of the problems to curate new problems by the lead author.