We thank the reviewer for the review. Here is our response.

W1) Please see our General response on Novelty. In addition to that, we provide 62,798 unit tests (Table 2), more than double the number in previous benchmarks. These tests comprehensively cover all necessary scenarios to verify the correctness of a solution. The correctness, defined by a set of unit tests, is initially confirmed by a human annotator/problem setter when the problem is created on Codeforces and later validated by contestants and the problem-solving community. Thus the problems and unit tests are of high quality. We have also ensured that our evaluation data includes the full unit tests, a step not taken in the CodeContest and APPS benchmarks. We believe this addresses a major shortfall in previous benchmarks and is rectified in ours. XCodeEval can be a good resource to train models for automatic test case generation, which is not within the scope of this (already dense) benchmark paper.

W2a) As of now, one of the main issues with the new genre of Large Language Models (LLMs) is that not all models are open-source. The way right now open-source and closed-source models “communicate” is via evaluation benchmarks. While open-source models are a significant part of the ecosystem, our proposed benchmark aims to ensure its applicability to any black-box (closed-source) model. The objective of the experiments presented in Table 5 is to demonstrate the impact of the finetuning data given along with the benchmark. The experiment in Table 5 reveals that a fine-tuned version of the 3B model can outperform a general-purpose 7B SFT model, but it cannot surpass a 13B general-purpose SFT model. Also this experiment highlights the trade-offs between model size and fine-tuning on specific corpora.

W2b) Temperature is a critical hyperparameter in the decoding strategy of language models. Analysis involving temperature demonstrates a significant performance gap when decoding at different temperature values. Given that generating programs necessitates maintaining consistent reasoning in long-range sequences, we illustrate that model developers should consider this parameter attentively during evaluation.

The reasoning spectrum provides a singular visualization across the entire evaluation benchmark, comparing unit tests across languages. For instance, a vertical green line on the y-axis indicates that all languages can pass that particular unit test. Additionally, the reasoning spectrum offers a more detailed comparison of unit tests between languages.

W2c) Please refer to Table 3 for the complete results of the APR and Code-Translation Task. Due to spacing issues, a detailed description of all the tasks is also provided in Appendix E.

W2d) The distribution of tags can be found in Appendix E.2. Additionally, the effect of our data selection strategy, as explained in Section 2, is detailed in Table 7.

W3) Let's assume that you have 25 million samples from 7,500 unique problems in 11 programming languages across 31 domains (where 'domains' refers to tags). This presents a unique combinatorial challenge in selecting data for training, development, and test distribution. In Section 2, we propose a novel data splitting and selection schema based on the geometric mean and graph-theoretic principles.

Q1) Yes, since we use Dinic's algorithm to implement circulation flow with upper and lower bounds, the direction of the edges negates some edge weights in the sum. Thank you for pointing this out. To account for the full simulation and the backward flows, we will revise the our text $f: E\longrightarrow\mathbb{Z}+$ to $f: E\longrightarrow\mathbb{Z}$ for better understanding of the context. However, this is just to re-iterate that $f: E\longrightarrow\mathbb{Z}+$ holds true for all the potential sets of flow to be selected as a solution. In contrast, $f: E\longrightarrow\mathbb{Z}-$ is necessary for the internal computations of Dinic's algorithm.

Q2) In our context, Top-k accuracy determines whether a PASSED solution exists within the top-k retrieval of the code for a query problem ID. We adopt the standard interpretation of retrieval accuracy in NLP and IR literature which denote the fraction of queries for which the top-k retrievals contain any of the corresponding solution/answer code. We refer to BEIR and DPR for more details.

[1] Muennighoff, Niklas, et al. "Scaling Data-Constrained Language Models." arXiv preprint arXiv:2305.16264 (2023)

We thank the reviewer for his comment. Here we respond to your comments and concerns.

On the Clarity of the paper (Dense contents): In order to demonstrate the remarkable aspects and distinctive features of xCodeEval and our contribution, it is essential to depict the bottlenecks of the existing code benchmarks, how xCodeEval addresses them, our novel data selection schema based on a “Circulation problem with lower and upper bound strategy”, developing a full NL-Code benchmarking peipeline including our multilingual code execution engine, ExecEval, presenating data properties and uniqueness, all featured tasks, a comprehensive analysis and ablation study on the reasoning capabilities of LLM models considering the possible data leakage which is really a lot of contents to present. Unfortunately, ICLR has a page limit. In Revision, we will improve the clarity of the paper and leverage the addition page. However we believe that in the current context, our paper is self-contained in terms of benchmark attributes, evaluation and analysis.

On the discussion of Novelty: Please follow the general response for the Novelty Description. In our General reply, we explain how xCodeEval differs from other datasets and its unique properties.

Fine-Tuning Dataset: We disagree with the claim that with LLs, the finetuning is no longer needed, rather finetuning models still prevail especially for adaptation to specific code-tasks or domains, enhancing performance, protecting data privacy and allowing benchmarking on code-specific tasks. It enables customization, addresses model analysis and so on. Nonetheless, the fine tuning dataset is specially intended to be used for finetuning smaller models considering that the evaluation benchmark may not be suited for zero-shot evaluation of such models (at least in today’s standard). In Figure 5, we show that fine-tuning a 3B model with training data can outperform a 7B general purpose SFT model. This shows that the training data can be used for enabling document level executability to smaller models. We also take extra precaution for train-dev-test overlap (Section 2.1) so that the integrity of the benchmark remains robust and training/finetuing data doesn’t interfere with benchmark evaluation. Also open sourcing fine tuning data helps the model evaluators to do data contamination analysis.

We thank the reviewer for his comment. Here we respond to your comments and concerns.

The benchmark draws exclusively from Codeforces ...

There are many fundamental issues we solve that exist in prior evaluation frameworks like CodeContest and APPS. Here are some description (also include in the general response and paper)

• How xCodeEval differs from APPS, CodeContests: For CodeContests (repository), 44 out of 165 test samples had no private unit tests. Additionally, there were inconclusive test cases. For instance, consider this problem in row 1 which contains 7 private test cases, while in the original codeforces competition, there are 34 test cases (accessible through the link here, you might need to log in and then click on Click to see test details), and some of them were incomplete (ended by ...) even in the codeforces API. We excluded problems with incomplete test cases from our development and test datasets. It's important to note that a problem is considered solved only when all the unit test cases pass successfully. We are grateful to our insightful reviewer for bringing up this aspect, and we plan to provide more context on these statistics in the final version of our paper. We mention that in “Evaluation and its granularity” section (Page 3). Similar problem occurs with APPS. For example: https://huggingface.co/datasets/codeparrot/apps (test split, row index 6) and codeforces source https://codeforces.com/contest/1217/problem/B, There are 365 test cases, while APPS included only 224. Note that these test cases, created by expert humans, as a whole ensure the correctness of a solution: a solution passed on all except one is considered as a wrong answer. xCodeEval ensures that this hypothesis preserves, while both APPS and CodeContest don’t ensure that.
• APPS and CodeContests offer only program synthesis tasks. xCodeEval proposes 5 different tasks from Classification, Generation and Retrieval tasks. From Table 1, APPS has 22,711 and 27,220 unit tests while xCodeEval covers 62,798 unit tests.
• APPS and CodeContests problems do not provide some metadata, while xCodeEval provides the following crucial metadata:
  • Timestamp: Problem release date which helps to analyze data contamination (check Fig 4-left). This is indeed very crucial see this interesting blog and paper on beating GPT-4!
  • Difficulty level: Human annotated difficulty level in the range 800 to 3500). APPS problems are divided into Competition, Interview and Introductory These are more like domains. For example, an Introductory problem can be harder than a Competition level problem. For example, in APPS competition test, row 5 has a difficulty of 1300 whereas in APPS introductory test row 3 has a difficulty of 2100 at codeforces. Both of these diffuculty levels are annotated by human experts.
  • Tags/category: xCodeEval comes from 31 different domains. Each of the domain problems can be identified during evaluation to understand about the domain biases of the code generation in LLM; see Figure 8 for details.

The reliance on a single platform like Codeforces means the benchmark might not capture ...

In our General Response, we explain xCodeEval’s novelty and how it differs from other datasets. In addition, Table 6 shows that most of the datasets typically cover a single domain. Specifically, the popular module-level benchmarks like HumanEval, MBPP, MBXP, HumanEval-X, MultiPL-E and BabelCode evaluate language specific features which usually exist in book exercises. The DS-100, Exe-DS, ARCADE are mainly python-dependent execution based Data Science problems. Other than that, most of the competitive programming language based evaluation benchmarks [1][2][3][4][5] use problems from ICPC style problems. Though xCodeEval primarily uses codeforces problems, it has the largest ICPC style problem collection. In addition to the regular contest, codeforces regularly host contests for graduate university, past ICPC contests, educational rounds (traditional problems) as well as legacy problems. We want to emphasize that though codeforces sounds like a single site, it hosts the largest variety of problems which covers more than 31 domains of problems.

We thank the reviewer for his comment. Here we respond to your comments and concerns.

Comparison with Stack V2 and the Objective of the paper:

We want to emphasize that the objective of Stack v2 and xCodeEval dataset is completely different thus they cannot be compared with each other. Stack v2 is a pre-training (usually via self-supervised) corpus while xCodeEval is an (NL-Code, Code-Code annotated) evaluation benchmark taken from 25M samples using a novel data selection strategy. It comes with the largest set of languages (see Table 1 & Table 6 ) compared with other benchmarks. It also provides an additional training corpus which can be used to do fine tuning. The training corpus is specially required to enable executability to the smaller models.

What would it take to expand the work to include other low resource languages including from a different domain?

Since xCodeEval is a unit test based benchmark, the program synthesis and code translation tasks of xCodeEval can be extended to any number languages as long as the evaluation framework supports that language. The APR task can be modified to Cross-Lingual Program-repair task and extended to any number of languages. For retrieval and classification tasks, we need data for low resource languages to extend the evaluation framework. However, these tasks can also be used for contrastive representation learning.

Since you used StarCoderBase as one of comparison points, could you provide value differentiation of the new benchmark over the github content in Stack used to train it?

Again, we want to highlight that Stack is a large corpus intended to perform large-scale pretraining, while xCodeEval is an evaluation benchmark. We also include a fine-tuning corpus to facilitate fine-tuning of smaller models that don’t have good document level executability. We use StarCoderBase to showcase this hypothesis. Please also follow the General response to get a good idea how xCodeEval differentiates from other benchmarks.

In Section 1, we summarize the bottlenecks of existing datasets and benchmarking systems to evaluate the code generation capabilities of recent generative LLMs and how xCodeEval addresses those limitations specifically by filing the evaluation gaps of (i) reasoning (ii) long-range document level multi-lingual seven code tasks (iii) following extreme sophisticated details and (iv) accounting for smallest errors that result in inaccuracy, and (v) a comprehensive analysis of model’s performances considering possible data leakage. With the release of ChatGPT and GPT-4, it is evident that most of the benchmarks are either already memorized by the LLMs from the training data or the LLMs are generalized enough so that the benchmark becomes too easy. See this interesting blog and paper why we need to rethink benchmarking and contamination for LLMs. We believe that a verifiable robust benchmark like xCodeEval can enable evaluation on the current genre of LLM as well as the next genre of more capable frontier models.