CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs

Q1: Evaluation reliability

Not clear whether this is a reliable evaluation set. The correlation with human judgement has not been measured.

We want to clarify that our dataset is both reliable and trustworthy, as it is constructed using rigorous filtering and validation procedures. Firstly, we source data from widely recognized sources for programming knowledge (e.g., W3School, CommonCrawl) and research works, which have already undergone measurement and validation. Secondly, for the real-world tasks, we validate both the correct answers and the distractors through execution, ensuring their correctness. Finally, we manually verified a small subset of the data to confirm the reliability of the construction process. While directly measuring the correlation with human judgment across 20,000 examples would provide additional insights, it is practically infeasible and prohibitively expensive for such a large-scale evaluation.

Q2: Data leakage issues

The authors motivate this work by highlighting the issues of potential data leakage with existing benchmarks (L036). However, it seems that CodeMMLU is susceptible to the same issue. Data sources like W3Schools, Geeks4Geeks, and Sanfoundry are likely already in the pretraining sets of existing models.

Additionally, the real-world tasks are based on existing benchmarks, which have leakage issues, as the authors claimed.

For clarification, we mitigated this issue by implementing rigorous filtering processes to ensure high-quality data. A key aspect of CodeMMLU’s design is the reformulation of raw data into the multiple-choice question (MCQ) format (as detailed in Section 3.2), which involves generating synthetic distractors as incorrect options. This transformation can reduce the likelihood that the questions in CodeMMLU have been encountered by LLMs during training, as LLMs are predominantly trained on raw code, bug reports, and similar data sources.

Table 1. ppl of benchmarks (higher is better)

Table 2. 5-gram of benchmarks (lower is better)

To further measure the degree of data leakage in benchmarks, we adopted the methodology from BenBench [1], utilizing perplexity and n-gram metrics. As shown in Table 1 and Table 2 (and in Appendix A.3 of the revision), CodeMMLU demonstrates lower levels of data leakage—evidenced by higher perplexity and lower n-gram overlap—compared to existing benchmarks like CodeScope and CodeApex. These results highlight the effectiveness of CodeMMLU’s pre-processing pipeline in mitigating data leakage.

Q3: MCQ biased and untrustworthy.

Next, Figure 7 and Table 4 suggest that the performance is very sensitive to the position of the correct option, which suggests that there are factors beyond code comprehension at play in MCQA. Therefore, it is not clear whether we can rely on this for evaluation code comprehension.

Thank you for raising this concern. We believe the factors mentioned stem from the inherent bias problems of LLMs, which have also been extensively discussed in the context of MCQ evaluation in the NLP domain [1]. We want to clarify that this issue does not undermine the validity of evaluating code comprehension using MCQs but rather highlights a limitation in the current capabilities of LLMs. Addressing such biases is a challenge that ultimately lies within the responsibility of model developers. We refer reviewers to appendix B.1 in revision and the discussion of MCQs bias and their evidence report in Q3 of reviewer xQZ6.

Dear Reviewer wsRv26,

We are delighted that our responses addressed your concerns. We thank the Reviewer for carefully assessing our work and adjusting the score accordingly after the discussion.

Best,

Authors

We extend our sincere gratitude for your valuable insights and thoughtful evaluation of our work.

Q1: Distractor quality concern

-: For most tasks, the authors use LLMs to generate distractors. The quality of these generated distractors should be discussed.

Thank you for your valuable suggestion. We have provided detailed explanations of how distractors are constructed for each subject in Sections 3.2, 3.3 and Appendix A1. Specifically, these distractors are generated using LLMs (Mistral 8x7B Instruct, GPT-3.5) and designed to appear plausible while being intentionally incorrect. To ensure their validity as incorrect answers, we verify them through an execution-based process, confirming that they are executable but do not lead to the correct solution (we kept those that met the threshold of lower than 50% test cases passed). This approach helps maintain the quality and challenge of the MCQs in our benchmark.

Q2: Concern about Code understanding task

-: The code completion and fill-in-the-blank tasks are more related to code generation instead of code understanding. Especially, the code completion task is based on the existing HumanEval dataset.

We agree that code completion and fill-in-the-blank tasks are more closely related to code generation. However, in our benchmark, we reformulate these tasks into an MCQ format, requiring models to comprehend the questions and options to select the correct answer. For the code completion task, we selected HumanEval due to its simplicity. Nevertheless, our MCQ formulation introduces new challenges that go beyond its generative format, as evidenced by the performance drop in Table 4 (main paper) and the misalignment of solved problems between the two formats shown in Figure 7. These results help us to highlight reasoning and comprehension weaknesses in LLMs, which are not effectively captured by generation-based benchmarks, even relatively simple ones like HumanEval.

Q3: Analysis CoT and other prompt settings

The decline in accuracy with COT prompts is interesting. Perhaps it's better to analyze the LLMs' answers with COT in detail.

Answer:

Thank you for your insightful suggestion regarding the detailed analysis of CoT (Chain-of-Thought) results. To address this concern, we have added a specific example of CoT usage and included a performance comparison against zero-shot prompting in Appendix B2. Additionally, we have updated the complete experimental results for all prompt settings studied in the paper in Appendix B3.

For a focused study, we selected the Object-Oriented Programming subset—a smaller subject in CodeMMLU’s knowledge test set consisting of 64 questions—to evaluate the effectiveness of the CoT technique. Our findings align with the conclusions of Sprague et al. [1], which suggest that while CoT introduces additional reasoning steps, these steps can either assist in overcoming challenges or inadvertently make the questions more complex due to added distractions. In our experiments, CoT did not exhibit a consistent pattern of overcoming new challenges, regardless of whether short or long prompts were employed, highlight reasoning might not always yield a clear advantage against knowledge-seeking tasks, such as MMLU and CodeMMLU.

Q4: Name confusion

In section 3.2, why is predicting execution output under the same category as defect prediction?

We acknowledge that the name "defect detection" may have been misleading and have updated it to "Execution Prediction" in Section 3.2 of our revision.

[1] Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., … & Durrett, G. (2024). To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183.

Dear Reviewer 6QYn,

Thank you for your time and valuable feedback. We hope our previous response has adequately addressed your concerns regarding the CodeMMLU sources and filtering quality. We eagerly await your feedback and are happy to clarify any additional questions.

Best,

Authors

Q7: Data leakage not discussed

It is unclear to what extent the LLMs were exposed to the underlying resources for this new data set (leakage). This risk is not discussed nor mitigated.

We agree that data leakage is a critical issue to address when constructing a benchmark and we need to provide more discussion. For clarification, we mitigated this issue by implementing rigorous filtering processes to ensure high-quality data. A key aspect of CodeMMLU’s design is the reformulation of raw data into the multiple-choice question (MCQ) format (as detailed in Section 3.2), which involves generating synthetic distractors as incorrect options. This transformation can reduce the likelihood that the questions in CodeMMLU have been encountered by LLMs during training, as LLMs are predominantly trained on raw code, bug reports, and similar data sources.

Table 1. ppl of benchmarks (higher is better)

Table 2. 5-gram of benchmarks (lower is better)

To further measure the degree of data leakage in benchmarks, we adopted the methodology from BenBench [1], utilizing perplexity and n-gram metrics. As shown in the table 1 and table 2 (and in Appendix A.2 of the revision), CodeMMLU demonstrates lower levels of data leakage—evidenced by higher perplexity and lower n-gram overlap—compared to existing benchmarks like CodeScope and CodeApex. These results highlight the effectiveness of CodeMMLU’s pre-processing pipeline in mitigating data leakage.

---

[1] Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2023, September). Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations.

Q4: MCQ biases treatment

The treatment of multiple-choice bias (models prefering to avoid option A) in the paper is unsatisfactory. The bias is stated, but not really studied / explained, nor are mitigation measures such as proposed by Zheng et al applied. [..] If the multiple choice format itself is a problem, what is the point of having a large multiple choice data set?

We appreciate the reviewer’s concern regarding the multiple-choice biases in our work. However, our primary objective is not to solve the bias issue but to bring attention to and investigate the challenges that MCQs in the coding domain present to LLMs. Hence, the responsibility for addressing bias should be done by the model builders.

While biases in MCQs have been observed in LLMs for natural language processing (NLP) [1], their manifestation in programming languages remains underexplored. Our results, as shown in Figure 7 and Table 4, highlight significant biases in LLMs when answering MCQs in the programming language domain, with the bias being even more pronounced than in NLP tasks. We disagree with the statement that the multiple-choice format itself is inherently problematic; rather, the issue lies in the capability of current LLMs. The use of a large-scale dataset like CodeMMLU allows us to demonstrate that these observations are not skewed by a small corpus, further underscoring the limitations of LLMs. Addressing these challenges will require advancements in model robustness and comprehension, which we hope our benchmark will help to inspire.

We refer the reviewers to appendix B1 and response Q3 of reviewer xQZ6 for further discussion on MCQ bias.

Q5: Ambiguous filtering processing

The filtering process is described, but the exact numbers involved (before/after filtering) are not provided. The filtering involves various manual steps -- applied to how many cases? Deep learners are used here, but no details are provided.

Thank you for your suggestion. We have expanded the description of the filtering process in Section 3.2 and Appendix A.2 to include detailed explanations of both the rule-based and deep learning approaches. Besides, we manually review a small subset of the data (100 instances per subject) to guarantee the effectiveness and quality of the automated filtering processes. For further clarification, please refer to our response to Q1. Additionally, we have updated the manuscript to include the result of filtering process (Figure 8 in Section A.1).

Q5: Improve writing and misleading information

The writing and presentation are generally good, yet is sloppy at places (the abstract speaks about "over 10,000" questions -- there are 19,900, which is more like 20,000, 3.2 speaks about "five distinct" tasks, but there are four, there is no table 8 (only a figure 8), ...). It is confusing that the text summarizing table 2 gives very different numbers from what is in the table ('over 3000' when in the table it appears to be closer to 5000, and 6000 when it is in fact 7000). I'm not sure why section A.3 is entitled "visualization" (nothing is visualized -- examples are given).

We sincerely appreciate your detailed observations and have addressed and corrected these errors in the revised version. We have updated the abstract to reflect the correct number of samples and addressed writing issues in Section 3.2. Additionally, we have relocated Section A.3 to B.1, included examples of CodeMMLU.

We genuinely value your thoughtful feedback and constructive questions.

Q1: Benchmark license and distributing concern

It is unclear how the dataset will be distributed. I would believe some of the data is copyright protected (e.g., W3Schools). This would mean you cannot redistribute their multiple choice questions.

Can you explain how you will distribute the dataset and under what licence the original material was made available, and which license you intend to use?

We thank the Reviewer for raising this critical concern. In short, we will release CodeMMLU under the MIT license.

We construct CodeMMLU by curating data from the web, most of which are from the Common Crawl, and thus can be used for academic purposes. For data crawled from websites such as W3schools and Geeks4geeks, we fully complied with their copyrights or sought their permission to use such data for this project. Thus, the MIT license satisfies all the sources’s copyrights.

Lastly, we wish to clarify a typo we made in the initial submission, where the category “programming language syntax” of the knowledge task was collected from CC but was annotated as “sanfoundry”. We double-checked the license of all data and fixed this typo. For a detailed breakdown of the licensing from each source, we refer the Reviewers to appendix A.3, revision version.

Q2: Filtering and processing quality concern

While filters and manual processing are applied, these are described only very briefly. As a result, the quality of the questions is unclear.

We acknowledge that the description of our filtering and manual processing methods could be more detailed. We have addressed this by expanding Section 3.3 and A.1 in the revision to provide a clearer explanation of our quality assurance process. (we refer reviewers to discussion Q6 of reviewer xQZ6)

Specifically, we employ a rule-based filtering approach to remove data with incomplete information, non-textual content, or irrelevant and redundant samples. Additionally, we utilize LLMs to assess and filter questions based on completeness, coherence, clarity, and relevance. To ensure the dataset remains challenging, we also use LLMs to evaluate and exclude overly simplistic questions based on difficulty. This combined approach ensures that while our dataset is extensive, it maintains a high standard of quality and relevance. Finally, we manually review of a small subset of the data (100 instances per subject) to guarantee the effectiveness and quality of the automated filtering processes.

Q3: Real-world concerning on real-world subset

The questions for the 'real world tasks', e.g., for defect detection or line completion, are very artificial. [..] Referring to the tasks as "real world performance" is misleading.

[..] 'code completion' tasks based on HumanEval [..] (which) suffers from many problems. There is a vast amount of literature on LLM assisted code-completion using data that is better than HumanEval.

The defect detection task appears to be about predicting the correct execution result -- which is a different task from defect detection. Again, there are lots of defect benchmarks around, with real bugs collected from actual systems (e.g., defects4j). It is not so clear what these mutliple choice questions add to that, especially with the weak distractros (like compile error, "internal error" (??))

We want to clarify that all samples in CodeMMLU are presented in a multiple-choice question (MCQ) format, which, while not explicitly present in practical software engineering tasks, provides a structured way to evaluate comprehension and decision-making. The task names, such as "real-world tasks," are suggestive and refer to the transformation of practical code scenarios (such as generation or debugging) into the MCQ format for evaluation purposes. Although these MCQs are abstractions, implicit decision-making scenarios akin to MCQs are part of a developer's daily work, such as choosing between implementation strategies or configuring environments.

Regarding the use of HumanEval, we would appreciate it if the reviewer could elaborate on the specific concerns with HumanEval and suggest datasets they consider superior. We agree that HumanEval is relatively simple; our MCQs present new challenges that extend beyond its generative format. These challenges highlight reasoning and understanding weaknesses in LLMs, which may not be captured effectively by generation-based benchmarks.

We acknowledge that the names "defect detection" and “real-world task” may have been misleading and have updated them to "Execution Prediction" and “Fundamental coding skill test" respectively, in Section 3.2 of our revised manuscript.

Dear Reviewer aVYU,

Thank you for your thoughtful comments and the time you have dedicated to our work. We hope our previous response has sufficiently addressed your concerns regarding the CodeMMLU license, data and filtering quality, and solutions for MCQ bias. We look forward to hearing your thoughts on our response and are happy to provide further clarification if needed.

Best,

Authors

First of all, we would like to express our gratitude for your constructive feedback. We address your concerns or questions as follows.

Q1. Novelty concern

[..] what issues CodeMMLU addresses that current benchmarks do not [..]

[..] there are many comprehensive and thorough benchmarks for code understanding and code generation, such as CodeXGlue, XLCoST, xCodeEval, CodeScope, LiveCodeBench, and BigCodeBench. Notably, CodeScope (ACL 2024) has constructed a code understanding benchmark that includes four tasks and multiple-choice questions.

We appreciate the reviewer’s detailed feedback and suggestions regarding the Related Work section.

CodeMMLU introduces a multiple-choice question (MCQ) benchmark that focuses on evaluating large language models (LLMs) on code understanding at scale, unlike prior benchmarks still rely on generation evaluation to assess code understanding (HumanEval, MBPP, BigCodeBench) and use match-based metrics such as BLEU, MRR, or ROUGE in tasks like code translation, code review, etc (CodeXGLUE, XLCoST, CodeScope).

We argue that the MCQ format of CodeMMLU is critical in assessing the model’s code understanding at scale for two reasons. First, it is more straightforward and efficient to evaluate the MCQ’s answers compared to code generation or match-based metrics. CodeMMLU offers a large-scale evaluation that overcomes the scalability limitations inherent in execution-based metrics.

To our knowledge, we are the first benchmark that attempts to bring multiple fundamental coding tasks, like code completion and code repair in the form of MCQs, to evaluate LLM. Thus, CodeMMLU focuses on both coding skills and evaluating programming knowledge understanding spread in diverse areas, making CodeMMLU the largest coding MCQ benchmark (with 20K questions, 52 topics, and 4 fundamental coding skill tests).

Second, the data curation process to build CodeMMLU alleviates data leakages via several filtering steps and the usage of distractors. Furthermore, by swapping the ground truth position, the MCQ format engages the models in complex reasoning rather than simply memorizing the training data.

We have revised sections 2, 3 in the main paper and appendices A.1 and B to highlight CodeMMLU’s novelty and discuss its contribution over existing benchmarks.

Q2. Data leakage issues

LiveCodeBench addresses the issue of data leakage through dynamic data set updates. [..] (CodeMMLU) the paper does not offer any solutions to the data leakage problem mentioned in line 36.

We agree with the reviewer that data leakage is an important problem to consider when building a benchmark. Thus, we have taken extra efforts to alleviate this issue when building CodeMMLU. First, we employ several filtering processes to ensure data to be high quality. Then, the key contribution made in CodeMMLU is the reformulation of the raw data into the MCQ format (Section 3.2 and 3.3), which involves introducing synthetic distractors as incorrect answers. As a result, the questions in CodeMMLU are unlikely to be observed by the LLMs during training since they are more commonly trained on raw code, bug reports, etc. To quantify the data leakage degree of each benchmark, we follow BenBench [1] to report the perplexity and n-gram metrics.

Table 1. ppl of benchmarks (higher is better)

Table 2. 5-gram of benchmarks (lower is better)

As shown in the following Table 1 and Table 2 (also Appendix A2 in the main paper), our CodeMMLU exhibits lower levels of data leakage (indicated by higher ppl and lower n-gram) than existing benchmarks such as CodeScope and CodeApex. This result demonstrates the effectiveness of CodeMMLU’s pre-processing pipeline in alleviating data leakage.

Dear Reviewer xQZ6,

Thank you for taking the time to provide your valuable comments. We hope our previous response has sufficiently addressed your concerns regarding the novelty of CodeMMLU, potential data leakage, and the reliability of MCQs against selection biases. We look forward to your feedback on our response and would be happy to clarify any additional questions.

Best,

Authors