CMMLU: Measuring massive multitask language understanding in Chinese

Thank you for your insightful feedback. We appreciate the opportunity to clarify and expand upon certain aspects of our work.

Novelty and Special Contributions

While we acknowledge that the paper is inspired by the MMLU paper, CMMLU is distinct in its data and analysis. Our dataset was independently developed, not translated from existing English datasets. The analyses we conducted differ significantly from the MMLU paper and are specifically tailored to current LLMs. We believe our analysis offers unique insights and contribute meaningfully to LLM research.

All questions are formatted as multiple-choice with 4 choices. This may make it difficult to comprehensively test the performance of LLMs.

We acknowledge that the multiple-choice format of testing has its limitations. However, this format offers a standardized and accessible evaluation method, as evidenced by its prevalence in popular LLM evaluation frameworks and leaderboards, such as the one hosted by Hugging Face (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), of which around half is multiple-choice tasks. Additionally, multiple-choice questions can be converted to a perplexity comparison of the different choices, which is the dominant approach for pre-trained LLMs evaluation, given that this is how instruction fine-tuning is carried out. The decision to use a multiple-choice format was made after careful consideration.

Currently, there is no universally-recognized benchmark capable of comprehensively evaluating the performance of LLMs. Most existing leaderboards use a combination of multiple datasets to provide a holistic assessment of LLMs. In this context, CMMLU is not intended to provide a holistic evaluation of LLMs, but rather be one of several benchmarks in a comprehensive evaluation strategy.

The experimental methodology of most experiments is language-agnostic. It only simply compares general LLMs vs Chinese LLMs and non-Chinese-specific tasks vs Chinese-specific tasks. I think more experiments should be deeply combined with the Chinese cultural and Chinese linguistic characteristics.

Your suggestion to incorporate a more in-depth exploration of Chinese cultural and linguistic characteristics is well-received. To avoid misunderstanding, we want to first clarify that all data in CMMLU is in Mandarin Chinese. Our initial focus was to establish a broad comparison between multilingual and Chinese-specific LLMs using CMMLU. This approach was intended to set a baseline for future, more detailed studies. We agree that delving deeper into the nuances of the Chinese language and culture will further enrich our understanding of LLM capabilities, but this is beyond the scope of this work.

This work needs to analyze the correlation between the performance reported by CMMLU and the real performance measured in the representative downstream NLU tasks. Otherwise, it is difficult to determine whether CMMLU can reflect the NLU performance of a LLM.

We have included an analysis of the correlation between CMMLU performance and five prominent downstream English benchmarks in Appendix Section I. Our findings indicate that performance on CMMLU has a strong correlation with results on four of these benchmarks (with Pearson’s Correlation r > 0.94), which span areas such as mathematics, commonsense reasoning, and coding. The slight exception is the PIQA task (with Pearson’s Correlation still high at r = 0.88), where the task relevance is somewhat reduced due to most models achieving high scores (>80%) on this task.

What form will this dataset be released in the future?

The CMMLU dataset will be released under a 'Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License' to facilitate open-source research and ensure accessibility for non-commercial use.

Besides the Chinese-specific tasks and data source, is there any other Chinese-specific feature that has been considered in CMMLU?

No. Our focus is on the dataset tasks and sources, which we believe adequately address the knowledge-based evaluation needs of Chinese LLMs, consistent with MMLU for English and IndoMMLU for languages of Indonesia. We are open to exploring additional features in future iterations of CMMLU based on community feedback and evolving research needs.

Thank you for your detailed review of our paper. We appreciate your concerns and have addressed each of them below with the aim of clarifying our approach and methodology.

An LLM passing professional exams doesn't necessarily indicate its true capabilities, raising concerns about construct validity.

To comprehensively evaluate the capabilities of an LLM, evaluation needs to encompass aspects including knowledge, commonsense reasoning, fairness, and generative capabilities. This paper focuses on evaluating knowledge-related aspects in the Chinese language, as a component of a broader evaluation, consistent with the use of MMLU in English.

School exam questions have been extensively employed as a proxy for evaluating knowledge in LLMs. Examples include the use of the English MMLU dataset in the official release documentation/technical reports for GPT-4, LLaMA-2, Falcon, and various other LLMs, and the HuggingFace Open LLM leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), the most popular LLM leaderboard today.

We believe that developing such a benchmark for Mandarin Chinese is essential for more inclusive evaluation of LLMs beyond the English language.

The crisis of research replication based on language models was severe, and the evaluation methods had limitations.

We absolutely share your concerns regarding the importance of reproducibility in research, but do not agree that this is a limitation of the dataset. To ensure transparency, we have meticulously documented the experimental settings in Section 4, including the number of shots for few-shot learning, and the exact prompt used. As such, all experiment results in the paper can be reproduced. Additionally, a number of subsequent studies have successfully replicated our results based on the arXiv preprint version of this paper, and we've also made our code completely public. In adherence with the policy of anonymity, we refrain from specifically referencing these papers here as they cite the arXiv preprint, compromising the anonymity of this paper, but we would be happy to provide the links after the review period.

Additionally, the issue of LLM research replication is an issue with models and model training rather than evaluation datasets, which is the focus of this paper.

Assessing the political biases inherent in the language models presented in the benchmark was challenging and required naturalistic observation.

Our paper focuses on knowledge evaluation, not assessing the political biases of LLMs, which is an orthogonal dimension of LLM evaluation, as stated above.

One concern I had was that this Chinese test benchmark did not include evaluation criteria for Chinese machine translation.

Once again, this benchmark specifically focuses on knowledge evaluation, as a critical component of comprehensive LLM evaluation. This focus is consistent with earlier work such as English MMLU (https://arxiv.org/pdf/2009.03300.pdf) at ICLR 2021 and Indonesian MMLU (https://arxiv.org/pdf/2310.04928.pdf) at EMNLP 2023. Machine translation is an instance of a task which could be included in a test suite to separately evaluate the generative abilities of the LLM, as mentioned above. That is, we agree that machine translation is an important aspect of broader LLM evaluation, but it is orthogonal to knowledge-based evaluation.

Inherent unpredictability of LLMs and prompt sensitive

This is a more general issue with any evaluation of LLMs, and not specific to our benchmark. We have strived to mitigate this by employing simple and commonly-used prompts, to minimize variance in model outputs. This approach leads to fairer comparison of different models. Also of note is that we tested several (i.e., 3) widely-used prompts for multiple choice questions on more than 5 models. We find that the standard deviation in the average score is <1%.

Typically, the Chain-of-Thought method has proven successful on LLMs. However, this paper concluded that the Chain-of-Thought was not effective in enhancing model performance, which contradicted the feedback received from the practical use of LLMs with the Chain-of-Thought. A more detailed analysis and explanation were requested.

The effectiveness of Chain-of-Thought (COT) in LLMs primarily comes about in tasks requiring reasoning. For memory-intensive tasks (e.g., recalling historical facts), COT can not only be less effective but potentially impede performance.

In the CMMLU benchmark, the majority of tasks are knowledge-intensive rather than reasoning-based. This explains why COT does not enhance performance in our case. Furthermore, as discussed in Section 4.2 of our paper, issues like regex matching failures can further diminish effectiveness.

The authors adequately explained the issues raised and all my other concerns were answered in the comments and responses of the other reviewers. CMMLU has been used as a benchmark in many papers. I hope the authors will keep the dataset up to date in the future.

Soundness changed to 3. Rating raised to 6.

Thank you for your insightful and detailed review of our paper. We have tried to address your concerns by doing additional human annotation on answer agreement, and include 2 more sections relating to difficulty and emergent abilities, as detailed below.

human baseline

Thank you for your suggestion on incorporating a human baseline. Given the wide range of knowledge areas covered in this benchmark, including STEM and social science, establishing a comprehensive human baseline is challenging. The broad knowledge spectrum makes it difficult for any single individual to provide a meaningful baseline. Moreover, recruiting subject matter experts for each domain, given the extensive scope of the benchmark, is not feasible.

However, we appreciate the need to provide some form of human comparison. To address this, we propose using statistical data from exams as a proxy for human performance. In China, most school exams require a minimum passing score of 60%, drive licence exams require a score of 90%, etc., suggesting that a competent human test-taker would likely achieve at least that accuracy rate in each subject area. We can supplement this with additional statistics on average scores for various levels of tests to provide a more nuanced view of human performance, although this will be at an aggregate level from educational authorities

We believe this approach, while not perfect, offers a practical and informative way to include a human baseline in our paper. It would provide readers with a point of reference to better understand the performance of LLMs in relation to human capabilities.

We are open to further suggestions and would be grateful for your opinion on this proposed method.

Difficulty distribution of questions in each subset

We appreciate your insight into the importance of question difficulty distribution. We've analyzed this from two perspectives. Firstly, the CMMLU benchmark encompasses a diverse range of difficulty levels: 5 subjects at primary school level, 10 at middle/high school level, 23 at college level, and 29 at professional level, ensuring a comprehensive difficulty spectrum.

Secondly, to estimate the difficulty distribution within each subject, we evaluated the top 20 models from our main results table. Each question was treated as a data point, and we recorded the number of models that correctly answer each question. This approach allowed us to map out the difficulty distribution across subjects. We've added a section in the Appendix (Section C) to analyze this. Briefly, our findings show that most subjects exhibit a single peak in difficulty, indicating a smooth challenge gradient within those subjects. However, some areas, like machine learning and professional law, display bimodal difficulty distributions, suggesting a mix of easy and hard questions, with fewer intermediate ones. Additionally, the maximum for the distribution varies quite a bit, suggesting some subjects such as “arts” are generally easy/LLMs cover the domain knowledge well, while others such as “college mathematics” are much harder/LLMs do not have good coverage of the necessary domain knowledge.

Should we add answers that are not generally agreed upon worldwide to this dataset?

Our stance is that a language-specific benchmark should reflect the knowledge and perspectives relevant to its target audience. For contentious historical or political topics, aligning with regional consensus is in line with the benchmark's purpose.

We first analyzed the different subject areas, and concluded that such questions will only appear in education, history, politics related subjects..

Through careful selection, we identified 7 subjects which may contain such questions:: Chinese history, world history, high school politics, world religions, journalism, legal and moral basis, and Marxist theory. The total number of questions across these subjects is 1362. We then sampled 10%=136 questions to do human annotation, sourcing an expert with a Master's degree in International Relations, currently researching Chinese history, world history, and politics. Their role was to identify questions whose answers might not be universally accepted or agreed upon globally.

This annotation process revealed that 19 questions within our sample had answers potentially subject to regional or cultural differences in interpretation. Based on this finding, we estimate that approximately 190 questions in our dataset (roughly 2% in total across the 7 subjects) might have answers that are not universally agreed upon. Given this small proportion, we believe that the presence of such questions will not significantly impact the overall evaluation results of our study

We can add this analysis to the paper if you consider it will enhance the content.