INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

We thank the reviewer for their recognition of our work in addressing the prevalent challenge of resource disparity between English and non-English languages in LLM development and that the reviewer values our focus on assessing LLMs’ regional knowledge, an important but often overlooked aspect in the field. Finally, we appreciate that the reviewer did not find any major weaknesses in our work.

Additional experiments with English prompts: The reviewer suggests exploring the effects of translating user prompts into English and comparing this approach with our existing experiments, where we prompt models using the native language of each sample. Based on the reviewer’s suggestions, we conducted new experiments where the user prompt instructions were translated into English while keeping the text of the question and the choices in the native language. We present new results for the most performant model variants within each model family, evaluated using both English and native language prompts:

Table: Accuracy of the model for answering INCLUDE questions with user prompt instructions in the native language of each sample and with user prompt instructions in English.

We expanded the analysis presented in the paper to include these new results, which suggest that English prompts can provide modest benefits for most models (with aya-expanse-8b and 32b being the exceptions), but that performance is generally within 1-2% of the performance of native language prompts.

Missing references: As noted by the reviewer, we have included the referenced materials regarding generating evaluation resources for LLMs in low-resource settings in the updated version of the paper.

Changes in Table 1: We will reformat the results in Table 1 to enhance clarity and improve reader comprehension and remove the bolded comparison for Answer Acc. At the same time, the reviewer brings up an interesting conceptual question! MCQ evaluations are proxy tasks for assessing the understanding that these models must display in open-ended, real-world interactions. While Total Acc. entangles the evaluation of regional knowledge understanding with other behaviors, such as outputting answers in the correct format, we think Answer Acc. provides an important comparison point where the model is only evaluated on examples where regional knowledge understanding errors can be disentangled from formatting errors.

However, we also acknowledge that it does change the effective test set across models (as models will make formatting errors on different questions) and allows models to incidentally subvert the metric with incorrectly formatted answers, which may, in fact, indicate questions in which the model is less confident in a particular answer. We are happy to add this context to the paper as a footnote!

Experiments with Claude: We would like to have results for the Claude model variants, but we are currently rate-limited by the API resources provided by Anthropic, preventing us from obtaining results across all settings in our paper. Once these results are available, we will add these models to our updated paper and launch a public leaderboard for INCLUDE.

Limitations of the scope of existing benchmarks: The INCLUDE benchmark makes three significant contributions: (1) introduces original datasets from languages that are either not covered or only partially covered by existing benchmarks, (2) leverages publicly available, knowledge-intensive multiple-choice question (MCQ) benchmarks in various languages, (3) organizes both the existing and newly introduced data under a unified taxonomy of knowledge, differentiating between regional knowledge and region-agnostic knowledge. In this context, INCLUDE incorporates existing datasets, which account for 39.8% of the total collected data and 31.7% of the INCLUDE-lite benchmark.

To address the reviewer’s comment, we analyzed model performance on languages where both existing benchmark data and newly-collected data are available. We compared performance across these datasets and examined the correlation between them. Results were stratified by language and by model type. We visualized the performances using plots and calculated the $R^2$ scores to quantify the correlations. The updated paper includes these plots in the Appendix A9 in Figure 10, and the corresponding correlation scores are presented in the table below.

Overall, there is correlation between a model’s performance on a language-specific benchmark and its performance on the INCLUDE subset of that language, though with enough difference in most languages to justify the development of a new benchmark focused on regional knowledge. The analysis further reveals two key conclusions: (1) for a given language with multiple models having published performance data, a new model's performance range on a language-specific INCLUDE subset is generally predictable given a performance for that model on the same published benchmark in the same language (Language $R^2$ in table above). However, for a given model with published performance across different languages, its performance on INCLUDE cannot reliably be predicted when applied to a newly published language benchmark (Model $R^2$ in table above). This indicates that while INCLUDE may be less impactful for languages with existing published benchmarks, it is particularly valuable for assessing performance in languages with no prior resources, which is true for many of the languages in INCLUDE.

Answer extraction: As highlighted in the paper and noted by the reviewer, our approach follows the prompting strategy outlined in the MMLU benchmark by Hendrycks et al. Specifically, we employ a 5-shot prompting method, with the prompts translated into the language corresponding to the question.

We expect the model to follow the demonstration and output the number corresponding to the correct answer, similar to the behavior observed in English benchmarks. However, as discussed in Section 5.4, experimental results reveal that widely used prompting strategies in English benchmarks often fail to produce consistent outputs in multilingual settings. In some cases, the model's outputs vary depending on its capacity and proficiency in a particular language, potentially generating explanations, repeating the prompt, or providing definitions of answer choices—sometimes without outputting an answer at all. Below are examples that showcase different erroneous formats in generations (Outputs are translated from Armenian and Greek into English for presentation purposes):

To ensure a fair evaluation of the models and accurately measure their ability to answer questions regardless of output format, we decided to extract the answer directly from the generated responses. This was achieved by manually inspecting a sample of questions for each language to identify patterns in the models' outputs. Based on these patterns, we designed a heuristic pipeline to systematically extract the answers.

We also experimented with providing explicit instructions for the model to include the answer within a specific tag. However, this approach was unreliable, as some models in certain languages failed to follow the instructions, resulting in outputs without the specified tags.

Analysis of output errors on INCLUDE: As outlined in Section 5.3 of the paper, model performance—and consequently, the errors—are heavily influenced by the model's proficiency in the specific task and language. Following the reviewer's suggestion, we conducted a more detailed error analysis. We focused on six languages spanning high-resource (Chinese, Turkish), medium-resource (Bengali, Greek, Korean), and low-resource (Armenian) categories, selecting subject areas with the largest performance gaps compared to other subjects within the same language. For a breakdown of performance differences across subjects and languages, refer to Tables 6 and 7 in the Appendix. For the detailed analysis, we manually examined 150 examples, covering at least two subjects per language with 10 examples per subject, analyzing the questions and answers generated by GPT-4o. We observed four main types of errors: computational mistakes, factual mistakes, lack of regional knowledge, and model hallucinations. Each error type highlights distinct limitations in the model's capabilities, from arithmetic and factual knowledge to regional understanding and prompt adherence.

Our analysis revealed that the model's errors were distributed as follows: 38.6% were due to a lack of regional knowledge, 32% resulted from hallucinations, 26.7% were factual mistakes, and 2.7% were computational errors. We have added these results in Appendix A10 of the updated paper.

In general, considering all models and their performance, we notice the following:

• Output errors on the INCLUDE benchmark vary significantly based on model performance. High-performing models generally produce consistent output formats, whereas lower-performing models often generate inconsistent or hallucinated responses. In the most capable models, the primary error involves providing incorrect answers, often accompanied by flawed reasoning, as revealed in CoT experiments.
• When analyzing errors by language resource levels, distinct patterns emerge: in high- and medium-resource languages, models often produce computational or knowledge-related errors and occasionally repeat the prompt in the 5-shot setting instead of answering the question. In low-resource languages, models frequently struggle with well-formatted responses and sometimes hallucinate information based on the demonstrations. These findings highlight the challenges of maintaining consistency and reasoning quality across languages with varying resource levels.

We thank the reviewer for their thoughtful feedback and for recognizing the value of our dataset in reflecting authentic regional and cultural knowledge rather than relying on Western-centric datasets. We are also glad that the detailed discussion of results in Section 5 and the Appendix was found to be thorough and informative.

Existing Benchmarks Covering Regional Knowledge: As discussed in the motivation and related work sections, there is currently no multilingual benchmark that offers both high language coverage and incorporates regional knowledge. Most existing benchmarks typically focus extensively on a single language across various (including regional) dimensions, or they cover a certain number of languages with large amounts of agnostic knowledge (i.e., translated versions of MMLU). As suggested by the reviewer, below, we provide more details on the list of existing benchmarks that were mentioned in the paper, which feature original content (not machine-translated), highlighting their language and knowledge coverage. As can be seen, INCLUDE contains a larger number of regional questions compared to other baselines. Furthermore, among regional categories, we note that most of these benchmarks contain large proportions of region-implicit questions, which is generally the weakest category of regionality (e.g., a question from a business school exam is considered region-implicit as regional perspectives may be reflected in the exam questions even if the subject material should be the same among different regions).

For now, we have added these results to Appendix A9 of the updated paper and will consider how to integrate this into the main body (without going over the page limit).

References:

[1]: Koto, Fajri, et al. "Arabicmmlu: Assessing massive multitask language understanding in arabic." arXiv preprint arXiv:2402.12840 >(2024).

[2]: Li, Haonan, et al. "Cmmlu: Measuring massive multitask language understanding in chinese." arXiv preprint arXiv:2306.09212 (2023).

[3]: Ghahroodi, O., et al. "Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language? arXiv 2024." arXiv preprint arXiv:2404.06644.

[4]: Koto, Fajri, et al. "Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU." arXiv preprint arXiv:2310.04928 (2023).

[5]: Dao, Xuan-Quy, et al. "VNHSGE: VietNamese High School Graduation Examination Dataset for Large Language Models." arXiv preprint arXiv:2305.12199 (2023).

[6]: Yüksel, Arda, et al. "Turkishmmlu: Measuring massive multitask language understanding in turkish." arXiv preprint arXiv:2407.12402 (2024).

[7]: Hardalov, Momchil, et al. "EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering." arXiv preprint arXiv:2011.03080 (2020).

Regional and cultural dimensions of INCLUDE: The reviewer suggests that prompting the model with the specific region the question is from as part of the context would improve performance for cultural and region-implicit questions. In our original experiments, we do not explicitly mention the country or cultural background of the exam topics for two reasons: (1) language often serves as a cultural proxy, allowing us to test models on regional knowledge implicitly (e.g., answering a question in Greek using Greek cultural knowledge). (2) users may not explicitly specify cultural context or origin in their queries, so we expect models to demonstrate regional knowledge based solely on the language of the prompt.

While this approach motivates our work and experimental design, we incorporated the reviewer’s suggestion to conduct experiments that include the region and the language of each sample in the prompt instructions, asking the model to consider the cultural and linguistic nuances specific to that region. We present results for the best-performing large, medium, and small-scale models on specific language subsets across these two settings. We further stratify the results based on the type of knowledge: region agnostic or region related.

Table: Accuracy of the models for answering INCLUDE questions for two prompting settings stratified by the regional feature.

The results in the table show that providing explicit region and language information does not enhance model performance on regional questions. We have added these results to Appendix A8 of the updated paper. We encourage future INCLUDE users to experiment with different configurations and prompting strategies to further explore the extent of model performance.

MCQ format for automatic large-scale evaluation: We agree that open-answer performance can provide different insights into model capabilities, including more nuanced assessment for real-world applications compared to MCQ. However, as the reviewer also points out, MCQ enables faster, closed-form evaluation while open-answer evaluation would require using human evaluators (considerably slowing the process of evaluation), or LLM-as-a-judge pipelines (which are likely less reliable in multilingual settings compared to English). Our aim for this benchmark is to enable developers by providing an efficiently-evaluated, large-scale multilingual benchmark similar to widely used English benchmarks such as MMLU, MMMU, and MedQA (which are all MCQ as well).

Finally, on an implementation level, we also note that each MCQ item in INCLUDE can also serve as an open-ended generation question by prompting the model without providing answer choices, allowing our benchmark to provide an opportunity for future work in multilingual LLM-as-a-judge open-answer evaluation methods.

We thank the reviewer for their positive feedback and for highlighting the extensive coverage of domains and application scenarios in our dataset. We also appreciate the acknowledgment of the clarity of our writing.

Key takeaways around regional question types in INCLUDE: The reviewer suggested conducting more fine-grained analyses of relative performance across region-explicit, region-agnostic, region-implicit, and cultural questions. We think this analysis would be beneficial too, and internally, have had long discussions about how fine-grained we could make such evaluations. However, due to the diverse exam sources, levels, and topics in our questions, direct comparisons of the relative "difficulty" of these regional question types might end up being misleading. To the reviewer’s point, while Figure 6 showed that models generally perform worse on cultural questions compared to region-agnostic ones, this could also be due to confounding factors (this is why Figure 6 was included in the appendix rather than highlighted in the main paper). For example, a collection of PhD-level cultural questions could simply be more challenging for models than elementary-level region-agnostic questions.

In version 1 of INCLUDE, the distribution of difficulty levels across languages is imbalanced, complicating analytical comparisons across regional question types. Controlling for all of these factors this would lead to categories with limited examples when stratified by language, level, regionality, and topic. Consequently, we leave this to future work with an expanded data collection that enables rebalancing question types in INCLUDE to control for these factors. Instead, our current version prioritizes augmenting the evaluation of multilingual models with assessments of broad regional knowledge. Ultimately, the goal is for robust models to achieve high performance across all these regional dimensions.

Despite this imbalance, we summarize the strong insights that still made it into our work:

• Knowledge transfer is more effective between languages with shared scripts or typological similarities. However, when investigating models for which pre-training language mix is disclosed, we find they often perform unexpectedly well in untrained languages, potentially due to incidental data contamination during pre-training.
• Lower performance in certain languages often relates to questions requiring regional knowledge, but this may also reflect the model's subject-specific capabilities (or the difficulty of the question) rather than regional knowledge alone, as regional labels are subject-dependent.
• Models often struggle with non-English instructions, and instruction-tuning provides limited improvement, likely because most instruction-tuning data is predominantly in English.
• When models are prompted with non-English data, they often fail to adhere to the exact format provided in the demonstrations. Our findings highlight the difficulty of standardizing evaluation for tasks that can produce varying output patterns, a challenge further amplified in multilingual evaluations.

Based on these findings, we encourage LLM developers to evaluate model performance holistically by incorporating diverse geographic and cultural knowledge and increasing language diversity during instruction training, ensuring better local relevance for deployment contexts.

Additional experiments on monolingual models: The reviewer suggests that the performance of monolingual models for each language might provide an upper bound for expected performance of multilingual models. Based on data from open-LLM leaderboards for specific languages [1, 2, 3, 4], we observe that the top-performing models in each language are rarely monolingual models trained exclusively in that language, but rather multilingual models such as Qwen and Llama 3.1, so we do not believe that monolingual models would necessarily provide an upper bound on performance.

Nevertheless, following the reviewer’s suggestion, we evaluated seven open-source monolingual models on the relevant language-specific subsets of INCLUDE, (i.e., the languages these models were pre-trained on). We compare their performance with the results of the most performant large-, medium-, and small-scale models on the specific language subsets. The results of this evaluation are presented in the table below.

Table: Accuracy of the multilingual and monolingual models for answering INCLUDE questions for specific target languages.

The table reveals that most monolingual models underperform the state-of-the-art small multilingual model Qwen-2.5 (7B), with the exception of the German monolingual model, SauerkrautLM-v2-14B-DPO, which performs on par with Qwen in the German language. We have added these results to Appendix A7 of the updated paper.

References:

[1]: Chinese Leaderboard

[2]: Arabic Leaderboard

[3]: Japanese Leaderboard

[4]: Korean Leaderboard

Data contamination prevention: The reviewer asks whether some of the tested LLMs may have already been trained on some of the questions in INCLUDE. As described in Section 3.1, INCLUDE is made up of a few previously-published benchmarks incorporated into INCLUDE, but also newly-collected exam materials from sources contributed by our multilingual community of native speakers. A significant portion of the newly-collected data was derived from PDFs and textbooks, which are less likely to have been included in models trained primarily on web-based data.

Following the reviewer’s suggestion, we analyze the degree of contamination within the models using the mink%++ [1] method for training data detection in LLMs. This method determines whether an input next token forms a mode or has relatively high probability under the conditional categorical distribution. Using this scoring mechanism, one can predict if an input sequence is part of the model’s training data based on a decision threshold. This method achieves SOTA on the WikiMIA [2] benchmark for training data detection. We use the decision threshold that achieves the best performance on WikiMIA as the decision threshold for our analysis. Using this method, we computed the contamination rate for each language on four main-stream multilingual models: Aya-8B, XGLM-7B, LLaMA-3.1-8B, and Qwen-2.5-7B. We show the contamination rate results in the following table.

To further mitigate the risk of benchmark saturation as a result of data leakage when new models are trained, we have held back the complete dataset, comprising 197,243 entries. Instead, we will release these further questions and answers incrementally over the next year. We have also reserved a held-out dataset covering a wide range of the collected languages to be used for future experimental studies specifically aimed at analyzing data leakage over time.

References:

[1] Zhang, Jingyang, et al. "Min-k%++: Improved baseline for detecting pre-training data from large language models." arXiv preprint arXiv:2404.02936 (2024).

[2] Shi, Weijia, et al. "Detecting pretraining data from large language models." arXiv preprint arXiv:2310.16789 (2023).

We thank the reviewer for recognizing the value of INCLUDE as a benchmark encompassing regional knowledge across diverse languages and domains. We also appreciate the acknowledgment of our community-driven approach to selecting exams, which helps capture authentic linguistic and regional diversity, and are pleased that the reviewer found our analysis of multilingual LLM performance to have clear takeaways.

New Chain-of-Thought (CoT) experiments: We only ran CoT in our initial paper for the larger GPT4o and Aya models as prior research indicates that small-scale language models (under 8 billion parameters) generally struggle to perform chain-of-thought (CoT) reasoning effectively [1,2,3]. However, based on the reviewer’s recommendation, we tested this setting on our benchmark and conducted CoT experiments on smaller (<= 8B parameters) and medium-sized models (14B parameters), selecting the most performant variants from the model families used in our initial experiments. The table below reports the accuracy of these models in both 5-shot and zero-shot CoT settings. Additionally, to expand the scope of our CoT experiments, we included the Llama 3.1-Instruct model with 70 billion parameters in our experimental process and analysis. For completion, we also include the results for aya-expanse-32B and GPT-4o models presented in our original submission.

Table: Accuracy of the model for answering INCLUDE questions for 5-shot and zero-shot CoT settings.

Our results align with findings in existing literature, confirming a fairly large performance gap between 5-shot and zero-shot CoT settings for smaller models. We have added these results to the paper in Table 1.

References:

[1]: Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

[2]: Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." arXiv preprint arXiv:2410.05229 (2024).

[3]: Wang, Yu, et al. “Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation.” arXiv preprint arXiv:2409.03271 (2024). APA

Experiments with English prompts: The reviewer suggests prompting with English prompt instructions on non-English questions. In the submitted paper, we prompt models using the native language of each sample. Based on the reviewer’s suggestion, we conducted new experiments where the user prompt instructions were translated into English while keeping the text of the question and the choices in the native language. We present new results for the most performant model variants within each model family, evaluated using both English and native language prompts:

Table: Accuracy of the model for answering INCLUDE questions with user prompt instructions in the native language of each sample and with user prompt instructions in English.

We expanded the analysis presented in the paper to include these new results, which suggest that English prompts can provide modest benefits for most models (with aya-expanse-8b and 32b achieving more significant increases), but that performance is generally within 1-2% of the performance of native language prompts. We added these results to Appendix A6.