Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

We appreciate the reviewer's recognition of this dataset as a major contribution to Persian NLP, as well as the value of its metadata. Below, we provide clarifications and answers to the reviewer's questions in each section.

Reasons To Reject: Publish the dataset

For clarification, we commit to fully publish the dataset for free under the terms of the CC BY-ND license.

Questions To Authors

1. 1st and 2nd paragraphs: Remove promotional sentences

R4.1 We agree that the current wording may be interpreted as promotional. To address this, we have revised Section 3.1 to focus on factual details about Cultural Educational Center's operations relevant to the dataset.

2. 3rd paragraph: Annual update of the dataset

R4.2 We plan to update our dataset annually with new samples and insights. This ensures that our dataset remains up-to-date and relevant.

3. 4th and last paragraphs: Compute human performance

R4.3 We thank the reviewer. This point has been addressed in R2.1 (reviewer tpuE, answer to item 1).

4. 5th and 6th paragraphs: Answer extraction methods

R4.4 The input prompt did not restrict the model to single-letter answers, as it might hinder the reasoning process. Therefore, we used three methods to extract the desired answer from the model output detailed as follows:

The "Regex" method employs regex patterns to extract answer choices (step 1). If this fails, a pre-trained language model generates embedding vectors for each answer choice and the model output. Then, the option with the most similar vector to the output is selected (step 2). Therefore, incorporating a pre-trained language model is part of the "Regex" method. For example, the regex patterns can extract answer choice from "c) 3π" at step 1, but it will fail with "According to the given equation, the third option is correct," requiring step 2.

The "Single Token Probability" method uses the probability distribution for the initial token in the model's output to extract the probabilities associated with answer choice labels (e.g., 'a', 'b', 'c', 'd' in prompt-1 template). The label with the highest probability is selected, leading to the choice of its corresponding option.

The "Full Answer Probability" method utilizes the model's output initial token probability distribution to compute the average log of probabilities for each token of all answer choices. It selects the option with the highest average.

A more detailed explanation is incorporated in the revised version.

Thank you for the thorough reviews. We are pleased that you appreciated the comprehensive coverage and richness of metadata in PersianMMLU, as well as our experiments benchmarking various settings - key strengths of our work.

Regarding your questions:

1. Section 3.2 talks about the ways the dataset was augmented -- were these all done with human experts or were some extracted from original metadata from the learning ladder?

R3.1 We will clarify in Section 3.2 that all metadata attributes, including difficulty level, were extracted from Learning Ladder. Based on our investigation, Learning Ladder automatically classifies question difficulty using student performance data, marking questions answered incorrectly by more students as difficult. They also determine the "trap" attribute, identifying common wrong answers, by analyzing frequently selected incorrect options.

2. The dataset seems to be heavily based on data from the Learning Ladder website. Were there additional processing steps that the authors have implemented.

R3.2 You're correct that the data is primarily from Learning Ladder. It's worth noting that the Cultural Educational Center ensures the quality of the questions through student assessment, and a team of question designers and editors. However, we further refined the raw data by:

• Parsing HTML to extract relevant attributes
• Filtering out questions containing images or tables
• Removing explanatory answer questions to maintain multiple-choice format consistency
• Deduplicating questions

We will expand Section 3.1 with more details on these steps.

3. I did find that mT0XL outperforming a persian-english LLM to be interesting, especially given it's fairly strong performance on other MCQA datasets like Belebele (wondering if the authors dug deeper into this)?

R3.3 We agree this is an intriguing result worthy of further investigation. Although examining the specific factors was beyond our current scope, it's possible that the amount of Persian pretraining data for PersianMind (2 billion tokens) may not have been sufficient.

4. Were the authors able to do any fine tuning experiments?

R3.4 We did not fine-tune in this work, as our primary goal was to evaluate models' inherent zero-shot reasoning capabilities. However, fine-tuning, especially on other Persian MCQA datasets, is a logical next step. We will note this as future work.

Thank you again for your constructive feedback.

We appreciate the reviewer's recognition of the importance of language-specific benchmarks and their view on the value of metadata, including descriptive answers that foster community discussion.

We understand the concern about dataset privacy and publication. This issue has been already discussed with the data owner organization and necessary permission has been obtained for non-commercial use. The PersianMMLU dataset is distributed under the CC BY-ND license.

Regarding the reviewer's questions:

1. What does 'We chose the answer selected by humans if the combined total of the remaining three options was lower' in 4.2 means? Does this mean that in the regex method, in some cases it was just assumed that the model would go with the same answer as the human?

R2.1 In our dataset, we provide the percentage of respondents who chose each answer option for every question. We label a question as correctly answered by humans if the percentage selecting the correct answer is greater than the combined percentages of the other options; otherwise, we label it as incorrectly answered. Human accuracy is assessed by counting the number of correctly and incorrectly answered questions.

2. Why do the prompts provided in the appendix not use the same terminology for the grade levels as the main text?

R2.2 We thank the reviewer for their precise review. As noted in the figure captions, the English machine translation was provided only for readability purposes. In response to this comment, we have made the translations consistent and corrected them in the revised paper.

3. I'm not able to see the results corresponding to '4.3 Impact of translation quality' anywhere. Assessing the claims in that section without seeing qualitative results is not viable.

R2.3 Thanks for pointing this out. We will include the relevant results in the camera-ready version. Our results show that model performance on automatic translations improves when high-quality translations (denoted with an asterisk in the following table) are used instead of automatic translations. Specifically, model performance on automatic translations improves for samples in three major subjects—Physics, Math, and Chemistry—by 18%, 2%, and 12% respectively when we use high-quality translations. The translation results are as follows:

   En2Fa | Fa2En
   En  Fa| En  En* Fa
   Physics| 42  26|Physics| 58  76  50
   Biology| 76  74|Chem| 74  82  86  
   ML| 46  40|Math| 50  52  56

Thank you for the thorough reviews. We are grateful for highlighting the importance of having evaluation benchmarks for languages other than English, including Persian. The benchmark we introduced offers rich metadata and aims to address this need. We are also grateful that you found our current multilingual assessment and analysis useful. Regarding your questions:

1. 1st paragraph: bias and potential licensing issues

R1.1 To address concerns of bias, we note that the website serves as a platform for questions contributed by a diverse group of educators from various backgrounds and disciplines in education. This diversity helps to mitigate individual biases. Furthermore, the inclusion of questions from the nationally standardized Konkur exam, which are carefully selected from submissions by top educators, adds an additional layer of quality control. These properties minimize the presence of potential biases in PersianMMLU.

Moreover, the website features an authentication-required login interface that prevents access through common web crawling methods, thereby helping to prevent data contamination and ensuring the integrity of our dataset. Also, regarding the license, the PersianMMLU dataset will be distributed under the terms of the CC BY-ND license.

2. 2nd paragraph: contamination detection method

R1.2 We tested the hypothesis that the data from this collection was included in the training data of the models based on paper [1]. We conducted this evaluation using the Aya101 model, the best publicly available model, which had access to the token probabilities. As most contamination detection methods require access to the model's probabilities or weights, which are not available for private/closed models, we used the best performing public model for this evaluation. The hypothesis that the evaluation data was included in the training data was not proven, with a p-value of 0.3909. A detailed explanation of this experiment is included in the supplement of the revised paper.

3. 3rd paragraph: translation effect analysis

R1.3 Thank you for pointing this out. This issue has been addressed in our response to Reviewer 2, specifically in section R2.3.

4. 4th paragraph: example

R1.4 We have provided a full example in the following anonymous link

https://drive.google.com/file/d/1lgdgb09s4KNbGbfvZk486E6nuGSXEo2T/view?usp=sharing.

[1] https://openreview.net/forum?id=KS8mIvetg2