Mexa: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

Thank you for your constructive review of our paper. We will refer to raised issues in the updated version of the paper, thanks!

Weaknesses:

1. Lack of variety in LLMs considered.

Thanks for the reviewer’s suggestion. Our paper targets the widely used scenarios where the LLM is decoder-only and English-centric. Mixtral would serve as a starting point based on the reviewer’s suggestion. For Mixtral, it would not be significantly different in how we apply MEXA, as it applies to the hidden states at each layer for any given input. However, we will make sure to include this in the paper.

2.1) Benchmarks used. Mexa is quite good at predicting performance on the three multilingual benchmarks considered (table 2). However, it's not clear to me why only three benchmarks are considered;

We include all the possible tasks in Table 4 to explain why we chose these tasks. As you can see, for non-generative tasks, only these tasks support a high number of languages, including low-resource ones. Benchmarks with only 10 languages, mostly high-resource, would not support our claims, as MEXA would achieve high results for all of them. Belebele is the benchmark with the highest number of languages, while ARC and MMLU are the most important tasks on the community leaderboards for comparing models.

2.2) Also, there is a bit of tuning on these datasets (weighted average vs. last token, max pooling vs. mean pooling, FLORES vs. bible), and there is no held-out benchmark to truly evaluate generalization.

All of these settings are ablations to determine which works best in each scenario. To reduce confusion, we introduced one setting as the optimal choice. We explain the differences between these design choices in lines 361–388.

3. Inability to benchmark closed-source models.

True, this paper provides a method of evaluation for open science, and only model weights are needed. Although, developers of closed-source models could use MEXA under the hood and report their multilingual results to provide insight of their model's multilingual capabilities.

Questions:

1. The idea of an "English-centric" LLM is referenced several times in the paper, but I do not see a concrete definition of "English-centric" anywhere in the paper.

Thanks for pointing that out! "English-centric" is not a new term used in this paper; it is also used in other papers (including https://arxiv.org/abs/2312.12683, https://arxiv.org/abs/2408.10811). Most of the models used in this paper did not disclose how much of their pre-training data is in English. We define English-centric decoder-only models as models where the latent language is English, which typically occurs when the majority of the data is in English. We will provide a clarified definition for this term in the paper.

2. L252 "Although there are multilingual models [...] our focus here is on LLMs which are state-of-the-art in English based tasks": Why is the focus only on evaluating SOTA LLMs? The intention of Mexa is to be a public benchmark and leaderboard, so it will be used for a variety of LLMs, right?

We tested on 9 different models. The English-centric models we chose span a range of performance levels, from weak to strong. For example, LLaMA 1 7B achieves a performance of 0.42 on English BeLEBELE, while LLaMA 3.1 reaches 0.95. Again, this selection of English-focused decoder models highlights those widely used in the community. We acknowledge that the text is ambiguous and will clarify this in the paper.

3. More generally, as a suggestion, I wonder if positioning this work as a benchmark was the best strategy.

We did not develop MEXA first and then search for a use case. Most of open science today focuses on English-centric decoder-only LLMs. However, their capabilities for many languages remain frustratingly undocumented. We developed a solution that provides a clearer vision and a rough estimate of the multilingual capabilities of these models.

Thank you for your very accurate review of our paper! All of the reviewer’s points are valid. We will revise the text in the introduction to reframe what we take as an assumption into a hypothesis strongly supported by the paper's findings. Additionally, we will clarify the lines mentioned that are somewhat ambiguous.

Thank you for your thoughtful review of our paper. The review is very thorough and completely understands the strengths of our paper (“problem is very well motivated”, “proposed method is straightforward”, “paper is well-structured”, “exhaustive experiments across a wide range of languages and models”); however, we did not expect this review score from this review.

We hope we have addressed the reviewer’s questions so they can reconsider and provide a better score. We will address the issues raised in the revised version.

Weaknesses:

1. The MEXA formulation relies on relative scoring and does not account for the magnitude of cosine similarity ... Adding an absolute component to the formulation, would be beneficial.

In all setups, with the same configurations and without exception, MEXA achieves better results than absolute cosine similarity. The idea suggested by the reviewer seems interesting, and we need to further investigate it. However, we are afraid that adding such a component would make the MEXA score not comparable between different models per language.

2. The authors report correlation scores but omit absolute scores. Including absolute scores, at least for select tasks, would provide a more comprehensive evaluation and further strengthen the paper.

For Llama 3.1 8B, for two tasks and two parallel datasets shown in Figure 1, we included the MEXA scores. In Appendix A.5, we also include MEXA scores for additional models on FLORES-200.

3. The authors’ evaluation is restricted to discriminative tasks, omitting open-ended text generation tasks. It is therefore unclear whether MEXA scores correlate with the LLM's multilingual text-generation capabilities.

The scope of this paper is limited to non-generative tasks. Generation is generally more challenging than understanding, and it is unsurprising that many languages may struggle to generate content in their language. While NLP has advanced toward generative capabilities, a significant portion of evaluation still focuses on non-generative tasks (e.g., Hugging Face Leaderboards) due to their convenience and standardized metrics. Assessing generated output remains challenging, even in English benchmarks. For example, model-based approaches (e.g., "LLM-as-a-judge") require an LLM fully competent in the target language—a capability that is both questionable and the focus of our evaluation.

4. FLORES (derived from Wikipedia) and BIBLE (potentially a component of the pre-training data) are used as parallel datasets; and many benchmarks may also be Wikipedia-based, however, it remains uncertain whether MEXA is generalizable in terms of demonstrating reliable correlations with other parallel datasets from unseen distributions. Further, the authors do not provide any method of determining the optimal choice of the parallel data to be utilized in the MEXA score computation.

For FLORES, only the English part is derived from Wikipedia, while the rest of the languages are human-translated. In the case of the Bible, some models tend to memorize the English portion. We demonstrate that in both scenarios, using the weighted average embedding approach, we can achieve generalizable scores without concerns about memorization. We believe that testing on more unseen distributions (we would gladly consider any unseen parallel dataset recommendations from the reviewer) could yield lower MEXA scores. However, this only illustrates how the model performs on unseen distributions in other languages and does not undermine the validity of the MEXA approach.

5. The paper’s ablations are somewhat conventional, with previous studies having compared similar design choices. While the plots offer some valuable insights, the authors only discuss these superficially and omit in-depth analysis.

We will reference the suggested papers when discussing romanization. Additionally, we will expand the discussion of the findings in the figures.

Thank you for your valuable feedback on our paper. We will incorporate your suggestions in the revised version.

Questions:

1. The authors use only 100 examples in their evaluation which is a very small number while they have more examples available.

Thanks for pointing this out! Initially, we used 1000 sentences from the FLORES dataset for Llama series evaluations and found only a small difference (±1%) in correlations when reduced to 100. In Appendix 4, we show that 100 samples are sufficient to ensure MEXA's robustness, with high pearson scores validating this. Using 100 samples also allows scaling to more languages, many of which lack 100+ parallel samples. We will include a discussion of the small difference observed with larger batches in the paper.

2. The authors mention several tasks in Appendix but then decided to use only three tasks.

We include all the possible tasks in Table 4 to explain why we chose these tasks. As you can see, for non-generative tasks, only these tasks support a high number of languages, including low-resource ones. Benchmarks with only 10 languages, mostly high-resource, would not support our claims, as MEXA would achieve high results for all of them. Belebele is the benchmark with the highest number of languages, while ARC and MMLU are the most important tasks on the community leaderboards for comparing models.

3. Presentation of results in tables make it very hard to digest them. Authors may select one best strategy like mean/max and present the results. Table 1 and 2 both are very hard to understand.

Thanks for the suggestion! As explained in L381, mean and max pooling don’t have an advantage over each other but capturing different levels of understanding, both of which are important. To simplify Table 2, we will present only the average results and will reformat the table, moving the remaining numbers to the appendix for clarity.

4. What is the motivation of keeping the proposed method limited to monolingual models?

The models discussed in this paper are certainly not monolingual. For example, if the pretraining data contains 0.1% of one non-English language and the total training data consists of 3 trillion tokens, this equates to 3 billion tokens in that language. However, the reason these models are English-centric is that MEXA is based on the premise that the model has one latent language, and performance in other languages can be predicted by their level of alignment with the model's latent language.

5. I did not understand which of the final combination of layers work the best for MEXA? Is it a combination of all layers or a specific layer? From Figure 2, it seems that the alignment scores are best for middle layers. Is it correct? What is the optimal layer for MEXA?

There is no single optimal layer; it can vary depending on the model and language. For some languages and models, the last layer or the middle layers are sometimes the best. We did not select additional hyperparameters that could bias our results. Instead, we used two general pooling methods to provide generalized solutions for new models and tasks.

6. A few arguments need empirical support or support from literature. - Line 50-51: for these models to be effective in other languages, it is important that the other languages align with the main language. The basis of this assumption is unclear and authors should support it empirically or from literature.

In an English-centric LLM, the main language drives better performance, reasoning, and knowledge, while most other languages suffer from insufficient data relative to the main language. It is logical to say, in this scenario, if there is no alignment between the main language and other languages, the model will not provide meaningful coverage for the other languages.

7. What is the exact prompt given to models and what is their impact?

For MEXA, no prompt templates are given to the models, only the parallel sentences exactly as they are. For the tasks, we use the template exactly as in the lm-evaluation-harness.

8. Figures and tables are extremely small. I was unable to read them on paper. I am not sure this small size is even allowed by the conference. Authors should move a few models to appendix and present fewer models.

We will be sure to do this for the model tables and ensure that the figures are larger. We will move the similar models in performance and represent only one of each kind in the main text.

** We appreciate the reviewer's feedback and most of these suggestions are simple fixes that we would certainly apply to final revision. Based on the strengths, (refer to the general response), we hope the reviewer can reconsider and provide a better score.

Thank you for your positive review of our paper! We appreciate your suggestions and will address them in the updated version. Thanks!

Weaknesses:

1. There is prior work on prediction of performance of multilingual NLP models ...

True, prior work has explored features such as linguistic similarity, vocabulary overlap, embedding similarity, and pretraining data size, primarily in encoder-only models. However, these features often require access to pretraining data, which is unavailable for open-weight LLMs. This paper focuses on MEXA, a feature derived from the pre-trained model, without relying on pretraining data details. The novelty lies in demonstrating that a single representational-based measure, MEXA, is sufficient for performance estimation in English-centric LLMs. Thank you for pointing this out. We will include the suggested papers and incorporate this discussion to better differentiate our work in the paper.

2. The work focusses only on classification tasks, and does not cover generation tasks. When the NLP landscape have moved towards generative capabilties, evaluating prediction of generation quality is also important. This has been mentioned as a limitation of the work in the paper.

Yes, the scope of this paper is limited to non-generative tasks. Generation is generally more challenging than understanding, and it is unsurprising that many languages may struggle to generate content in their language. While NLP has advanced toward generative capabilities, a major portion of evaluation still focuses on multiple-choice tasks (e.g., Hugging Face Leaderboards) due to their convenience and standardized metrics. Assessing generated output remains challenging, even in English benchmarks. For example, model-based approaches (e.g., "LLM-as-a-judge") require an LLM fully competent in the target language—a capability that is both questionable and the focus of our evaluation.

3. The comparison with absolute cosine similarity is very limited (Section 5.3) ...

In all of the setups, with the same configurations, without exception, MEXA achieves better results than absolute cosine similarity. We have included the comparison with the settings that were already explored by previous works in the main text. The remaining comparisons will be included in the appendix.

Questions:

1. Table 2 is complicated to read ...

Sure, we will move the detailed results to the appendix and only keep the different settings with average results in the main paper.

Dear Reviewers,

Thank you again for reviewing the MEXA paper; we truly appreciate your efforts. We have taken the time to address all of your questions. As of now, none of the reviewers have responded to indicate whether our comments resolve the issues you raised. We understand you may be busy on your time, but we would be grateful if you could take a moment to provide your feedback on the author responses—it would make a significant difference. Thank you for considering!

December 2 is the final day reviewers may post a message to the authors. Given this recommendation, it would be useful to articulate answers to the following questions in 1-2 sentences each when responding to the author rebuttal:

0. Does the response from the authors resolve the issues you raised?
1. Why not a lower score?
2. Why not a higher score?