EuroBERT: Scaling Multilingual Encoders for European Languages

Thank you for your positive review! We are glad that you found our work a valuable resource.

“The biggest model performs a bit worse than expected, as the authors also acknowledge and investigate in the appendix.”

Thank you for your suggestion! We agree it would be interesting to explore hyper-parameter scaling techniques, such as MuP, which could have improved the performance of our largest model. However, given the complexity of pre-training pipelines and our limited budget (which only allowed for a single run), we chose to follow established practices in the literature when choosing our hyper-parameters. We view this aspect as an important area for future work.

We would like to thank the reviewer again for their thoughtful feedback. Please let us know if you have any other questions that we can clarify.

Thank you for your thoughtful review. We appreciate that you value our efforts towards extensive evaluation, reproducibility and transparency.

“My major concern of this paper is the innovation seems limited.”

We respectfully disagree. Research on pre-training is naturally expensive (pre-training the EuroBERT family required 80 days with 100 GPUs), limiting the breadth of our experiments, but we believe our work contributes with several meaningful insights towards scaling encoder training, both on the training distribution (Figure 4) and hyper-parameter choices (Figure 5). For example, we show that educational data filters used in decoder models lead to degraded performance in encoders (see Figure 5 right). We also show that training on math and code data, a decision where we diverge from previous work, improves multilingual retrieval quality, but degrades multilingual classification (see Figure 4, second and third plots). Overall, our findings highlight several important aspects in training encoders at a large scale, and we believe they are valuable contributions to the research community.

“And the results seems a little bit empirical and lack in depth explanations”

Indeed, similar to other works in the field, several of our findings are empirical. We conducted ablation experiments and tried to provide explanations/hypotheses for some of them, especially when results diverge from expectations. In many cases, however, finding a full concrete explanation beyond speculation is not possible or would require deeper elaboration worthy of a paper by itself, and to this end, we release all training artifacts (checkpoints, codebase, etc). Overall, our main goal with this paper was to reignite the exploration of encoders, highlighting unexpected behaviours in light of recent advances in decoder models, and proposing effective solutions, which we believe will be valuable to the community.

“Why does instruction tuning degrade encoder performance, contrary to decoders?”

This is a very interesting point. Similar to the data quality results in our paper, we hypothesise there may be a domain mismatch, as most downstream tasks for encoders are non-conversational. A more promising strategy may be to reserve this data to a later fine-tuning stage, particularly for use cases that may benefit from conversational input, such as evaluating dialogs.

We would like to thank the reviewer again for their feedback. We hope our responses clarified any remaining concerns, and that you'll consider increasing your scores. If you have any remaining questions, we are happy to address them.

Thank you for your thorough review and valuable feedback. We are grateful that you found our work well written and experimentally thorough, and that you appreciate our efforts towards publicly releasing all artifacts.

“XLM-Roberta is ranked 1st 7 times while EuroBERT is ranked first 6 times (see Table 1) could prevent adoption”

We understand the concern, but while XLM-RoBERTa-XL ranks first slightly more often in Table 1, EuroBERT-2.1B offers complementary strengths that we believe make it valuable to the community. Notably, EuroBERT-2.1B is smaller (2.1B vs 3.5B parameters), and supports longer contexts, which enables other types of downstream applications. This is exemplified by comparing both models on the MLDR dataset, a long context retrieval dataset, where all EuroBERT models outperform the XLM-RoBERTa family at longer contexts (Figure 3, left). Additionally, in the interest of facilitating the adoption of our models by the community, we release the code for pre-training and fine-tuning the models, significantly lowering the barrier using the EuroBERT family.

“EuroBERT outperforms all other baselines on math/code but this is likely because related data was added to training.”

Yes, that is correct. Including math and code data during pre-training, a choice where we diverge from previous work, likely explains the improved performance in those domains. Nevertheless, this choice was not made to optimize performance on these domains, but instead to analyse whether there was transfer to other domains, an hypothesis that has been demonstrated for decoder LLMs [1], but not investigated in the context of encoder models. Therefore, we believe this decision is an advantage: it makes our model more versatile, and thus more useful to the community, and allows us to show that these domains are beneficial for multilingual retrieval but not for multilingual classification (see Figure 4).

[1] Aryabumi et al. To Code, or Not To Code? Exploring Impact of Code in Pre-training

“a comparison on decoder architecture would be informative”

In this work, we chose to align with prevailing approaches in training small encoder models, which traditionally leaned towards bi-directional architectures. In contrast, recent work has also shown some promising results in adapting larger decoder LLMs to become general purpose embedding models. However, these approaches typically start with models of much larger sizes than the EuroBERT family [2,3]. Further investigating how to apply these approaches using smaller model sizes would be an interesting experiment, but we consider it beyond the current scope of this paper.

[2] Wang et al. Improving Text Embeddings with Large Language Models.

[3] Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models.

“due to pre-processing constraints”

Our dataset preprocessing pipeline pre-packs documents into fixed-sized arrays, which makes implementing variable-length training challenging without significant modifications to the already in-use pre-training codebase. Given the significant risks of these changes, and the fact that our budget only allowed for a single full run, we chose to adopt a simpler alternative: splitting sentences after packing, which only involved changing the model’s attention mask.

“Could increasing capacity bridge the gap?”

We believe so. Our results indicate that increasing model size leads to significant gains, aligning with broader trends in LLMs. However, we also aimed to have an efficient model that could be more easily used by the community. This factor motivated our choice towards a smaller model trained on more data.

"whereas XLM-RoBERTa suffers notable degradation."

We agree that the degradation is more visible for the MLDR dataset, but we also observed some degradation on SeaHorse. For the quantiles with larger documents, the gap between similarly sized EuroBERT and XLM-RoBERTa models increases, favouring EuroBERT. In the camera ready, we will update this sentence to further clarify the level of degradation for the different tasks.

We would like to thank the reviewer again for their detailed feedback. We hope to have addressed all your concerns and questions, and that you'll consider raising your score. Please let us know if you have any remaining concerns, we are happy to clarify them.

Thank you for your review and suggestions. We are happy that you found that our paper provides several key insights, is clearly written and reproducible.

“EuroBERT only focuses on a select few (relatively high-resource) European languages (...) acceptance of this paper does a disservice to the multilingual and low-resource languages literature”

We respectfully disagree that our paper “does a disservice to the multilingual and low-resource languages literature”: we cover a total of 15 languages, including some widely spoken but relatively low resource languages, such as Vietnamese and Hindi, and at a smaller scale Polish and Turkish. We also contribute with several insights on training multilingual encoders (for example, with extensive experimentation on the data distribution) which we believe to be valuable for the research community. While we agree that including all 24 official EU languages would be very useful, in this first iteration we chose to prioritize a smaller set of languages for which sufficient high quality data is available. Given our limited computational budget, we preferred quality over quantity, supporting only a subset of European languages and prioritizing several non-European languages in the interest of making the model more useful for the community.

“In Tables 9 and 10, I would suggest rephrasing "Extra-European Languages" to "Non-European Languages" to avoid confusion and reduce Eurocentricity.”

Thank you for your suggestion. We agree that “Non-European Languages” is a clearer and more appropriate term, and we will make this change for the camera ready version.

Given the large hyperparameter sweep for the larger EuroBERT, did the authors investigate applying MuP?

This is an interesting suggestion! While MuP is a promising technique for transferring the hyper-parameters from the smaller models to the bigger ones, it is primarily designed for pre-training. However, our main hyperparameter search was done for fine-tuning. Nevertheless, we consider that applying MuP at this stage is an interesting research direction for future work.

We would like to thank the reviewer for their feedback and thoughtful suggestions. We will update the paper to use the term “Non-European Languages”. We also hope that our answers have addressed your concerns, but please let us know if you have any other questions that we can address during the rebuttal period, we are happy to answer them.