Scaling Laws for Multilingual Language Models

[Strong claims – page 1] Our intent is to convey that the proposed framework is designed to be applicable to a large number of languages, as long as they meet our minimal cross-group transfer and data sufficiency requirements. While we indeed test our approach on 23 languages, the framework itself is general enough to adapt to additional languages under similar conditions. We hope this explanation clarifies this distinction and better reflect the potential, rather than exhaustive, scalability of our approach.

[Strong claims – page 7] Our aim with this section is to provide context for common settings and challenges in multilingual scaling, and to clarify how our approach addresses these by highlighting differences in functional forms and applications. We include this comparison as a substitute for a traditional “related works” section, as the work we compare against is one of the few representatives in this field and shares similar use cases and problem formulations with related studies. We hold the prior work in high regard, as it provides an essential foundation for our contributions.

In addition, we use the term “simple and predictive” in the paper, indicating that the simplification not only does not sacrifice faithfulness, but actually enhances utility. This is demonstrated by the accuracy of scaling law fitting as well as the optimized sampling ratios. In line with Occam’s Razor, we find that the simpler explanation proves to be more effective in this context.

[Alternative grouping strategies] Please refer to the second point in our general response. In the first point of our general response, we also provide additional evidence about the validity of language-family-based grouping.

[Language diversity] Our language selection follows the practice in [1] for representativeness and easier evaluation. Furthermore, upon revisiting the composition of the CC-100 dataset, we find that the only other major language family not covered in our experiments is the Afro-Asiatic family (e.g., Arabic). Unfortunately, due to current limitations in available multilingual datasets, expanding our experiments to the diversity level the reviewer suggests is challenging. We hope future efforts will continue to enhance the multilingual dataset landscape, as more diverse language coverage would allow even broader validation of our proposed scaling law. Based on the consistent results observed across our chosen language families, we are optimistic that our findings will hold on a more diverse language corpus.

[1] Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. Lai et al. 2023.

[Tokenization] While we acknowledge that tokenizer vocabulary imbalance is a key issue for multilingual LMs in general, this is not a problem specific to our setting. We actually discuss this problem in our paper (line 423, page 8), which motivates our use of the normalized losses for evaluation. In general, tokenization for high-resource languages tends to align more with semantic units, whereas low-resource languages often experience less meaningful tokenization due to data imbalance during tokenizer training. This can indeed lead to higher LM losses for languages with larger tokenizer vocabulary. Nevertheless, improvement to multilingual tokenizers is not the focus of our work, so we will leave it for future works in related areas.

[Larger models] We acknowledge that the experiment scale is on the smaller side compared with SoTA LLMs, which usually starts at 7B parameters. However, our model sizes are still comparable to or even larger than those used in recent studies of scaling laws, such as [1] (40M to 1.5B parameters), [2] (20M to 1B parameters), and [4] (64M to 312M parameters).

When considering data scales of EuroLM (4T tokens), practical limitations arise due to model learning capacities. For instance, training an 85M model on 4T tokens would likely be an inefficient allocation of compute. Our experiments, therefore, focus on data and model size combinations that are computationally practical and reflective of the goal of scaling laws: to extrapolate performance predictions at larger scales by training on smaller models. In addition, using more tokens could push us to a data-constrained regime, as low-resource languages might be trained for more epochs to adjust to the larger token horizon, possibly leading to overfitting. This regime often requires specific treatment [3, 4], which is beyond the scope of our paper.

[1] Scaling Laws for Neural Machine Translation. Ghorbani et al. 2021.

[2] Scaling Laws for Multilingual Neural Machine Translation. Fernandes et al. 2023.

[3] Scaling Data-Constrained Language Models. Muennighoff et al. 2023.

[4] On the Pareto Front of Multilingual Neural Machine Translation. Chen et al. 2023.

Dear reviewer Czqd,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, the author-review discussion period will end tomorrow. Should you have any further inquiries or seek additional clarifications, please do not hesitate to let us know. If the questions have been resolved, we would greatly appreciate it if you could acknowledge our response and consider updating the score.

Thank you again for your time and effort!

Authors

Dear reviewer Czqd,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, tomorrow is the last day that the reviewer can post messages to authors. Please do not hesitate to let us know if you have further inquiries. If the questions have been resolved, we would greatly appreciate it if you could acknowledge our response and consider updating the score.

Thank you again for your time and effort!

Authors

[Q2. OOD performance] We believe separate OOD testing is not suitable for our scaling law, as our pretraining data mixture (CC-100 dataset) already encompasses an extensive and diverse corpus of web crawled content. Given this broad coverage, defining a clear boundary for what constitutes “out-of-distribution” data becomes challenging. In line with common practice in the scaling law literature [1,2,3,4], our evaluation is based on test losses from held-out, in-distribution data to conduct fitting and further analysis.

[1] Scaling Laws for Neural Language Models. Kaplan et al. 2020.

[2] Training Compute-Optimal Large Language Models. Hoffman et al. 2022.

[3] Scaling Laws for Neural Machine Translation. Ghorbani et al. 2021.

[4] Scaling Laws for Multilingual Neural Machine Translation. Fernandes et al. 2023.

[Q3. Minority language families] The Niger-Congo family has very limited data, and due to this data limitation, it does not meet our data sufficiency requirement (line 189). Therefore, the proposed scaling law may not be directly applicable in this case. In fact, in the scaling law literature, extremely low-resource language settings often require specific adjustments and modeling [1,2], which is beyond the scope of this paper and left for future works.

[1] Scaling Data-Constrained Language Models. Muennighoff et al. 2023.

[2] On the Pareto Front of Multilingual Neural Machine Translation. Chen et al. 2023.

[Q3. Minority language within families] To investigate whether higher-resource languages skew the results within each family, we examine individual losses within the Romance family by comparing Spanish (the highest-resource language, with 28.1% of Romance tokens) and Catalan (the lowest-resource language, with 7.5%). We measure their individual losses across different Romance sampling ratios. From the following table, both languages show a consistent decrease in loss as the Romance ratio increases, indicating that each benefits from a higher family sampling ratio.

We also fit a power-law relationship (as in Equation (2)) to the individual language losses, with the following results:

The high R-squared values confirm that each language’s loss follows the power-law relationship well. Notably, Catalan, as a lower-resource language, has a smaller $\mathcal{L}^\star$ due to its smaller tokenizer vocabulary, consistent with our findings in Table 2. Its larger decay rate $\gamma$ suggests that it benefits more from an increased family ratio, similar to the pattern we observe for other low-resource families (e.g., Indic). Overall, these results indicate that our approach benefits both high- and low-resource languages within each family, and our reported results accurately reflect the overall performance without sacrificing the gains for less-represented languages.

[W1. Downstream performance] Please refer to the third point in our general response.

[W2. Benefits of loss reduction] We want to clarify that we focus on the problem of multilingual pretraining, where the primary objective is to minimize test cross-entropy loss. The absolute magnitude of cross-entropy loss has limited interpretability, making it challenging to judge whether a specific reduction is "significant" solely based on its size. As observed in prior works [1], the scope for improvement in cross-entropy loss is naturally constrained, which highlights the importance of even small, consistent reductions.

The key strength of our approach is its consistency in achieving the lowest loss across diverse weighting scenarios and model sizes. This consistency across different settings demonstrates the effectiveness of the proposed data-mixing strategy and ensures systematic improvements in achieving the pretraining objective compared to alternative mixtures. Moreover, our approach enables predictive ability of losses across language families for larger compute budgets—whether by scaling parameters, data, or both—without the need for extensive training runs. This predictability, prior to this work, is only enjoyed in the monolingual setting. Crucially, our method introduces control over the loss as a function of the sampling ratios, an aspect not addressed in prior work.

Finally, as discussed in the third point of general response, using downstream tasks as an evaluation metric introduces additional sources of variability and task-specific noise. Compared with the test losses, downstream metrics are indirect and unreliable for testing the effectiveness of our data-mixing strategy for pretraining.

[1] On the Pareto Front of Multilingual Neural Machine Translation. Chen et al. 2023.

[W2. Results in Table 6] We apologize for the discrepancy in the individual values for the last two entries in Table 6; they were inputted incorrectly, but the total normalized loss is accurate as reported. Specifically, the normalized losses for Germanic and Sino-Tibetan should be 1.134 and 1.216, respectively. This can be verified by noting that the results in Table 6 are derived from Table 5 values divided by the mono-family losses. For example, the updated values maintain the expected ratios, as seen with Sino-Tibetan: 1.714/1.216 (Uniform) = 1.759/1.248 (By tokens) = 1.743/1.237 ($\alpha=0.5$) = 1.409, which matches the mono-family loss for Sino-Tibetan. The original value of 1.134 led to an incorrect ratio (1.714/1.134 = 1.511), confirming this as an input error. Other results can be verified in a similar manner. We have updated the manuscript to correct this discrepancy, and we confirm that the overall conclusions and benefits of the proposed sampling strategy remain valid and consistent across model sizes.

[W3. Language family hypothesis] Please refer to the first point in our general response, where we point out the distinctions between cross-family and cross-lingual transfer, and add two additional family combinations to further validate our hypothesis.

[W3. Alternative grouping strategies] Please refer to the second point in our general response.

[Q1. Comparison with previous works] To the best of our knowledge, the only prior work attempting to relate losses with model size and sampling ratios for multilingual model is [1], which focuses exclusively on neural machine translation (NMT) in bilingual settings. At the end of Section 3.4, we provide a detailed comparison with it. In short, our scaling law generalizes and addresses key limitations in [1]. We reiterate some key advancements compared with previous works here:

• Explicit modeling of sampling ratios: In terms of the functional form, we explicitly model the relationship with sampling ratios $p$ by the form $\frac{A_i}{N^\alpha_i}p_i^{-\gamma}$ instead of $\frac{\beta_{p,i}}{N^\alpha_i}$ in [1], where $\beta_{p,i}$ is dependent on $p$ in an undefined manner. This explicit formulation enables our scaling law to be predictive for unseen sampling ratios, addressing a key limitation in [1]. In contrast, fitting for $\beta_{p,i}$ in [1] requires surrogate measures and additional heuristics.
• Incorporation of dataset size: We introduce the dependency on dataset size $D$ to provide a holistic understanding of multilingual training dynamics.
• Broader applicability: Our work extends beyond bilingual NMT to general-purpose multilingual language modeling, accommodating multiple language families. This broader scope makes our framework more versatile and suitable for a wide range of multilingual tasks.

[1] Scaling laws for multilingual neural machine translation. Fernandes et al. 2023.

Thank you for the authors' response and hard work to perform more experiments to validate their arguments (notably for the proof of the hypothesis and Q3).

[W1 + W2] Thank you for your detailed response. I appreciate the focus on pretraining test losses and the comparison with prior scaling law studies [3,4,5]. I also acknowledge the challenges introduced by variability and task-specific noise, as highlighted in [6]. However, "general-purpose multilingual models" naturally raise questions about their downstream utility. As the authors note, there is no consensus on how cross-entropy loss translates to downstream task performance, so for the sake of interpretation and comparison, I think including even a small downstream evaluation would be somewhat nice, similar to practices in related work, such as scaling laws in machine translation [8]. While the authors mention in Q1 and their response to reviewer j6WP that comparing directly with [8] may not make sense, it would be nice if a minimal downstream task evaluation can be proposed as this will provide a better practical applicability of the proposed scaling law.

[W3] Thank you for providing more experiments and the table. I have looked over Appendix C, but the graph in Figure 6 seems to only present 2 languages in both axes. Could you please clarify how to interpret the graph considering it is supposed to represent 3 languages?

The discussion of significance in loss reductions appears inconsistent. For instance, while differences in sampling ratios are deemed 'insignificant', in other contexts, a reduction of 0.017 (in the result) is considered meaningful elsewhere. As the authors brought up as well, loss is very limited in interpretation, and it is unclear to me how significant such a difference in loss is. I think statistical analysis may help provide a more meaningful comparison.

References

[1] Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. Liu et al. 2022.

[2] Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation. Xu et al. 2024.

[3] Scaling Laws for Neural Language Models. Kaplan et al. 2020.

[4] Scaling Laws for Autoregressive Generative Modeling. Henighan et al. 2020.

[5] Scaling Laws for Fine-Grained Mixture of Experts. Krajewski et al. 2024.

[6] Are Emergent Abilities of Large Language Models a Mirage? Schaeffer et al. 2023.

[7] Scaling Laws for Predicting Downstream Performance in LLMs. Chen et al. 2024.

[8] Scaling laws for multilingual neural machine translation. Fernandes et al. 2023.

Dear reviewer ig2E,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, tomorrow is the last day that the reviewer can post messages to authors. Since all concerns have been addressed, please let us know if there are any remaining reservations that prevent raising the score to a level of acceptance. If so, we would be more than willing to address them further.

Thank you again for your time and effort!

Authors

[Missing baselines] To clarify, [1] models the scaling laws for each discrete value of $\beta_{p,i}$, which in turn cannot be used for predicting the optimal sampling value. In contrast, we demonstrate that the scalar constants obtained in their work actually follow a continuous power law as a function of $p$. This continuous formulation of $p$ allows us to get an optimal sampling distribution, which is one of the fundamental contributions of this work. In addition, the study in [1] is limited to bilingual settings, and this limitation means that Equation (7) cannot be applied to a true multilingual setting with more than 2 languages.

[1] Scaling laws for multilingual neural machine translation. Fernandes et al. 2023.

[More experiments needed] Please refer to the first point in our general response, where we add two additional family combinations to further validate our hypothesis.

[Overstatements] We agree with the reviewer’s suggestion, and we have updated the wording in the manuscript to better reflect our contribution.

[Slope in Figure 4] We report the slopes in the following tables. The values are close among different lines, validating our claims.

[Ratio difference between 85M and 1.2B] The difference in sampling ratios between the 85M and 1.2B models primarily reflects variations in the benefits that each language family receives from scaling up the model size. During optimization (Equation (8)), the values of $\mathcal{L_i}^\star$ differ between 85M and 1.2B models, leading to slight adjustments in the optimal sampling ratios. Empirically, we observe that scaling up the model size provides a greater reduction in loss for the Sino-Tibetan family compared to Romance, which explains why the Sino-Tibetan ratio increases slightly with the 1.2B model. However, this shift is minor and does not significantly impact final performance, as demonstrated by the minimal difference of 0.003 in those ratios.

[100 models] 100+ models refers to the number of experiments we ran spanning different sampling ratios, model sizes and dataset sizes to propose and empirically validate the scaling laws spanning all the three scaling dimensions.

Dear reviewer j6WP,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, the author-review discussion period will end tomorrow. Should you have any further inquiries or seek additional clarifications, please do not hesitate to let us know. If the questions have been resolved, we would greatly appreciate it if you could acknowledge our response and consider updating the score.

Thank you again for your time and effort!

Authors

Thank you for your responses. My concerns have been addressed, and reviews are updated correspondingly.

[Figure 1] We have updated the caption of Figure 1 to explicitly point out that $N$ refers to the number of parameters. In addition, the legend at the bottom of the x axis specifies the unit of the parameters (1e8).

[Figure 5] As detailed in the caption, Figure 5 shows the fitted law on 50B and 100B tokens. The main takeaway is that the scaling law across sampling ratios holds for different model sizes as well as different dataset sizes, as the empirical results (points) lie closely on the fitted curves (dashed curves).

[Downstream performance] Please refer to the third point in our general response.

[Sampling ratios] We define the sampling ratios as the proportion of a language family within the entire training mixture, so the ratios must sum up to 1. While it is true that a language family performs optimally when all training tokens are dedicated to it alone (i.e., a sampling ratio of 1.0 for that family and 0.0 for others), this scenario limits the model’s multilingual capabilities. Our objective is to study the effects of different data allocations within a fixed token count budget (50B, 100B, and 200B tokens), which are subsets of the full CommonCrawl data.

The strategy of using 0.2 for all families is referred to as uniform sampling in Table 3 through 6. Our optimization approach, which determines optimal sampling ratios, consistently outperforms this uniform baseline across the different token count budgets.

[Random grouping] We have demonstrated this result in the right figure of Figure 2. In short, we find that there exists significant cross-group transfer in the randomly grouped scenario because languages from the same family are split into separate groups. This demonstrates the effectiveness of family-based grouping. Please refer to Section 3.2 for more details.

Dear reviewer ojYU,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, the author-review discussion period will end tomorrow. Should you have any further inquiries or seek additional clarifications, please do not hesitate to let us know. If the questions have been resolved, we would greatly appreciate it if you could acknowledge our response and consider updating the score.

Thank you again for your time and effort!

Authors

Dear reviewer ojYU,

We sincerely appreciate the valuable insights you have provided in the review, and we have diligently worked to address your comments through our rebuttal. As a kind reminder, tomorrow is the last day that the reviewer can post messages to authors. Please do not hesitate to let us know if you have further inquiries. If the questions have been resolved, we would greatly appreciate it if you could acknowledge our response and consider updating the score.

Thank you again for your time and effort!

Authors

Thank you for engaging with the discussion.

We believe the reviewer may have misunderstood the purpose of Figure 2. The aim of this analysis is not to study the degree of similarity between families, but rather to demonstrate that cross-family transfer is minimal. This claim holds true regardless of similarities between families, as shown by the consistent results observed across four different family combinations (the other three in Figure 6). While it is reasonable to assume some degree of relatedness among individual languages in these families, our experiments show that this relatedness does not lead to noticeable cross-family transfer. The key intuition here is that each language family already has sufficient internal data, reducing its reliance on other families for performance improvements. In fact, this family-based analysis is a key contribution of our work, distinguishing it from traditional analysis based on individual languages.

Regarding the random grouping experiments, the inclusion of Indic language refers to dividing Indic languages across multiple groups (e.g., Hindi in Group 1 and Kannada in Group 2). In Figure 2 (right), we can see that when languages from the same family is distributed across two groups, significant cross-group interaction occurs. This highlights that random grouping fails to meet the criteria outlined for our hypothesis (lines 188–190), reinforcing the importance of family-based grouping in ensuring minimal cross-group transfer.

We hope this resolves the misunderstandings and clarifies the design and intent of our analysis.

[Downstream performance] Analogous to prior scaling law works [3,4,5], we focus on the scaling behaviors during the pretraining stage, where the objective is to minimize test cross-entropy loss. Consequently, our evaluation centers around test losses, which directly measure the effectiveness of the pretraining process in achieving this goal. In contrast, there is no common consensus on how test losses translate to downstream performance, and this problem alone is an active field of study [1,2,7]. It is highly likely that a better-pretrained model might perform better on certain downstream tasks but worse on others, making it challenging to draw definitive conclusions from downstream evaluations.

In addition, for general language modeling, it is a well-established practice to fit scaling laws based on test losses instead of downstream metrics [3,4,5]. A key reason is the discontinuity and limited resolution of the evaluation metrics (e.g., accuracy), which causes downstream performance to often deviate from smooth and predictable trends, leading to the so-called “emergence” phenomenon [6]. This unpredictability makes downstream evaluation less reliable for reflecting the effects of different pretraining mixtures.

While downstream performance is not the primary focus of our study, we believe our proposed scaling law provides foundational understanding necessary to guide improvements in multilingual pretraining. Developing an accurate law for predicting downstream performance from losses, especially with the added complexity of multilinguality, is an important and challenging problem. While it is beyond the scope of this work, it presents a promising direction for future research building upon the results presented in this paper.

[1] Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. Liu et al. 2022.

[2] Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation. Xu et al. 2024.

[3] Scaling Laws for Neural Language Models. Kaplan et al. 2020.

[4] Scaling Laws for Autoregressive Generative Modeling. Henighan et al. 2020.

[5] Scaling Laws for Fine-Grained Mixture of Experts. Krajewski et al. 2024.

[6] Are Emergent Abilities of Large Language Models a Mirage? Schaeffer et al. 2023.

[7] Scaling Laws for Predicting Downstream Performance in LLMs. Chen et al. 2024.