Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining

Thank you for your detailed and positive review.

Q4.1: Can more experimental details be moved from the appendix?

Thank you for the suggestion. We will revise the camera-ready version to move key experimental details from the appendix into the main text, particularly information on the number of models trained, token budget intervals, and training configurations. We agree that this will improve clarity and self-containment.

Q4.2: Is it appropriate to claim “state-of-the-art” performance?

Thank you for pointing this out. We agree that the term may be misleading given the limited training budget and data quality. We will revise or remove the phrase “state-of-the-art” from the abstract and main text. Instead, we will emphasize that CLIMB achieves competitive results under constrained training and compute conditions.

Q4.3: Why not use simpler models to fit validation loss?

Thank you for this valuable suggestion. We fully agree that it is important to justify the complexity of our parametric model. To this end, we implemented and compared several simpler baselines, as summarized below:

• Model 2 is a linear regression model with explicit transfer terms. It serves as the simplest meaningful baseline beyond uniform allocation.
• Despite being lightweight, simpler models (ID 1–3) yield significantly higher error or lower R² scores.
• Even Model 3, which introduces exponentiation and has more parameters than our model, still underperforms our proposed formulation (Model 4).

This demonstrates that our final parametric form is not only compact but also more expressive and better aligned with the empirical patterns in the data. We will include this comparison in the appendix of the final version to make the justification more transparent.

Q4.4: Did you include upsampling-style baselines like XLM-R?

Thank you for raising this point. We included two strong and widely-used upsampling-style baselines:

• Temperature sampling (Temp), which smooths the token distribution to favor low-resource languages.
• UniMax (suggested by Reviewer 1T2b), which boosts the contribution of the most frequent language in each batch.

We evaluated both baselines under identical 1B2 and 7B training setups, and report results in Tables R4.2 and R4.3. Across all multilingual benchmarks, CLIMB consistently outperforms both Temp and UniMax, demonstrating its superior effectiveness under comparable conditions.

We will include these results in the revised version for completeness.

Table R4.2 1B2 model Performance on new baselines and new benchmarks.

Table R4.3 7B model Performance on new baselines and new benchmarks.

Thank you for your thoughtful review and valuable suggestions. We are eager to further address any additional concerns you may have. We sincerely hope this response helps clarify the key points and merits of our work, and would greatly appreciate the opportunity to improve the evaluation of our work.

Q1.1: Weak Baselines

Thank you for raising this point. We have conducted additional experiments to include two stronger baselines: UniMax and temperature-based sampling (Temp). The results below show that our proposed method, CLIMB, consistently outperforms both baselines across multiple benchmarks and model sizes (1.2B and 7B):

Table R1.1 1B2 model Performance on new baselines and new benchmarks.

Table R1.2 7B model Performance on new baselines and new benchmarks.

In addition, we conducted statistical significance testing. Results show that the improvements of CLIMB over UniMax and temperature sampling are statistically significant (p < 0.05) on the majority of benchmarks. Detailed significance tests are provided in our response to Reviewer iYcL. These results confirm that CLIMB is a strong and reliable method for multilingual data allocation.

Q1.2: Benchmark Quality: Human vs. Machine Translated

We appreciate this feedback. To address the concern, we have included results on additional human-translated benchmarks such as Belebele, Include, MultiBLiMP and MGSM (suggested by Reviewer iYcL), as shown in Table R1.1 and R1.2. The consistent advantage of CLIMB on both human-translated and machine-translated benchmarks demonstrates its robustness and generalizability. We will include these results in the camera-ready version.

Q1.3: Controlling for Model Size in Scaling Law

Thank you for this thoughtful suggestion. Our current work primarily focuses on understanding how training data scale influences optimal language allocation. This is the central variable in our formulation and analysis.

We also evaluated CLIMB on models of different sizes, including 530M, 1.2B, and 7B, and applied the same scaling law fitting procedure. We found that the fitted parameters remain highly consistent across model sizes, with only minor differences—primarily in the scaling constant $E$, which does not affect the final computed ratios.

We further quantified this consistency by computing the relative standard deviation (RSD) across all parameters, which was only 1.62%, indicating strong robustness to model scale. In addition, as shown in Tables R1.1/R1.2 and Figure 6, CLIMB performs well on both 1.2B and 7B models, demonstrating its practical effectiveness across scales.

We agree that explicitly modeling parameter count as an additional input to the scaling law could further improve allocation quality, and we have included this as a future direction in Appendix G.

Q1.4: Complexity of CLIMB Training Procedure

Thank you for raising this concern. We clarify that the overall training complexity of CLIMB is linear in the number of languages $N$, not exponential.

As reported in Line 171, CLIMB performs the following steps:

• Monolingual Scaling Fitting: For each language, we train 2 monolingual models with different data scales → total: $2 × N$
• Cross-lingual Interaction Measurement: For each language, we sample 2 random language ratios, each evaluated under 2 different data scales → total: $4 × N$

Total training runs $= 2 × N + 4 × N = 6 × N$

This $O(N)$ complexity is manageable in practice, and we believe it is justified by the substantial improvements in convergence and downstream performance.

Q1.5: Related Work on Language Imbalance and Generalization

Thank you for pointing out this relevant work on language imbalance and its effect on cross-lingual generalization. We acknowledge its relevance and will include a discussion and proper citation in the camera-ready version.

Dear Reviewer 1T2b,

Thank you once again for your thoughtful and constructive feedback on our work.

We hope our responses have addressed your concerns clearly. If you have any further questions or suggestions, please feel free to let us know— we would be more than happy to provide additional clarification.

We sincerely appreciate your review, which has played an important role in improving the overall quality of our paper.

Best regards, All authors

Thanks for the new results. I’ll increase my rating by 1 (2 to 3). I had mentioned some sampling methods (like temperature-based sampling) as potential baselines worth comparing, and I appreciate that you addressed them in your response. I also noticed that other reviewers raised similar concerns, and you put considerable effort into the rebuttal, especially in response to reviewer iYcL. That said, the paper would benefit from a more comprehensive review of existing methods, or at least a mention of additional ones in the related work section. This could have been a much stronger paper if the authors had demonstrated a deeper familiarity with the relevant literature.

Thank you for your detailed and thoughtful review. We have carefully addressed your comments and made concrete improvements accordingly. If there are any remaining concerns, we would be more than willing to engage further and are genuinely eager for a reconsideration of the score.

Q3.1: Are the experiments in Section 2.2.1 circular in motivation?

Thank you for raising this important concern. We confirm that there is no circularity in our experimental design. Our methodology follows the standard approach in scaling law research. Specifically, we trained over 500 different 1B2 models with diverse, randomly sampled language ratios and data scales across 16 languages. These controlled experiments allow us to quantitatively measure cross-lingual interactions, rather than relying on assumptions. The patterns shown in Figure 2 are derived purely from these randomized runs.

We then fit a scaling law function (CLIMB) to these empirical results, capturing how validation loss depends on both data scale and multilingual composition. As shown in Table 1, the fitted function generalizes well to unseen settings, validating its accuracy. Based on this fitted function, we compute the optimal language ratios, and use them to train new models for the downstream evaluations reported in Table 2.

In this workflow, the formulation of CLIMB is based entirely on randomized experiments, while its effectiveness is verified through extrapolation and downstream performance—there is no overlap between hypothesis generation and evaluation, and thus no circular reasoning.

We appreciate the reminder that some qualitative observations in Figure 2 (e.g., diminishing returns, influence of language similarity) have been discussed in prior work. These works indeed support our empirical findings, and we will revise the paper to cite them accordingly.

Q3.2: Why not compare with open multilingual models like XGLM?

Thank you for the suggestion. We have added XGLM-1.7B and XGLM-7.5B results to Table R3.1 and Table R3.2.

Direct comparisons with open models like XGLM are not entirely fair, due to differences in:

• Training data: Our models use only FineWeb-2, while open models always include curated data such as QA, code, math and synthetic corpora.
• Training scale: Open models are trained on more tokens (e.g., 26T for QWEN3), compared to 1T for our models.
• Language coverage: Open models are trained on more languages than CLIMB.
• Initialization: Some open models benefit from distillation from larger proprietary models.

Despite these disadvantages, CLIMB-trained models achieve comparable or stronger results on several multilingual benchmarks.

Table R3.1 Comparison of CLIMB with open-source baselines on 7B

Table R3.2 Comparison of CLIMB with open-source baselines on 1B

Q3.3: Why are the bold values in Table 2 not the best overall?

Thank you for the suggestion. As noted in Line 269 and Q2, direct comparisons with open-source models are not fully fair due to differences in data, scale, and setup.

Therefore, in Table 2 we bold only the best results among our sampling methods to ensure fair intra-group comparison. We will clarify this bolding strategy in the table caption in the camera-ready version.

Q3.4: Why are standard errors and significance testing missing?

Thanks for the question. We report standard errors (from the evaluation harness) in Table R4. However, they are consistently small across benchmarks and fail to reflect meaningful differences.

To better assess significance, we conducted paired $t$-tests between CLIMB and each baseline. As shown in Table R5, CLIMB shows statistically significant improvements on 13–15 out of 17 benchmarks under both 1B2 and 7B settings (p < 0.05).

Table R3.3 Standard errors (stderr) from Harness for 1B2 models.

Table R3.4 Significance test on 7B and 1B2 models (✅ means p < 0.05)

Q3.5: Are 7B results fairly compared to other models?

Thank you for highlighting this. 7B results are in Table R3 and the significance results are in Table R5.

Q3.6: Is “state-of-the-art” appropriate?

Thank you for pointing this out. We agree that the term “state-of-the-art” may be misleading in this context. We will revise this wording in the camera-ready version.

Q3.7: Missing Details about Translated Benchmarks

Thank you for the detailed feedback. We clarify that our translated benchmark was first generated using GPT-4o, and then evaluated via human review. Translations were rated on a 5-point scale, with GPT outputs scoring 4.3 on average, and final human-reviewed versions scoring even higher.

Human evaluators were selected based on CEFR standard: C1 level in the target language and B2 in English. In total, human evaluation cost $31,212, ensuring quality and reliability.

We also applied the same evaluation process to datasets from [5–9] and found our translations to be comparable or stronger. As a result, we prioritized our own translated benchmark to better align with the 16-language setup in our study.

To address concerns about translation artifacts, we further added non-translated benchmarks (Belebele, MGSM, MultiBLiMP) in Table R3/R4. Since our model is not trained on code/math data, we excluded MCLM (competition-level math) to reflect our domain coverage.

We will clarify these details in the revised version.

Q3.8: Does the tokenizer design affect results?

Thank you for the insightful question. We used a custom tokenizer to better study the link between tokenizer design and cross-lingual transfer. Public tokenizers (e.g., XGLM, mT5) often have opaque training distributions, making such analysis difficult.

To ensure transparency, we trained our tokenizer on 6B tokens from the 16 target languages, with 50% English and the rest evenly split. This resulted in:

• English compression rate: 21.3%
• Non-English average: 29.4%

We found that CLIMB remains robust across different tokenizers, though the fitted parameters do vary. This suggests that tokenizer design influences cross-lingual interactions, which we plan to explore further in future work.

Q3.9: Which GlobalMMLU subset is used?

Thank you for your question. We used both the culturally specific and culturally agnostic subsets of GlobalMMLU in our evaluation.

Q3.10: Why does CLIMB underperform QWEN on GMMLU, GPQA, and TruthfulQA?

Thank you for the question. This is addressed in Q3.2.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We will incorporate all these experiments and clarifications in the revised version of the paper. If any concerns remain, we would be more than happy to address them. We sincerely hope our clarifications will support a positive assessment.

1. Clarifying the Relationship Between Models in Section 2 and Section 3

Thank you for the question. We would like to clarify the relationship between the models used in Section 2 and those in Section 3.

In Section 2, our goal is to explore how language ratios, training token amounts, and validation loss are related. We trained around 500 different models with diverse language allocation settings, from which we observed consistent patterns and derived a predictive function.

In Section 3, we validate whether this function can reliably capture and generalize the observed trends. This validation is carried out in two steps:

• In Section 3.2, we evaluate the fitting accuracy of the function on the same 500 models from Section 2 (the "Fitting Results" in Table 1). We understand the concern that this may appear as circular validation. However, this step is similar to reporting "loss on the training set" in standard machine learning tasks—it helps assess whether the function has enough capacity to fit the trends. Importantly, by fitting on a large and diverse set of 500 models, we further reduce the risk of overfitting to specific error patterns.

• In Section 3.3, we further validate the function on an additional set of 40 models that are completely separate from those in Section 2 (the "Extrapolation Results" in Table 1). There is no overlap between these two sets. These held-out models serve as a validation set to test generalization. Results on these held-out models demonstrate that the fitted function maintains strong performance, even on models trained with larger token budgets, confirming its ability to generalize beyond the model in section 2.

Overall, this validation procedure aligns with standard scaling law methodology [1-3] and does not constitute circular validation. We hope this explanation helps clarify the experimental setup and addresses concerns about potential circularity.

2. On Misleading Comparisons with Open-Source Models (LLaMA, Qwen, Gemma)

We appreciate the reviewer’s understanding that direct comparisons with open-source models are not entirely fair. Our goal is simply to provide a reference for readers, not to overstate our model's performance.

To avoid potential misunderstanding, we will make the following revisions:

• Clearly state that bolded results represent the best performance among data allocation methods in the table caption.
• Avoid using terms like "state-of-the-art" in the main text to prevent overclaiming.

References:

[1] Hoffmann, Jordan, et al. "Training Compute-Optimal Large Language Models." Advances in Neural Information Processing Systems 35 (NeurIPS 2022).

[2] Que, Haoran, et al. "D-CPT Law: Domain-Specific Continual Pre-training Scaling Law for Large Language Models." Advances in Neural Information Processing Systems 37 (NeurIPS 2024).

[3] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

Thank you for your thoughtful review and valuable suggestions.

Q2.1: Figures are hard to read—can they be enlarged?

Thank you for pointing this out. We will enlarge all figures and improve their readability in the camera-ready version.

Q2.2: Did you compare CLIMB with temperature sampling?

Thank you for the suggestion. We have added two strong baselines—temperature sampling (Temp) and UniMax (suggested by Reviewer 1T2b)—to our updated experiments. We also included non-translated benchmarks (Belebele, MGSM, MultiBLiMP, Include, suggested by Reviewer iYcL) to better assess multilingual capabilities.

As shown in Table R2.1 and R2.2, CLIMB consistently outperforms both baselines across benchmarks and model sizes, confirming its advantage over heuristic sampling methods.

Table R2.1 1B2 model Performance on new baselines and new benchmarks.

Table R2.2 7B model Performance on new baselines and new benchmarks.

Q2.3: Why only 16 languages? Can CLIMB scale to 100+?

This is a great question. To handle massive multilinguality, we applied CLIMB at the language family level instead of individual languages.

We filtered out families with fewer than 100B tokens in FineWeb-2 and retained 58 language families, covering 300+ languages in total. Results from Table R2.3 on the 1.2B model show that CLIMB still outperforms temperature sampling under this broader setting. What's more, CLIMB outperform open-source models on many benchmarks. Given the disadvantage of this comparison (details are listed in Rebuttal to iYcL Q3.2), this further claims the advantage over heuristic sampling methods.

Table R2.3 1B model Performance on CLIMB-300+.

We plan to further extend CLIMB to scale efficiently in even larger language scenarios.

Q2.4: Can you provide per-language or per-family analysis?

Thank you for the suggestion. To better understand CLIMB’s behavior across languages, we analyzed cross-lingual interaction ratios among the 16 languages in our setting. This helps explain how CLIMB assigns language proportions in a data-driven manner.

Table R2.3 average transferability for each language and the top-1 transfer-out language.

Key observations:

1. Intra-family transfer is strong. Languages within the same family (e.g., Spanish–Portuguese, Japanese–Korean) tend to benefit from one another, which supports CLIMB’s use of family-level grouping in large-scale allocations.
2. High-transfer languages like English, Spanish, and Indonesian receive larger allocations, as their benefits generalize well to other languages (see also Figure 7).

This analysis provides interpretability for CLIMB’s language decisions and further validates its allocation strategy.

Q2.5: Which languages benefit the most from CLIMB?

Thank you for the question. To understand CLIMB’s allocation behavior, we analyze transfer-out and transfer-in scores for each language (see Table R2.3).

• A higher transfer-out score means the language contributes more to improving other languages.
• A lower transfer-in score means the language gains less from others and must rely more on its own data.

We observe that English, Spanish, and Indonesian have the highest transfer-out scores. Adding these languages benefits many others, so CLIMB tends to allocate more to them to maximize overall utility.

In contrast, Chinese (zh) has the lowest transfer-in score, indicating it gains little from cross-lingual transfer. As a result, CLIMB assigns it a higher share of training data to preserve its performance—this may also explain its relatively smaller improvements on downstream tasks.

These findings are consistent with CLIMB’s final allocation in Figure 7, confirming that the learned language proportions are grounded in measurable cross-lingual dynamics rather than heuristics.