The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model

Thanks for your constructive comment. Here is our response to your concerns.

[W1] How to compute the number of tokens in the target language using $P(\ell_j)$ and $\eta_{\text{python}}$ based on Eqn 3 ?

In Algorithm 1, after computing $P(\ell_j)$ using Equation 2, we assign its value to $\bar{P}(\eta_j)$ in Equation 3, since both $P(\ell_j)$ and $\bar{P}(\eta_j)$ represent the estimated proportion of the Python system. The number of tokens corresponding to the new language can be calculated based on Equation 3: $\eta_{\text{target}}=\eta_{\text{python}}/\mathcal{P}(\ell_j)-\eta_{\text{python}}$ Thank you for your suggestion. We provide a clearer description in the revision.

[W2] What do “PHP predicted” and “PHP Actual” mean in Figure 6 and 7? And why there are much more “PHP predicted” dots than “PHP Actual”?

Each point in Figure 6 and 7 represents one checkpoint, the y-axis is performance and the x-axis is the proportion of the Python system.

"PHP predicted" contains one pre-training process as Algorithm 1, and the pre-training corpus is PHP corpus. We evaluate the checkpoints during the pre-training process and calculate the proportion of the Python system based on the corresponding training loss using Equation 2.

"PHP actual" contains several pre-training processes as in Section 4.1, and the pre-training corpus is a mixed corpus of Python and PHP (with varying proportions for different pre-training processes). We evaluate the final checkpoint of each experiment and calculate the proportion of the Python system based on the corpus proportions of different languages using Equation 3.

In one training process, “PHP predicted” contains many checkpoints while "PHP actual" contains only one checkpoint, so the number of dots in "PHP predicted" is significantly higher than the number of points in "PHP actual".

[Q1] What is the optimized data distribution? Have you performed any checks on data quality? Is it possible that these results are actually due to some bad quality data (e.g. noisy/incorrect codebases), instead of the claimed knowledge transfer between languages?

The optimized data distribution is estimated using Algorithm 1. In this paper, we utilize high-quality pre-training data, having conducted quality filtering and decontamination beforehand. We apply Algorithm 1 to estimate the number of tokens in the target language corpus for an optimized data distribution. As noted in Section 4.3.3, during down-sampling, we employ a purely random strategy without excluding poor-quality data.

[Q2] I think a better set of experiments may be to use existing open-source multilingual datasets, and show that the proposed optimized data distribution can produce stronger pre-train models. In this case the comparison can include existing coding LLMs.

Thank you for your suggestion. Given the potential data quality issues inherent to open-source pre-training corpus, we use our own higher-quality data to mitigate the bias you mentioned above, i.e., performance variations arising from the inclusion or exclusion of low-quality data.

We have pre-trained our models for an extended number of steps. For the 1.3B LLM, the total number of processed tokens was set to 500 billion, while for the 6.7B LLM, the total number of processed tokens was set to 800 billion. We compared our models (both Original and Optimized) with other open-source models, and the results are presented below:

1.3B HumanEval performance:

6.7B HumanEval performance:

As the number of training steps increased, our model achieved performance that surpassed several open-source LLMs. Furthermore, the LLM pre-trained with optimized pre-training corpora (Optimized) outperformed the LLM pre-trained with original pre-training corpora (Original), demonstrating the effectiveness of our approach.

[Q3] Based purely on programming languages, it is not clear whether the “Babel Tower Hypothesis” is generalizable or not. It may be more informative if there are more experimental results on natural languages to support this general hypothesis.

Code LLMs have gained significant attention from researchers and represent an important direction in current research. In this paper, although we focus on programming languages, we do not make additional assumptions about language distribution, thus enabling potential extensions to other distributions, such as natural language. Our validation methods, such as working language and language transfer neurons, are derived from related works in LLMs of natural language. Due to resource limitations, we plan to further explore the adaptation of our work in the context of natural language processing in the future.

Dear reviewer,

The rebuttal discussion period is coming to a close and have provided responses to your concerns. We are happy to discuss more if you still have any concerns.

Dear Reviewer 4Qay

We hope this message finds you well. We apologize for reaching out repeatedly, but with less than two days remaining in the discussion period, we wanted to check whether our updates have addressed your concerns kindly.

If you have any further questions or require additional clarification, we are more than happy to provide it. Your feedback has been invaluable, and we greatly appreciate your time and effort in reviewing our work.

Thank you again for your continued support. We look forward to your feedback.

Sincerely

All authors

Thanks for your comment. Here is our response to your concerns.

[W1 and Q1] The submission's use of the term "multilingual capabilities" is unclear. It seems to refer to the model's capacity to understand multiple programming languages, rather than the more common interpretation of "multilingual" as understanding multiple natural languages. Why combine the concepts of multiple natural languages and multiple programming languages?

In this paper, we explore the multilingual capabilities of code LLMs, analogous to their natural language counterparts, where "multilingual" in this context refers to multiple programming languages.

"Multilingual" is a common expression used in code LLM and many existing open-source code LLMs use this term, such as CodeGeeX [1], StarCoder [2], and DeepseekCoder [3].

[1] CodeGeeX https://github.com/THUDM/CodeGeeX

[2] StarCoder https://huggingface.co/blog/starcoder

[3] DeepseekCoder https://deepseekcoder.github.io/

[W2] The methodology's flaws undermine the hypothesis presented. The claim that multilingual programming language capabilities are primarily developed during pre-training is unconvincing. It's likely that fine-tuning plays a significant role, and a more comprehensive mechanism should be considered.

Pre-training and fine-tuning are both essential stages for a LLM to acquire multilingual capabilities. This paper primarily focuses on the pre-training stage, as it is a critical stage in developing the multilingual capabilities of LLMs [1]. Understanding how LLMs achieve multilingual capabilities through unsupervised training constitutes an important research direction. And there are many related studies studying the multilingual abilities of pre-trained base LLMs [2, 3].

[1] Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models https://arxiv.org/abs/2402.16438

[2] LLMs Are Few-Shot In-Context Low-Resource Language Learners https://arxiv.org/abs/2403.16512

[3] Do Llamas Work in English? On the Latent Language of Multilingual Transformers https://arxiv.org/abs/2402.10588

[W3] The assertion that Python is the dominant language due to its widespread use lacks evidence and requires further clarification.

The selection of the primary programming language is influenced by the inherent characteristics of the language. We chose Python due to its widespread popularity and the availability of a large corpus of high-quality code. Several related studies highlight Python's dominance in the field of code LLMs. For example, CodeLlama has released a python-specialized version, CodeLlama-70B-Python [1]. Additionally, existing evaluations of open-source models [2,3] and benchmarks for assessing open-source code predominantly emphasize Python [4,5].

[1] https://ai.meta.com/blog/code-llama-large-language-model-coding/

[2] StarCoder: may the source be with you!, https://arxiv.org/abs/2305.06161

[3] Qwen2.5-Coder Technical Report https://arxiv.org/abs/2409.12186

[4] BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions https://arxiv.org/abs/2406.15877

[5] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code https://arxiv.org/abs/2403.07974

Note: Authors should not reveal their identities. Descriptions like "Specifically, we conducted preliminary experiments to analyze how monolingual LLMs learn a new language by continual pre-training the monolingual LLM on the corpus with the new language (Zheng et al., 2024)" should be revised to remove identifying information.

This sentence may potentially lead to some misunderstandings; we cite Zheng et al., (2024) to indicate that the setting of our preliminary experiments (in Section 3) is similar to the setting of the experiments in Zheng et al., (2024), i.e., learning a new language through continual pre-training. We will revise this sentence to avoid any misunderstanding.

Dear reviewer,

Thanks for your constructive comments! We have provided responses to your concerns. Do our responses address your questions? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

Dear reviewer,

The rebuttal discussion period is coming to a close and have provided responses to your concerns. We are happy to discuss more if you still have any concerns.

Dear Reviewer cRyw

We apologize for reaching out repeatedly, but with less than two days remaining in the discussion period, we wanted to check whether our updates have addressed your concerns kindly.

If you have any further questions or require additional clarification, we are more than happy to discuss more. Your feedback has been invaluable, and we greatly appreciate your time and effort in reviewing our work.

Thank you again and look forward to your feedback!

Sincerely

All authors

Thanks for your suggestions on our work. Here is our response to your concerns.

[W1] The scope of this paper is a bit limited, as it focuses exclusively on code LLMs. Due to the unique characteristics of programming languages, the findings may not be easily applicable to natural languages.

Code LLMs have gained significant attention and represent an important direction in current research [1,2,3]. Meanwhile, code LLM also supports numerous important applications, such as Cursor[4] and Copilot[5]. Therefore, we believe that code LLM itself is an important topic. In this paper, although we focus on programming languages, we do not make additional assumptions about language distribution, thus enabling potential extensions to other distributions, such as natural language. We plan to further explore the adaptation of our work in the context of natural language processing in the future.

[1] StarCoder: may the source be with you!, https://arxiv.org/abs/2305.06161

[2] Code Llama: Open Foundation Models for Code, https://arxiv.org/abs/2308.12950

[3] Qwen2.5-Coder Technical Report， https://arxiv.org/abs/2409.12186

[4] https://www.cursor.com/

[5] https://github.com/features/copilot

[W2] It remains unclear whether the hypothesis would still hold if a different primary language were used.

The selection of the primary programming language is influenced by the inherent characteristics of the language. We chose Python due to its widespread popularity and the availability of a large corpus of high-quality code.

Several related studies highlight Python's dominance in the field of code LLMs. For example, CodeLlama has released a python-specialized version, CodeLlama-70B-Python [1]. Additionally, existing evaluations of open-source models [2,3] and benchmarks for assessing open-source code predominantly emphasize Python [4,5].

[1] https://ai.meta.com/blog/code-llama-large-language-model-coding/

[2] StarCoder: may the source be with you!, https://arxiv.org/abs/2305.06161

[3] Qwen2.5-Coder Technical Report https://arxiv.org/abs/2409.12186

[4] BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions https://arxiv.org/abs/2406.15877

[5] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code https://arxiv.org/abs/2403.07974

[W3-1] In sections 4.2.1 and 4.2.2, it is not well explained how the estimated proportion of the Python system P(l) relates to α ( the initial loss) and β (the final loss) and how P(l) determines the optimal number of tokens for the target languages.

In Equation 2, we establish a linear relationship between the training loss $\ell$ and the proportion of the Python system $P(\ell)$, based on the initial loss $\alpha$ and final loss $\beta$. During pre-training, as the loss gradually decreases from \alpha to \beta, the corresponding proportion of the Python system is also gradually decreasing.

To determine the optimal proportion, we apply Algorithm 1. Specifically, we pre-train the Python monolingual LLM with the corpus of a new language and save several checkpoints. Then we evaluate the saved checkpoints and identify the checkpoint with the best performance. Futhermore, we use the training loss of the best checkpoint to estimate the current proportion of Python system $P(\ell_j)$ (via Equation 2). After computing $P(\ell_j)$ using Equation 2, we assign its value to $P(\eta_j)$ in Equation 3, since both $P(\ell_j)$ and $P(\eta_j)$ represent the estimated proportion of the Python system. The number of tokens corresponding to the new language can be calculated based on Equation 3: $\eta_{\text{target}}=\eta_{\text{python}}/\mathcal{P}(\ell_j)-\eta_{\text{python}}$.

[W3-2] According to equation (2), the proportion P(l) cannot exceed 1. What happens if there is sufficient amount of training data for the target language?

With the addition of more training data, the corresponding $\beta$ (the final training loss during pre-training) will change, ensuring that it does not exceed 1.

Dear reviewer,

Thanks for your constructive comments! We have provided responses to your concerns. Do our responses address your questions? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

Thanks for your constructive comments. We appreciate your feedback and here is our response to your concerns.

[Q1] It is also important to show the performance of Python to illustrate the impact on the learned languages.

As illustrated in Section 3.1, to trace the performance of Python, we incorporated a small amount of Python data (a total of 100 million tokens) into the monolingual corpus of other languages. The result is shown in the table and we also included a figure (Figure 8) in the revision of our paper:

In the first stage, the performance of the new language increases, while Python experiences a slight decline. During the second stage, as the new language system is established and competes with Python, the Python's performance significantly declines. Finally, Python's performance stabilizes.

[Q2] Could the authors show some examples of python-specific problems which are crucial to define the knowledge transferring procedure?

Python-specific problems primarily encompass tasks that Python excels at, often depending on the extent of related knowledge within the pre-training corpus. For example, compared to PHP, Python usually has more algorithm corpus and thus demonstrates superior performance in algorithmic challenges, such as "parentheses matching" and "computing polynomial derivatives".

[Q3-1] How to calculate the loss which might consist of python data only in Equation 2.

As illustrated in Algorithm 1, Equation 2 is applied in the new language acquisition experiment. In this experiment, we pre-train a Python monolingual LLM with the corpus of the new language. Consequently, the training loss is computed on the new language data rather than Python.

[Q3-2] The authors define a linear relationship between the training loss and estimated portion of the python system as in eq (2).

Further justification of the relationship other than results in Fig 6 and 7 to demonstrate the scalability of this hypothesis To further demonstrate the relationship between the training loss and the proportion of the python system, we conducted a working language analysis to trace the internal state of the LLM.

As illustrated in Section 3.3, working language is the language which is generated in the intermediate layers of LLMs. We can use the proportion of Python being the working language as an approximation for the proportion of the Python system.

Specifically, we calculate the proportion of Python being the working language at different training steps during the pre-training process. Then we compare them with the proportion of Python systems estimated from the training loss (Equation 2). The result is shown in Figure 9 of the revision of our paper and Figure 9 shows that the relationship between the training step and the Python system exhibits a logarithmic-linear correlation.

We used observation data (take PHP as an exmaple) to fit the relationship between training step and the proportion of the Python system:

Training loss: $y_1=-0.3758*log(x)+1.6166$

Working language: $y_2 = -0.3328*log(x) + 1.4973$

where x represents the training step, $y_1$ represents the proportion of the Python system estimated using training loss, and $y_2$ represents the proportion of the Python system estimated using the working language.

We calculate the difference between the two: $\delta = y_1-y_2 = -0.043 * log(x) + 0.1193$, where $\delta$ represents the difference between the proportion of the Python system we use to estimate the training loss and the corresponding proportion for which Python is used as the working language.

We can see that the difference between estimates using training loss and those using working language is minimal, thereby substantiating the relationship between training loss and the estimated proportion of the Python system.

Dear reviewer,

Thanks for your constructive comments! We have provided responses to your concerns. Do our responses address your questions? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

Dear reviewer,

The rebuttal discussion period is coming to a close and have provided responses to your concerns. We are happy to discuss more if you still have any concerns.