Scaling Laws for Downstream Task Performance in Machine Translation

1. Generalization to multiple languages:

In the revised paper, we provided a definition of alignment in Section 3.2 to be more concrete about what we meant by “more aligned” or “sufficiently aligned”:

$\mathcal{A}(D, T(L_{\text{source}}, L_{\text{dest}})) = P_{L_{\text{source}}} \cdot P_{L_{\text{dest}}} + 0.7 \cdot P_{L_{\text{source}}} + 0.8 \cdot P_{L_{\text{dest}}},$

where $D$ is the pretraining data mixture, $T(L_{\text{source}}, L_{\text{dest}})$ is the translation task from $L_{\text{source}}$ to $L_{\text{dest}}$, $P_{L_{\text{source}}}$ is percentage of $L_{\text{source}}$ in $D$, and $P_{L_{\text{dest}}}$ is percentage of $L_{\text{dest}}$ in $D$.

For instance, for an en-to-fr translation task, a pretraining data mixture with $50$% en and $50$% fr data would yield an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 0.5 \cdot 0.5 + 0.7 \cdot 0.5 + 0.8 \cdot 0.5 = 1$. Likewise, a pretraining data mixture with $100$% en would have an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 1 \cdot 0 + 0.7 \cdot 1 + 0.8 \cdot 0 = 0.7$.

This definition can be used for multilingual pretraining data with more than two languages as well. In such a case, the alignment between the pretraining data D and a set of translation tasks $T_1$, …, $T_k$, can be measured as $\frac{1}{K} \sum_{k=1}^K \mathcal{A}(D,T_k)$.

2. Discussion on new references:

We thank the reviewer for bringing these references to our attention. Their findings indeed support our conclusions about alignment and we added a discussion on this in the related work section in the revised paper.

3. Language similarity:

While we can reason about the language similarity based on our findings, e.g., french-english might be less similar than german-english due to the trend in scaling laws, we think such a conclusion requires more linguistic analysis. However, we do believe that we managed to highlight some interesting cases that are worth attention linguistically.

4. Lines 142-143: "Tran et al., 2019" is repeated 3 times:

We thank the reviewer for catching this. We fixed this in the revised paper.

5. Lines 264-265: it'd be nice to include fine-tuning dataset sizes here:

We thank the reviewer for the suggestion. We included dataset sizes accordingly in the revised paper.

We thank the reviewer for their time reading our paper and for providing very helpful and constructive feedback. If there is any remaining question or concern, we are happy to discuss further.

1. Applicability to other NLP tasks:

We agree that it is important to see to what extent the proposed logarithmic law applies to other NLP tasks. We actually provided empirical evidence in Figure 6 of Appendix that the same law successfully explains the scaling behavior of SuperGLUE benchmark scores, including Boolean Ques-tions (BoolQ), CommitmentBank (CB), Choice ofPlausible Alternatives (COPA), Multi-Sentence Reading Comprehension (MultiRC), Recognizing Textual Entailment (RTE), Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD), and Word-in-Context(WiC).

2. Can cross entropy be generalized to observe the scaling law?

Almost all existing scaling laws were proposed for cross entropy loss. We are unsure if we understand the reviewer’s question correctly and we are happy to discuss further if they can clarify their question a bit more.

3. Reference to OpenAI’s o1 model:

We thank the reviewer for this suggestion. The inference scaling law in OpenAI’s o1 model relates the train time and test time of the reinforcement learning step with the AIME accuracy. Our setup is different since we relate pretraining data to downstream performance with a finetuning step in between. We added the relevant reference in the revised paper.

4. Definition of alignment:

We use the alignment concept as a measure of how much language overlap there is between pretraining data and translation task, with a slightly higher weight on the target language rather than the source language. In the revised paper, we added a definition of alignment that measures language similarity between the pretraining and downstream task. Please see Definition 1 in Section 3.2 for the translation alignment score that reflects the way we interpret alignment for translation tasks:

$\mathcal{A}(D, T(L_{\text{source}}, L_{\text{dest}})) = P_{L_{\text{source}}} \cdot P_{L_{\text{dest}}} + 0.7 \cdot P_{L_{\text{source}}} + 0.8 \cdot P_{L_{\text{dest}}},$

where $D$ is the pretraining data mixture, $T(L_{\text{source}}, L_{\text{dest}})$ is the translation task from $L_{\text{source}}$ to $L_{\text{dest}}$, $P_{L_{\text{source}}}$ is percentage of $L_{\text{source}}$ in $D$, and $P_{L_{\text{dest}}}$ is percentage of $L_{\text{dest}}$ in $D$.

For instance, for an en-to-fr translation task, a pretraining data mixture with $50$% en and $50$% fr data would yield an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 0.5 \cdot 0.5 + 0.7 \cdot 0.5 + 0.8 \cdot 0.5 = 1$. Likewise, a pretraining data mixture with $100$% en would have an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 1 \cdot 0 + 0.7 \cdot 1 + 0.8 \cdot 0 = 0.7$.

We also included the corresponding scores in Figures 2, 3, 4, and 5 for each experiment.

We thank the reviewer for their time reading our paper and for providing very helpful and constructive feedback. If there is any remaining question or concern, we are happy to discuss further.

1. Definition of Alignment:

We use the alignment concept as a measure of how much language overlap there is between pretraining data and translation task, with a slightly higher weight on the target language rather than the source language. In the revised paper, we added a definition of alignment that measures language similarity between the pretraining and downstream task. Please see Definition 1 in Section 3.2 for the translation alignment score that reflects the way we interpret alignment for translation tasks:

$\mathcal{A}(D, T(L_{\text{source}}, L_{\text{dest}})) = P_{L_{\text{source}}} \cdot P_{L_{\text{dest}}} + 0.7 \cdot P_{L_{\text{source}}} + 0.8 \cdot P_{L_{\text{dest}}},$

where $D$ is the pretraining data mixture, $T(L_{\text{source}}, L_{\text{dest}})$ is the translation task from $L_{\text{source}}$ to $L_{\text{dest}}$, $P_{L_{\text{source}}}$ is percentage of $L_{\text{source}}$ in $D$, and $P_{L_{\text{dest}}}$ is percentage of $L_{\text{dest}}$ in $D$.

For instance, for an en-to-fr translation task, a pretraining data mixture with $50$% en and $50$% fr data would yield an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 0.5 \cdot 0.5 + 0.7 \cdot 0.5 + 0.8 \cdot 0.5 = 1$. Likewise, a pretraining data mixture with $100$% en would have an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 1 \cdot 0 + 0.7 \cdot 1 + 0.8 \cdot 0 = 0.7$.

We also included the corresponding scores in Figures 2, 3, 4, and 5 for each experiment.

2. Extensions:

Our focus in the paper is to understand the effect of the pretraining data and its size on the downstream performance. While we did not study how the model architecture or model size would affect the downstream performance, we do not expect this to change our conclusions about the effect of data and its size.

We thank the reviewer for their time reading our paper and for providing very helpful and constructive feedback. If there is any remaining question or concern, we are happy to discuss further.

1. Presentation:

We thank the reviewer for the feedback about the organization of the paper. We revised the paper, following their suggestions. We moved the highlight of the main results (Figure 1) from the introduction section to the methods section and connected it to the new section on alignment definition (Section 3.2).

2. Definition of alignment:

The reviewer asks valuable questions about the definition of “alignment” and how we can interpret this in the context of this paper. As the reviewer correctly stated, we use the alignment concept as a measure of how much language overlap there is between pretraining data and translation task, with a slightly higher weight on the target language rather than the source language. In the revised paper, we added a definition of alignment that measures language similarity between the pretraining and downstream task. Please see Definition 1 in Section 3.2 for the translation alignment score that reflects the way we interpret alignment for translation tasks:

$\mathcal{A}(D, T(L_{\text{source}}, L_{\text{dest}})) = P_{L_{\text{source}}} \cdot P_{L_{\text{dest}}} + 0.7 \cdot P_{L_{\text{source}}} + 0.8 \cdot P_{L_{\text{dest}}},$

where $D$ is the pretraining data mixture, $T(L_{\text{source}}, L_{\text{dest}})$ is the translation task from $L_{\text{source}}$ to $L_{\text{dest}}$, $P_{L_{\text{source}}}$ is percentage of $L_{\text{source}}$ in $D$, and $P_{L_{\text{dest}}}$ is percentage of $L_{\text{dest}}$ in $D$.

For instance, for an en-to-fr translation task, a pretraining data mixture with $50$% en and $50$% fr data would yield an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 0.5 \cdot 0.5 + 0.7 \cdot 0.5 + 0.8 \cdot 0.5 = 1$. Likewise, a pretraining data mixture with $100$% en would have an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 1 \cdot 0 + 0.7 \cdot 1 + 0.8 \cdot 0 = 0.7$.

We also included the corresponding scores in Figures 2, 3, 4, and 5 for each experiment.

3. Remark 2:

We thank the reviewer for pointing out the unclarity in Remark 2. In remark 2, we wanted to understand how each of the terms in BLEU score behaves as the pretraining data increases, to better understand which term(s) is responsible for the non-monotonic behavior of the BLEU score especially when the alignment is not sufficient, i.e., when the BLEU score fluctuates with more pretraining data. We found out that the brevity-penalty is always 1 in our experiments. This means, the non-monotonic behavior comes from the precision term. We clarified these take-aways in the revised paper.

4. Guide for Data Valuation: We revised the second step of the data valuation guide. We thank the reviewer for pointing out the unclarity.

We thank the reviewer for their time reading our paper and for providing very helpful and constructive feedback. If there is any remaining question or concern, we are happy to discuss further.

1. Clarification on value of pretraining vs finetuning data size”:

We understand that the conclusion from our empirical observations was not communicated clearly enough. We empirically observed that, in the particular setup we considered, as the finetuning data size grows, pretraining brings less value. This is an empirical finding, which led to the conclusion that “the value of pretraining data depends on the size of the finetuning data”. Notice that for the specific setup we considered, it is indeed empirically evident that pretraining does not bring almost any value when the finetuning data is large enough. However, we never state this as a general suggestion since high-quality finetuning data is more expensive and harder to obtain than unsupervised pretraining data. So, in practice, we typically do not have sufficiently large finetuning data and we agree that pretraining is still recommended before finetuning the model on supervised data, which aligns with the common practice, the reviewer’s comments, and other papers’ findings, e.g. the ones stating that “we need large pretraining and a small finetuning data”. We apologize for this confusion, we revised our paper to clarify this point further, see the text in blue in pages 7-8.

2. Metrics:

The reviewer stated that BLEU score is not as aligned with human judgments as other metrics like COMET. In case this was missed during their reading, we actually provided results with COMET score as well. In particular, Figure 1, Section C.1, and Figure 7 show the scaling behavior of COMET score under the setups considered in the paper. We found that the COMET results are similar to BLEU results, in that the COMET score is also predictable with the proposed scaling law as long as there is sufficient alignment.

3. Writing and Organization:

We thank the reviewer for their constructive comments. We reorganized the paper and moved figures closer to where they are referenced.

4. Alignment Definition:

The reviewer asks valuable questions about the definition of “alignment” and how we can interpret this in the context of this paper. As the reviewer correctly stated, we use the alignment concept as a measure of how much language overlap there is between pretraining data and translation tasks, with a slightly higher weight on the target language rather than the source language. A more general definition of alignment, e.g. for the games vs news, would be an important contribution to the field and definitely deserves attention from the community. In the revised paper, we added a definition of alignment that measures language similarity between the pretraining and downstream task. Please see Definition 1 in Section 3.2 for the translation alignment score that reflects the way we interpret alignment for translation tasks:

$\mathcal{A}(D, T(L_{\text{source}}, L_{\text{dest}})) = P_{L_{\text{source}}} \cdot P_{L_{\text{dest}}} + 0.7 \cdot P_{L_{\text{source}}} + 0.8 \cdot P_{L_{\text{dest}}},$

where $D$ is the pretraining data mixture, $T(L_{\text{source}}, L_{\text{dest}})$ is the translation task from $L_{\text{source}}$ to $L_{\text{dest}}$, $P_{L_{\text{source}}}$ is percentage of $L_{\text{source}}$ in $D$, and $P_{L_{\text{dest}}}$ is percentage of $L_{\text{dest}}$ in $D$.

For instance, for an en-to-fr translation task, a pretraining data mixture with $50$% en and $50$% fr data would yield an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 0.5 \cdot 0.5 + 0.7 \cdot 0.5 + 0.8 \cdot 0.5 = 1$. Likewise, a pretraining data mixture with $100$% en would have an alignment score of $\mathcal{A}(D, T(\text{en}, \text{fr})) = 1 \cdot 0 + 0.7 \cdot 1 + 0.8 \cdot 0 = 0.7$.

We also included the corresponding scores in Figures 2, 3, 4, and 5 for each experiment.

5. Suggestion for future work:

This is a very interesting and compelling idea. We believe this would be a great direction for future work and we think our work serves as an initial step towards understanding scaling behavior of translation quality.

We thank the reviewer for their time reading our paper and for providing very helpful and constructive feedback. If there is any remaining question or concern, we are happy to discuss further.

Thank you to the authors for their additional response; I greatly appreciate the effort. Regarding points 1 and 2, I acknowledge that the study's goals and findings provide valuable insights, alarming specific considerations for researchers and practitioners in machine translation. However, as I previously mentioned, the findings appear to be overly scenario-specific, which limits their broader applicability.

On point 3, I would like to emphasize that BLEU scores are widely recognized as a poor metric for evaluating machine translation quality, because of their high misalignment with human preferences. BLEU ranked 28/32 metrics for machine translation on WMT23. The reliance on lexical overlap for evaluation is fundamentally flawed; the primary reason BLEU has been used since 2002 is the historical lack of better alternatives. However, with the advent of neural-based metrics, especially for high-resource languages (that studied in the paper), we now have more robust and reliable tools for evaluation. This shift in the field is well-established, and I encourage the authors to consider this consensus and align their approach accordingly.

While it is true that many papers at top-tier conferences still report BLEU scores (including many of my own, my papers are typically done for completeness and comparability with prior work rather than as an endorsement of BLEU’s validity). The prevalence of BLEU in top-tier conferencesdoes not inherently justify its continued use as the primary metric. Instead, the authors should focus on understanding which metrics are currently considered higher quality for evaluation, rather than blindly trusting previous top-tier conference papers. This is a common pitfall for researchers. A lack of understanding of the metrics could stem from oversight or misinterpretation, but I encourage the authors to address this by conducting a more thorough investigation.

Some papers may help the authors understand why BLEU is not an ideal metric. As for ROUGE, I don't know because it is very rare for MT. Based on what the authors mentioned, only 1 out of 7 papers referenced ROUGE, and it was in the context of self-driving?

[1] Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies [2] Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust [3] Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent

There should be more papers mentioning this point, but it needs me take a while to find them, so I just stop here. Finally, I strongly encourage the authors to develop a solid understanding of evaluation metrics before attempting to make contributions to the field. While this feedback may come across as a bit harsh, I believe it is important to emphasize here.

(I believe the authors had a misunderstanding of my initial review. I did not say that you did not use COMET or you should considering use COMET, nor did I overlook its inclusion. My point was that BLEU is a poor metric compared to COMET, and I suggested that you consider alternatives such as BLEURT)