SCALE: Synergized Collaboration of Asymmetric Language Translation Engines

Concern #5: The selection of GPT-2 for experiments.

Response: We understand your concerns about our choice of using GPT-2 in our experiments. The use of GPT-2 in this context is solely as an evaluator to measure the fluency of translations, a role it has successfully fulfilled in recent studies such as [2]. Despite being a model from four years ago, GPT-2 still serves as an effective evaluator for assigning perplexity scores to English sentences.

To address your concern and ensure the reliability of our results, we also conducted the same experiments using Llama-2-7B and Llama-2-13B. The results of these tests are shown below. While Llama2 gives different absolute values of perplexity of STM, GPT-4, and SCALE, the relative order remains identical to that obtained with GPT-2, thereby reinforcing our confidence in the original findings.

This is the original results produced by GPT-2-XL:

This is the results produced by Llama-2-7B:

This is the results produced by Llama-2-13B:

[1] Raunak, Vikas, et al. "Leveraging GPT-4 for Automatic Translation Post-Editing." arXiv preprint arXiv:2305.14878 (2023).

[2] Hendy, Amr, et al. "How good are gpt models at machine translation? a comprehensive evaluation." arXiv preprint arXiv:2302.09210 (2023).

Dear Reviewer,

We sincerely appreciate your time and insightful comments on our paper. Your commendation of our idea is encouraging, and we are eager to address the concerns you raised. Further, we kindly request that you reconsider your score should our responses help alleviate some of your concerns.

Concern #1: The efficacy of the En→X translation setting

Response: We appreciate your insightful comment and agree that evaluating the efficacy of En→X translation is crucial. To address this, we have expanded our experiments to include translations from English to six low-resource languages, using the same experimental setup as that of SCALE-refine. The results, gauged by COMET-22 and COMETKiwi, are presented below, and all corresponding outputs have been appended to the supplementary zip files for your reference. These additional experiments lead us to several observations:

1. SCALE exemplifies its proficiency in translating from English to a diverse range of languages, spanning various scripts and linguistic families.
2. While SCALE shows improvement over few-shot GPT-4 and NLLB, the magnitude of this improvement is less pronounced compared to the X→En setting. This indicates that the present limitation for LLM translators mainly lies in the generation of the target language, rather than the comprehension of the source language, as also suggested in [1].
3. Due to the limited transparency of the GPT-4 model, it is challenging to attribute these inconsistencies to specific factors like tokenization, data coverage, or the proportion of languages in the training data. Regardless, we are striving to identify a potential indicator, specifically the number of native speakers, to illustrate the extent of a language's resources. As illustrated in the accompanying table, the performance of few-shot GPT-4 diminishes with the number of native speakers, while our framework, SCALE, consistently and effectively mitigates this language bias, outperforming few-shot GPT-4 and the supervised NLLB model in 6/6 directions.

Recognizing the significance of these findings, we would include this discussion in the final version of our paper.

Concern #2: Clear motivation for low-resource setting.

Response: We acknowledge your concern about the perceived focus of SCALE on low-resource languages and apologize if our manuscript did not communicate this aspect clearly. Your suggestion to clarify the motivation of our experimental design is well taken.

While it is indeed true that current LLMs demonstrate exceptional performance in high-resource settings, our curiosity was piqued regarding the performance of SCALE in high-resource languages. We carried out experiments in three languages: German, Czech, and Chinese, under the same settings as SCALE-refine. We utilized COMET-22 and COMETKiwi to measure the translations. As shown below, the results reveal that SCALE offers consistent improvements even for high-resource languages, albeit less significantly than for low-resource languages, and it exhibits robust performance even when paired with a less performant STM (especially in en-zh direction). This is remarkable, given the already strong performances of models such as GPT-4 and NLLB. All the generated output has been updated to the supplementary zip files for your reference.

We assure you that these insights will be integrated into the revised version of our manuscript to provide a more comprehensive understanding.

I appreciate the authors for providing comprehensive results. This indeed makes me better understand how SCALE works on en->xx and high-resource languages. It surprises me that SCALE also works well on en->xx (though less pronounced). Although this method is not very helpful for high-resource languages, I believe this paper should be interesting to researchers interested in low-resource translation. Thus, I have raised my score. The authors seem to conduct extra experiments for four new low-resource languages (only Amharic and Nepali overlap with the paper), not the original eight low-resource languages, i.e., Assamese (asm Beng), Armenian (hye Armn), Amharic (amh Ethi), Xhosa (xho Latn), Uyghur (uig Arab), Khmer (khm Khmr), Nepali (npi Deva), and Sindhi (snd Arab). Why is this?

I thank the authors for their response. I am inclined to maintain my score for the following reasons:

• The scope of the initial paper primarily focuses on xx->en translations in low-resource languages. While the authors have presented results for high-resource languages and en->xx translations, their effectiveness appears less pronounced.
• I would be more inclined to rate the paper based on the initial submission. I acknowledge that some of results were shown in the rebuttal, but the manuscript could be substantially improved further with these extra experiments, e.g., en->xx translation results, results for high-resource languages, and a more comprehensive ablation study.

Nonetheless, this work should be of interest to those engaged in LLM machine translation and research in low-resource languages.

Dear Reviewer,

Thank you for your comprehensive feedback and for investing your time in reviewing our paper. We appreciate your insightful comments and understand your concerns. Allow us to provide some clarifications.

Firstly, we would like to point out that the purpose of our Analysis section is to delve into the translation characteristics of the LLM, STM, and SCALE, rather than to present overall quality metrics such as COMET or BLEU. Regarding the perplexity score, we did not aim to decrease the perplexity of the GPT2-XL model. Instead, our intention was to utilize GPT2-XL as an evaluator to assess the fluency of the translations generated by these three translation systems. The conclusion drawn was that when STM-generated output is used as in-context demonstrations, GPT-4 has the potential to generate more fluent translations (similar pattern is also found when using Llama-2 as evaluator as shown in our response to Reviewer xCiG).

We understand your confusion regarding the connection between fluency and quality. However, we abstained from establishing a direct correlation as LLM-based translations often exhibit higher fluency, irrespective of the quality of the translation. For a more detailed analysis, we refer to Section 5.2 in [1].

Moving on to the NM and USW score, this metric is designed to measure the literalness of the translation output. Interestingly, the higher NM score observed for STM resulted in a lower NM score for SCALE. A conceivable explanation for this is that in the triplet in-context learning, the role of the STM-generated output is not to instruct the LLM to generate specific words, but rather to constrain the paraphrastic expression space, thereby helping the LLM to achieve a higher translation quality by comparing $Z$ and $Y$ in the triplet demonstrations. As mentioned in our paper, SCALE tends to maintain the original structure of the LLM output, whilst incorporating word selection cues from the STM.

With respect to your question about the selection strategies for in-context learning samples, we did conduct some preliminary studies to evaluate their impact. Surprisingly, the performance of SCALE did not show a significant variation with the selection strategy. The three strategies we used were (1) first-k, (2) random, and (3) maximum-coverage, as suggested in [2]. No significant performance differences were observed among these three methods. We value your suggestion and will ensure this aspect is included in the final draft of our paper.

We hope that our clarifications have satisfactorily addressed your concerns. We kindly request you to reconsider your score in light of these explanations. We welcome any further questions or concerns you may have and assure you that we are committed to improving our work based on your valuable feedback.

Thank you once again for your time and effort.

[1] Hendy, Amr, et al. "How good are gpt models at machine translation? a comprehensive evaluation." arXiv preprint arXiv:2302.09210 (2023).

[2] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context Examples Selection for Machine Translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.

Question: Experiments on high-resource languages

Response: In our experiment design, we primarily evaluate SCALE in the challenging low-resource setting. This decision was informed by current research findings that indicate Large Language Models already perform comparably with supervised models on high-resource languages, yet struggle with low-resource languages [1][2]. Therefore, we believe it is more meaningful and challenging to focus on improving LLMs in this lower-resource setting.

To resolve some of your concerns, we still would like to test SCALE performance in the high-resource setting. We carried out experiments in three languages on the Flores datasets: German, Czech, and Chinese, under the same settings as SCALE-refine. We utilized COMET-22 and COMETKiwi to measure the translations. As shown below, the results reveal that SCALE offers consistent improvements even for these high-resource languages, albeit less significantly than for low-resource languages as you predicted, and it exhibits robust performance even when paired with a less performant STM (especially in en-zh direction). This is remarkable, given the already strong performances of models such as GPT-4 and NLLB. All the generated output has been updated to the supplementary zip files for your reference.

We assure you that these insights will be integrated into the revised version of our manuscript to provide a more comprehensive understanding.

We hope that our responses have addressed your concerns adequately. We would greatly appreciate it if you could consider revising your score in light of these clarifications. Please feel free to ask any further questions or voice any other concerns you might have.

[1] Garcia, Xavier, et al. "The unreasonable effectiveness of few-shot learning for machine translation." International Conference on Machine Learning. PMLR, 2023.

[2] Vilar, David, et al. "Prompting palm for translation: Assessing strategies and performance." arXiv preprint arXiv:2211.09102 (2022).

Dear Reviewer,

Firstly, we would like to express our sincere gratitude for your time and effort in reviewing our paper. We appreciate your thoughtful feedback and are eager to address your concerns and questions.

Concern #1: Lack of scientific insight

Response: We acknowledge your viewpoint, but we stand firmly to dispute your claim that our work is predominantly engineering-oriented and lacks scientific insight. We wish to assert with utmost certainty that our work is not confined to the realm of engineering but makes significant strides in contributing to the scientific literature in these crucial ways:

1. Originality: We have pioneered an approach that seamlessly integrates two distinct translation paradigms within a single framework. This is a first in the field and enriches the existing body of knowledge by demonstrating how these two asymmetric translation models can work in synergy through the mechanism of triplet in-context learning.
2. Rigorous Experimentation: Our research encompasses stringent experiments that authenticate our approach. These tests extend beyond mere engineering prowess to involve scientific investigation, presenting empirical proof of how triplet in-context learning can bolster the performance of translation models in a variety of scenarios, including refinement, pivoting, and updating.
3. In-depth Analysis: We have conducted a thorough scrutiny of the translation traits of STM, LLM, and SCALE, delineating the subtle dissimilarities among these three systems. Furthermore, we have undertaken a sweeping analysis of SCALE to validate each critical design in our framework, enhancing our comprehension of SCALE in terms of its robustness and latency cost.
4. Far-reaching Implications: The outcomes of our research have profound implications for the design of future translation systems. They equip practitioners with strategic insights on when to employ large language models, specialized models, or a blend of the two. Additionally, our study tackles a pivotal question in machine translation literature in the era of Large Language Models: Should we continue to train specialized translation models, or should we wholeheartedly adopt LLM-based translation? Our paper breaks new ground by proposing a feasible solution that amalgamates these two systems within a single framework.

We are confident that this articulates the scientific merit of our work. We will make it a point to highlight these aspects more prominently in our revised manuscript to forestall any potential misconceptions.

Concern #2: Weak baseline

Response: We would like to clarify that we have selected three types of baselines for our study: supervised models (NLLB, M2M100), few-shot LLMs (XGLM, GPT-3.5, and GPT-4), and one commercial translation system (Microsoft Translator). To our knowledge, these models are among the strongest and most representative of their respective categories. In particular, NLLB and MS Translator have already employed techniques such as back-translation and pretraining, and NLLB remains the state-of-the-art model on the Flores datasets.

Moreover, our SCALE framework is not restricted to certain translation models. We believe that SCALE would also benefit from other strong baseline models by employing it as STM. If you could provide any additional models you believe we overlooked, we would greatly appreciate it. This would allow us to further validate and enhance the applicability of our SCALE framework.

Concern #3: Inference cost.

Response: We appreciate your valid concerns regarding the computational cost associated with the inference process of the SCALE model.

In Section 5.4 of our paper, we have indeed addressed this concern. We have considered two additional costs brought about by SCALE: the STM decoding and the extended context for LLM inference. Our conclusion is that the latency incurred can be primarily attributed to the extended context window due to the transformer's quadratic time complexity. For more details, we kindly refer you back to our paper.