Language Imbalance Driven Rewarding for Multilingual Self-improving

Q2: Does the approach for weaker models like Llama-2-7B and lower-resource langauges?

In response to the reviewer's inquiry, our experiments demonstrate that the proposed approach does indeed work for both weaker models (like Llama-2-7B) and lower-resource languages (Swahili, Thai, and Bengali). Specifically:

The Supplementary experiments on the weaker model, Llama-2-7b-Chat as the base model.

Table n3: The X-AlpacaEval Leaderboard evaluated by the latest GPT-4o.

Table n3 demonstrates that even when starting with a model with weaker multilingual capabilities, such as Llama2-7B-Chat, which exhibits extremely low performance in languages like Russian (ru), German (de), and French (fr) on the X-AlpacaEval, significant improvements can be achieved.

By leveraging language imbalance driven rewarding for self-multilingual optimization across two iterations, the model shows substantial enhancement across all training languages, particularly in those where the original model's performance was initially weaker.

The Supplementary experiments on lower resource languages (bn, th, sw), Llama-3-8b-Instruct as base model.

Detailed results and analysis for these languages can be found in W2.

We are more than willing to continue engaging in in-depth discussions. If you have any further questions, please feel free to comment at any time.

Thanks for your efforts to provide insightful comments. We will address your concerns point by point.

W1 & Q1: The concern about the evaluation's heavy reliance on GPT-4 for assessing response quality

Please see General Response about GPT-4 on Cross-Language Evaluation Bias.

W2: The comparison between self-translate and translation systems using low-resource languages

Thanks for your insightful question! Below, we address why our method prioritizes self-translation and when external translation systems could be considered.

Perspective 1: Why do we prioritize self-translation over external translation systems?

• Self-Improving Paradigm: Our method is designed as a self-improving paradigm, where the model iteratively refines its capabilities. Introducing external translation systems disrupts this iterative cycle, making it harder to achieve the intended self-improvement and model autonomy.

• Consistency: Self-translation aligns closely with the model's inherent generation distribution. This consistency prevents collapse during DPO training, as the model is optimizing against outputs it understands well. In contrast, external translation systems produce distributions that differ from the model's original generation distribution, which may hinder effective training.

• Efficiency: Self-translation eliminates the integration complexity associated with third-party systems, making the approach more efficient and streamlined for implementation.

Perspective 2: When do we use external translation systems?

The primary goal of using self-translation is to preserve the integrity of the self-improving paradigm. However, in cases where the model's generation capabilities are particularly limited for certain low-resource languages, relaxing this constraint and using an external translator is also a viable solution.

In such scenarios, external translation systems can be leveraged for mutual translation between dominant and low-resource languages to bootstrap performance. However, this would be a complementary strategy rather than the core of our proposed approach.

Perspective 3: External Translation Systems as a Complementary Strategy for Low-Resource Languages.

It is important to note that the effectiveness of post-training on these low-resource languages is inherently limited. The model's multilingual capabilities are primarily developed during the pre-training phase, where it learns from a diverse and extensive multilingual corpus. As such, the gains from post-training are incremental and cannot fully overcome the limitations of the pre-training data for low-resource languages.

Llama3-8b-Instruct demonstrates weak performance in low-resource languages such as Swahili, Thai, and Bengali. To assess the impact of post-training on these languages, we conducted experiments using Llama3-8b-Instruct as the base model, comparing the results with those from the external translation system.

Table n1: The X-AlpacaEval Leaderboard evaluated by the latest GPT-4o, with external Google translate system.

Table n2: The X-AlpacaEval Leaderboard evaluated by the latest GPT-4o, with model performing self-translation.

Table n1 and Table n2 present the results for Google Translate and self-translation, respectively. We summarize the following three points:

1. Even though the model performs weakly in these languages, our approach remains effective in low-resource settings and can iteratively improve the model's performance across all languages.
2. Self-translation more effectively improves the performance of English because it avoids introducing external translations, maintaining a consistent generation space. This prevents disruption to English's established capabilities, leading to better performance improvements.
3. The external Google translation system provides higher-quality data for low-resource languages, enhancing the model's capabilities in these languages during optimization due to the model's initially weaker generation capabilities in these languages.

By prioritizing self-translation while leaving room for the targeted use of external translation systems in extreme cases, our method aims to balance robustness, efficiency, and the ideal self-improving paradigm.

Dear reviewer n8tX,

As the discussion phase is nearing its conclusion, we would like to know if our responses have addressed your concerns. Looking forward to further discussion with you. We are happy to address any questions if such occur. Wish you have a nice day!

Best regards,

Authors

Thanks for your insightful questions and we believe they hold significant value for our work. We try to resolve your concerns below.

W1: The quality of GPT-as-a-judge in different languages

Please refer to the General Response about GPT-4 on Cross-Language Evaluation Bias for further details.

W2: The explanation of the benefits of incorporating negative samples to construct English preference pairs for alignment.

In the preference data for English, we construct preference pairs by randomly sampling responses from other languages and translating them into English to serve as negative samples. We argue that using negative samples to build preference pairs for DPO training helps the model penalize the generation of negative samples, a process similar to machine unlearning [r1]. This approach also leads to improved model performance.

Experimental results in Figure 2 and Table 4 of the manuscript, focusing on the dominant language, show that incorporating negative samples in preference pair construction significantly improves the model's performance. This finding aligns with concurrent work presented at EMNLP 2024 [r2], which also demonstrates that alignment can be achieved using solely negative samples.

W3: The concerns about "translationese bias" in multilingual benchmarks evaluation

The use of machine-translated benchmarks can sometimes introduce artifacts commonly associated with "translationese", such as simplified syntax, lexical choices closer to the source language, or other translation-induced patterns. However, our approach leverages LLMs for self-translation to construct training data, which offers key advantages:

• Different Data Distributions: Our method uses LLM self-translation to construct training data, while multilingual benchmarks are derived from machine translation of English datasets. This ensures that the training data and benchmark data have different distributions, effectively minimizing the risk of translationese bias influencing performance.

• Reduction of Translationese Artifacts: LLM self-translation significantly reduces translationese effects, producing fluent and natural translations that align closely with native text. This is supported by prior works ([r3, r4]), which highlight the high-quality outputs of LLMs.

• Robustness Across Tasks: The method's consistent improvements across open-ended (e.g., X-AlpacaEval, MTBench, and Reasoning) tasks and NLU (e.g., MMLU, HellaSwag, ARC challenge, and TruthfulQA) tasks demonstrate genuine multilingual capability rather than reliance on translation-specific patterns.

Q1: Which language is translated into English

To ensure the diversity of negative samples in English preference pairs, we randomly sample responses from non-dominant languages and translate them into English.

Q2: Why would arithmetic reasoning be a separate section/experiment?

1. By separating this experiment, we aim to evaluate the model's performance on the reasoning task, providing a clearer understanding of its strengths and limitations beyond the general instruction-following task. This separation allows for a more targeted analysis of how the "Language Imbalance Driven Rewarding" method performs on arithmetic reasoning task, which may differ from general instruction following task.

2. The dataset, evaluation, and metrics used in the Arithmetic Reasoning task differ significantly from those in the General Instruction-following task. To make the experimental section clearer, we have divided the experiments into two separate sections.

Q3: Typo Error in Eq(7).

Thank you for pointing out the error in the formula notation. We have corrected these in the revised version.

We hope the above clarifications have enhanced your understanding of our work. We appreciate your valuable feedback and are eager to discuss any additional questions or comments further.

[r1] Zhang, Ruiqi, et al. "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404.05868 (2024).

[r2] Duan, Shitong, et al. "Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

[r3] Chen, Pinzhen, et al. "Iterative translation refinement with large language models." arXiv preprint arXiv:2306.03856 (2023).

[r4] Kunilovskaya, Maria, et al. "Mitigating Translationese with GPT-4: Strategies and Performance." Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1). 2024.

Dear reviewer w67r,

As the discussion phase is nearing its conclusion, we would like to know if our responses have addressed your concerns. Looking forward to further discussion with you. We are happy to address any questions if such occur. Wish you have a nice day!

Best regards,

Authors

Thank you to the authors for additional information. I have carefully checked your response to my review as well as other reviewers' comments. Below are my thoughts.

• W1: it is not addressed because my concern is that LLM-as-a-judge cannot score different languages with the same "standard" or even score range. E.g. for the same high-quality response/content, you may get an 8 for English and a 7 for Chinese, just due to the judge's score bias. This is also raised by Reviewer n8tX. I think your current practice of adding more judges cannot prove or disprove this. Perhaps one thing you could consider is that:
  • 1. for the same question in different languages, create the same response/content but in different languages. Note that these questions should test for objective and factual knowledge and cannot have language/culture-dependent answers (you may need more constraints too to carefully control this)
  • 2. show that judges can award the same scores to those responses in different languages.

• W2, W3, Q1, and Q3 are addressed.

• Q2: I still think the writing structure is weird but I respect your choice.

• Novelty: The other two reviewers raised issues with originality. This is not the first paper that exploits/explores language capability differences as a reward signal, but I think looking at the intrinsic capability difference is still a new method. Whilst certainly not plagiarism, authors should discuss other works more thoroughly.

We would like to emphasize that the focus of our method is multilingual self-rewarding that self-improves the general multilingual capability without external resources. In Lines 53-55, we explicitly mention that our work is primarily inspired by Self-rewarding (Yuan et al., ICML 2024). Table r1 provides a detailed comparison, highlighting the potential similarities and differences between our work, Self-rewarding, and MAPO (She et al., ACL 2024).

Table r1: The relationship between our approach, Self-rewarding, and MAPO.

As shown in Table r1, our approach bears more resemblance to Self-rewarding (highlighted in bold), including the task we focused on, the training method, preference data form, and the self-improving paradigm. Therefore, we mainly contextualized our approach with LLM self-improving methods in related works. In contrast, the only similarity between our work and MAPO is that we all experimented on multilingual mathematical reasoning task (the italicized text).

Regarding the reviewer’s comment on the similarity between our approach and MAPO, particularly concerning the use of MT for building preference data and iterative DPO for optimization, we believe there are key differences that set our work apart.

1. MT utilization: Our work only leverages the LLM itself to translate responses across languages, while MAPO employs an external MT model as a reward model.

2. Preference pair construction:

   a) In our work, preference pairs are constructed based on language imbalance—specifically, the prior that dominant languages generally outperform non-dominant ones. This contrasts with MAPO, which does not directly construct preference pairs. Instead, MAPO first employs an MT model to assess the consistency between reasoning paths generated in different languages, where the MT model functions as a reward model. The reward score derived from this consistency assessment is then used to construct preference pairs.

   b) Our approach could construct preference pairs for both dominate and non-dominate languages, while MAPO can only derive reward signal for non-dominate ones.

3. Iterative DPO training: As shown in Table r1, we draw inspiration from the Self-rewarding method, which constructs preference pairs without external supervision and only adopts iterative DPO training. MAPO, on the other hand, employs both PPO and DPO training in their experiment, as they can obtain a reward score ('alignment' score) for each response using the external MT model.

In conclusion, we believe the similarity in Table r1 and the distinctions listed above make it clear that our approach is directly inspired by Self-rewarding instead of MAPO.

We did not include MAPO as a major related work, since it is designed for only reasoning tasks rather than general ones such as instruction following tasks. This is because off-the-shelf translators fail to assess consistency between LLM responses due to their limited context size. Besides, it does not completely obey the self-improving paradigm, due to the use of the external translator.

To clear up the misunderstandings, we will include a detailed discussion in the revised version on the relationship between our approach, Self-rewarding, and MAPO, emphasizing their major differences.

We hope the evidence provided above could address your concerns. We would greatly appreciate it if you could kindly reconsider our work with these distinctions in mind.

We sincerely appreciate your questions. We highly value your feedback and provide detailed explanations to address your concerns.

Q1: Explanation and Analysis of the relation between Language performance and reward accuracy.

In Lines 164-165, we claimed that the insight behind our work is Language Imbalance Driven Rewarding: The greater the disparity between the capabilities of the dominant and non-dominant languages, the stronger the reward signal, which subsequently leads to higher reward accuracy.

Therefore, (es, fr) are well represented in the training data, resulting in a narrow performance gap with English in preference pairs. As a result, the reward signal becomes weaker, leading to lower reward accuracy.

Here is a more detailed analysis, corresponding to Section 3 of the manuscript：

In Table 1, we can observe that the GPT-4 score for responses in (es, fr, it) are noticeably higher compared to those in (de, ja, ru), indicating that the former languages are well represented in the training data.

In Table 2, we can clearly observe that within the group (es, fr, it), the gap between the responses for "Self Generation" (the model generates responses in non-dominant languages) and "Self Translation" (the model translates English responses into non-dominant languages) is significantly smaller compared to the group (de, ja, ru). This indicates that the GPT-4 score gap between (es, fr, it) and English is narrower, suggesting a weaker reward signal, which results in lower reward accuracy.

The above analysis is consistent with the results in Table 3, where we directly obtain the reward accuracy of multilingual preference pairs. The model's weaker performance on (de, ja, ru) creates a large gap with English performance, which results in higher accuracy in preference pairs where English responses are chosen. In contrast, its stronger performance on (es, fr, it) creates a smaller gap with English performance, leading to lower reward accuracy in preference pairs where English responses are chosen.

We hope the above explanations have enhanced your understanding of our work. We appreciate your valuable feedback and are eager to further discuss any additional questions.

I do recognize the paper has its own merit, for example, in designing the reward signal and using the model itself as the source of preference. However, I don't think it is proper to regard it as a complete separate work from She et al, 2024.

For the record, the authors do not mentioned the technical similarity to She et al., 2024 throughout the paper. The only mention is "existing LLMs demonstrate inconsistent reasoning capabilities across different languages (She et al., 2024)." I don't think this is an acceptable practice.

As additional comments to the authors in the above reply:

1. "Ours" uses iterative DPO, "MAPO" uses Iterative DPO or PPO.

2. "Ours" uses preference pairs, "MAPO" uses preference pairs for the DPO training, but use preference score for PPO training.

How could the above two points been treated as a technical difference between the two work? The attempts in this paper is just a subset of the cited paper.

In my opinion, the differences in techniques is not directly related to the domain or tasks. There are many research in the general task/domain that are inspired by previous attempts in a certain task/domain, or vice versa.

Even for the technical part, the proposed work and She et al., 2024 both use translation as a bridge between the results of the dominant language and non-dominant language, though in slightly different way. I do suggest a comparison between the two, which might bring new insight in utilizing the translation knowledge.

Thank you for your response. In our initial response, we provide Table r1 to clarify that our approach is directly inspired by Self-rewarding instead of MAPO, to avoid potential concern regarding plagiarism.

After reading your response, we believe there are still misunderstandings to be resolved:

The differences in constructing preference pairs and the utilization of DPO

The purpose of Table r1 is to compare our approach with Self-Rewarding and MAPO, highlighting the source of our inspiration. While MAPO also conducts experiments with iterative DPO training using constructed preference pairs, their pairs are derived from reward scores generated by a reward model. In contrast, our approach constructs preference pairs directly based on the prior of language imbalance, which is a key difference in our methodology.

Ideally, a reward model could always be used to score responses and create preference pairs. When viewed solely from this perspective, all DPO-related works could technically be considered a subset of PPO works. However, we contend that the key consideration should not be this theoretical hierarchy but rather the rationale behind selecting a specific preference optimization method for a given approach.

From this standpoint, our method is clearly off-policy, as we train directly on self-translated responses that are not sampled from the model's output distribution for the given instruction. Consequently, our approach exclusively employs DPO training. In contrast, MAPO is on-policy, as it scores sampled responses directly and trains on them. MAPO's use of DPO is a trade-off aimed at improving training efficiency rather than a fundamental aspect of its methodology.

Translation as a bridge in cross-lingual alignment.

Regarding the usage of the MT technique. We mainly utilized MT to translate responses across languages. Such usage is similar to previous cross-lingual alignment works [r1, r2], where the translated responses are utilized to further fine-tune the model. In contrast, MAPO only uses the MT model as a reward model, they do not actually generate translations and use them for model training. Therefore, we believe there is a huge difference in the utilization of MT between our work and MAPO.

The discussion about MAPO has been added to the revised paper.

We thank the reviewer for pointing out this issue. In the revised version of our manuscript, we have added detailed comparisons and discussions (highlighted in blue) to provide a more thorough analysis of the method and contributions of MAPO.

We hope the above explanations will help you better understand our work. We are happy to communicate with you further for any questions to enhance our manuscript.

[r1] Zhang, Yuanchi, et al. "Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages." arXiv preprint arXiv:2402.12204 (2024).

[r2] Chen, Nuo, et al. "Breaking language barriers in multilingual mathematical reasoning: Insights and observations." arXiv preprint arXiv:2310.20246 (2023).

Dear reviewer AMKf,

As the discussion phase is nearing its conclusion, we would like to know if our responses have addressed your concerns. Looking forward to further discussion with you. We are happy to address any questions if such occur. Wish you have a nice day!

Best wishes!

We have further discussed the issue of Language Bias in LLM-as-a-Judge in our approach with Reviewer w67r, and have updated the results of this discussion in the general response.

In Table 1 of the manuscript, we evaluate the GPT score for individual responses in different languages, e.g., $y^{en}$ or $y^{ru}$. This approach may introduce language bias, as scores could vary depending on the language in which the response is generated. To mitigate this potential issue, we perform more detailed pairwise comparisons within the same language in Tables 2 and 3 of the manuscript, thereby avoiding cross-lingual scoring bias. Importantly, this language bias in Table 1 does not affect our conclusion, as we focus on constructing pairwise comparisons between responses within the same language for Tables 2 and 3.

Specifically, in each column of Table 2, both responses are in the same language. Take 'ru' as an example. The 'Self Generation' GPT score is calculated on $y^{ru}$ and the 'Self Translation' GPT score is calculated on $y^{en \to ru}$. As both responses are in 'ru', it does not introduce potential language bias of LLM-as-a-Judge, providing a fair comparison. Similarly, in Table 3, the reward accuracy is evaluated on preference pairs consisting of $y^{ru}$ and $y^{en \to ru}$, which also avoids the cross-lingual comparison issue.

Both Tables 2 and 3 evaluate GPT score based on pairwise comparisons within the same language. This methodology inherently avoids the cross-lingual comparison issue, ensuring a fairer and more consistent assessment since all evaluations are conducted within the same language.

Summary:

• In Table 1, GPT scores are assigned to individual responses in different languages, this could introduce language bias. However, this does not affect our conclusion.
• In Tables 2 and 3, pairwise comparisons are conducted within the same language, making the evaluation fairer and more consistent.
• By translating responses into the same language, we create preference pairs that avoid language-specific biases and focus on the relative quality of the responses.

Our approach, using the LLM-as-a-judge, ensures that the evaluation of responses is consistent and unbiased, as all comparisons are done within the same language.