Lens: Rethinking Multilingual Enhancement for Large Language Models

We deeply appreciate your valuable comments and your recognition of our contributions to enhancing multilingual performance in large language models. Below, we address the raised concerns point by point.

Weakness 1: The technique primarily builds on existing methods and lacks significant novelty.

Thank you for bringing this up. We would like to clarify that our work differs in both ideas and methodology from the referenced paper.

The referenced work focuses on improving cross-lingual performance through token-level and semantic-level alignment among different languages, an idea widely adopted by many current methods. This idea motivates part of our approach as well, specifically within the language-agnostic subspace to align different languages.

However, our work goes a step further by addressing a critical but underexplored aspect: separating language-specific representations within the model’s language-specific subspace. This highlights the importance of simultaneously aligning and separating representations across languages for effective and efficiency multilingual enhancement. The core novelty of our work lies in identifying this dual requirement and demonstrating its efficacy through extensive experiments. This idea and our methodological innovation have also been positively acknowledged by Reviewer U88a and Reviewer CKFG.

We appreciate your suggestion and will cite the referenced work in our revised manuscript while further elaborating on the distinctiveness and contribution of our work.

Weakness 2: Language selection for experiments could have accounted for the proximity of each language to the central language, which would add meaningful insights.

Thank you for pointing this out. The language selection in our experiments is now based on two principles:

• The target languages should be classified as out-of-scope in the official model card of the base model, ensuring that our experiments address under-represented language cases (lines 263 - 265).

• The selection should reflect a balance of diverse linguistic families and resource levels, allowing us to evaluate performance across a broad spectrum of languages (lines 259 - 262).

And the six languages we chose (Bengali, Swahili, Chinese, Japanese, Korean, and Arabic) also represent varying degrees of linguistic proximity to English, ranked approximately from closest to farthest: Bn, Sw, Zh, Jp, Ko, and Ar.

We greatly appreciate your suggestion and have conducted additional experiments on LLaMA-3-8B-Instruct with Spanish (Es), German (De), and French (Fr)—languages that are more linguistically closer to English, in terms of both multilingual understanding (MU) and generation (MG).

These new results provide further insights and show that our method consistently improves multilingual performance across languages, regardless of their distance from English. We will include these findings in the revised manuscript for completeness and to better address this concern.

Weakness 3: Performance inconsistencies are observed, such as the inferior results for the Phi dataset compared to llama's performance in the appendix.

Thank you for highlighting this point. We believe the observed inconsistencies may stem from differences in the underlying capabilities of the Phi and LLaMA model families. However, as they do not provide detailed training or dataset documentation, it is challenging to conduct a deeper analysis or pinpoint the exact causes of these differences.

Despite this, when evaluating the multilingual enhancement effects, our Lens consistently improves performance on both model families and outperforms current baseline methods. This demonstrates the scalability and robustness of our approach across different pretrained backbones.

We appreciate your suggestion and will emphasize this point more clearly in the revised manuscript to address this concern.

Question 1: Since the method doesn't leverage external data to augment knowledge, wouldn't languages closer to English theoretically benefit more? It raises the question: does the "alignment" operation inherently favor languages that are closer to or in the same family as English?

Thank you for raising this concern. We would like to clarify that our method is not inherently biased toward languages closer to English.

• First, the six target languages selected in our study are intentionally distant from English, ensuring a focus on languages that typically are under-represented in LLMs. Our experimental results demonstrate effective performance improvements for these languages, validating the method’s applicability even for linguistically distant languages.

• Second, as highlighted in our response to Weakness 2, we conduct additional experiments on linguistically closer languages—Spanish, German, and French. These results further confirm that our method achieves meaningful improvements regardless of the language’s proximity to English.

• Finally, we show in this paper that ’’alignment’’ alone is insufficient to achieve comprehensive multilingual enhancement. Instead, our approach combines alignment with ’’separation’’ operations in the language-specific subspace, a critical insight provided by this work. By leveraging this dual mechanism, our proposed Lens effectively enhances multilingual capabilities without being biased toward languages closer to English.

Question 2: Why do the results in Figure 2 show greater improvement for Chinese and Japanese, but limited improvement for Arabic and Bengali?

Thank you for raising this excellent question.

Our hypothesis is that it stems from the imbalanced proportions of languages in the pretraining corpus of the backbone, which result in varying representation capabilities across languages. Unfortunately, as LLaMA-3 only reports that approximately 90% of its pretraining data is English, with no detailed breakdown of the remaining 10%, it is challenging to verify this hypothesis with certainty.

However, based on general resource availability, we infer that the remaining 10% pretraining data likely favors high-resource languages such as Chinese and Japanese over mid-resource languages like Arabic or low-resource ones like Bengali. Consequently, our method’s relative improvement may be influenced by the uneven representation capability in the pretrained backbone. Despite this, our results demonstrate consistent performance gains across all languages compared to the backbone model, highlighting the robustness of our approach.

We deeply appreciate your question and agree that this is an important topic for further investigation. We hope that future multilingual LLMs with more transparent pretraining corpus details will provide better support for understanding such disparities and refining enhancement techniques.

We hope these clarifications address your concerns adequately. Thank you once again for your detailed and thoughtful feedback, which has been invaluable in refining our work.

Weakness 2: The results of the ablation study do not fully support the authors' claims.

We apologize for the unclear presentation of Figure 3, which may have led to a misunderstanding of the experimental results.

Regarding the “pull the target language representations closer to those of English”, it indeed helps to inherit the central language’s capabilities and contributes to multilingual performance enhancement. Specifically, as the hyperparameter increases from 0 to 1, the MG performance improves from 5.61 to 5.77, and the average MU performance improves from 73.11 to 73.67. However, this is only part of our claim.

Another critical claim in this work is the importance of separating the central and target language representations in the language-specific subspace (lines 413 - 414). This aspect has been overlooked in previous studies that almost all of them just focus on aligning representations from different languages (lines 86 - 90 and 416 - 419). Together, these two claims, which are all verified by our experimental results (in Figure 3), lead to our core conclusion: effective multilingual enhancement requires simultaneously aligning and separating representations across languages.

We sincerely thank you for pointing out this issue, as it highlights the need to present Figure 3 more clearly and ensure a stronger emphasis on our core claim in the revised manuscript. Your feedback is greatly appreciated.

Weakness 3: Incorrect Color in Table1.

We sincerely apologize for the confusion caused by the incorrect highlight of SDRRL’s performance on the COPA dataset in Table 1. We have corrected this to reflect the results more accurately in current revised version.

Regarding the use of green for comparable results, our marking principle is that if the performance drop in central language (English) is within 0.5 points, it is considered acceptable and thus marked in green. We apologize for not making this clearer, which may have led to misunderstandings.

As you acknowledge in the Strengths, our Lens demonstrates significant improvements over baselines, both in enhancing the target language’s capabilities and in maintaining the central language’s performance. We have made further adjustments in the presentation of the results in Table 1 and Table 4 to ensure clarity and avoid any misinterpretation. Thank you for your constructive feedback.

Question 1: Consider adding an explanation for what the bold and underlined text indicates in the caption of Table 1.

Thank you for pointing this out. The bold and underlined text in Table 1 indicates the best and second-best results in the comparison with the baseline methods, respectively. We have added an explanation of this in the revised manuscript in Table 1 and Table 4, which is highlight in blue .

Question 2: The caption for Figure3 (Line408) contains an error, please correct it.

We appreciate you highlighting the caption error for Figure 3. We have corrected this error (MU to MG in line 408, highlighted in blue) to ensure accurate description of the figure.

We hope these clarifications address your concerns adequately. Thank you once again for your detailed and thoughtful feedback, which has been invaluable in refining our work.

Thank you for your valuable feedback on our paper. We appreciate the recognition of resource-efficiency, competitive performance, and good interpretability of our work. We address your concerns and questions as follows.

Weakness 1: Missing Reference: Previous work has explored how to enhance multilingual abilities through aligning internal sentence representations, but there is a lack of detailed introduction to this relevant research.

Thank you for pointing this out. However, we believe there may have been a misunderstanding. In fact, we have already cited the mentioned work (lines 648 – 651) in the Related Work section (line 128) and discussed its limitations in the Experiment section (lines 416 - 418) of our paper. Since it shares a similar idea with SDRRL and QAlign in aligning internal sentence representations, we did not reproduce it separately in our original submission.

Following your suggestion, we have now included reproduction results for this method across three backbones under both bilingual and multilingual enhancement settings in terms of multilingual understanding (MU) and multilingual generation (MG), as shown in the table below (CLA is the baseline method you mention). The results indicate that it still struggles to effectively enhance target language performance while maintaining central language performance.

LLaMA-3-8B-Instruct

• Bilingual (En, Zh)

• Multilingual (En, Zh, Jp, Ar, Ko, Sw, Bn)

LLaMA-3.1-8B-Instruct

• Bilingual (En, Zh)

• Multilingual (En, Zh, Jp, Ar, Ko, Sw, Bn)

Phi-3.5-mini-Instruct

• Bilingual (En, Zh)

• Multilingual (En, Zh, Jp, Ar, Ko, Sw, Bn)

This further reinforces a key conclusion of our paper: aligning language representations alone cannot achieve substantial performance improvements. Further, we should also enhance the separation between representations of different languages in the language-specific subspace, a point overlooked by existing works.

We appreciate your suggestion, which has prompted us to further discuss and analyze this important related work.

Question 1: How does the paper analyze typical cases for the language-agnostic and language-specific subspaces in Parts 4 and 5, and what implications do these analyses have for the performance of LENS in handling multilingual tasks?

Thank you for your insightful question regarding the analysis of language-agnostic and language-specific subspaces. We are pleased to provide further clarification and address this point in detail.

• First, in Section 5.1, we independently analyze the contributions of language-agnostic and language-specific subspaces to the improvement of multilingual capabilities. The results highlight that both subspaces contribute to enhance multilingual performance, with language-specific subspaces contributing more substantially. This finding underscores the importance of explicitly modeling language-specific properties, which has often been overlooked in previous works.

• Second, in Section 5.2, we examine the impact of manipulating language-agnostic and language-specific subspaces at different layers of the backbone. The results align with conclusions from existing model interpretability research, showing that language-specific parameters predominantly reside in the higher layers of the model. This layer-specific behavior further supports the design of LENS and its focus on top-layer operation.

• Finally, in Section 5.4, we provide a visualization of the representations. The results show that representations of different languages tend to cluster more tightly within the language-agnostic subspace while being more dispersed in the language-specific subspace. This visualization effectively demonstrates how LENS balances shared semantic alignment with the preservation of language-specific distinctions.

Once again, we sincerely thank you for this question, which allows us to elaborate further on these analyses. We will ensure these points are more explicitly highlighted in the revised manuscript.

Question 2: In Figure 2, why does LENS not perform better than xSFT-Full for the Swahili language, and what factors might contribute to xSFT-Full's superior performance in this specific case?

Thank you for raising this important observation. We believe the performance gap for Swahili may be attributed to the uneven quality of the training data used in our experiments. Specifically, the Bactrian-X dataset used for training derives its input from Google Translate and its output from GPT-3.5-turbo, meaning the dataset quality depends heavily on these two sources. As a result, inconsistencies in translation and generation quality can introduce noise, leading to uneven performance gains from data-driven post-training approaches like xSFT-Full. This highlights one of the key limitations of the current data-driven paradigms.

In contrast, LENS seeks supervision signals internally from the backbone itself, bypassing the need for extensive reliance on potentially noisy external datasets. This intrinsic approach allows LENS to achieve consistent improvements over the backbone model across a wide range of languages, demonstrating better scalability and robustness. We have also demonstrated this phenomenon in our experiments, showcasing LENS’s broader applicability. In our future work, we propose combining the LENS training paradigm with advancements in data selection and filtering methods. We believe this hybrid approach holds great potential for further enhancing multilingual performance.

Once again, thank you for your thoughtful question. We will emphasize this point more explicitly in the revised manuscript.

We hope these clarifications address your concerns adequately. Thank you once again for your detailed and thoughtful feedback, which has been invaluable in refining our work.

We sincerely thank you for your insightful feedback and encouraging comments on our paper. We appreciate the recognition of the strengths of our approach, particularly the novelty of the methodology, its efficiency and effectiveness, the preservation of central language abilities, the comprehensive improvement across multilingual tasks, and the transparency and interpretability of the model. We will now address your concerns point by point.

Weakness 1: The approach may encounter challenges when dealing with languages that have unique grammatical structures or vocabularies significantly divergent from the central language.

Thank you for raising this important point. We would like to clarify that:

• The languages selected for enhancement in our study, particularly Chinese, Japanese, Korean, and Arabic, are structurally distant from the central language (English). Despite these linguistic divergences, our experimental results demonstrate that LENS effectively improves the performance for these languages, highlighting its robustness across diverse linguistic structures.

• Additionally, our evaluation benchmarks encompass a variety of challenging tasks, including multilingual commonsense reasoning, multilingual world knowledge, multilingual multi-turn instruction following. These tasks are designed to assess the model’s deep understanding of different languages, providing strong evidence of LENS’s capability to handle linguistic diversity.

We sincerely appreciate your suggestion and will emphasize this point more clearly in the revised manuscript.

Weakness 2: Although LENS improves multilingual performance by manipulating language representations, it might not fully address the complexities of tasks like machine translation, especially for low-resource languages.

Thank you for your thoughtful feedback and for highlighting the importance of machine translation as a benchmark for multilingual capabilities. In response to your suggestion, we have supplemented our experiments with evaluations on the FLORES-101 dataset [1]. Specifically, we assess the bidirectional translation performance between the target language and English, reporting scores using the COMET metric with the WMT22-comet-da model [2].

• X to En

• En to X

The experimental results demonstrate that LENS still effectively enhances the multilingual machine translation performance, further validating its robustness across diverse multilingual tasks.

We sincerely appreciate your valuable suggestion, which has enriched and completed our evaluation framework. This addition provides a more comprehensive demonstration of LENS’s effectiveness.

References:

[1] Goyal N, Gao C, Chaudhary V, et al. The flores-101 evaluation benchmark for low-resource and multilingual machine translation[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 522-538.

[2] Rei R, De Souza J G C, Alves D, et al. COMET-22: Unbabel-IST 2022 submission for the metrics shared task[C]//Proceedings of the Seventh Conference on Machine Translation (WMT). 2022: 578-585.

Weakness 2: I still recommend that the authors compare SFT based on the LoRA version.

Thank you for recommending the comparison with LoRA-based SFT. We acknowledge the importance of this perspective and here are the additional results under both bilingual and multilingual enhancement settings on all three backbones in terms of multilingual understanding (MU) and multilingual generation (MG) performance.

LLaMA-3-8B-Instruct

• Bilingual

• Multilingual

LLaMA-3.1-Instruct-8B

• Bilingual

• Multilingual

Phi-3.5-mini-Instruct

• Bilingual

• Multilingual

Based on our experimental results, we derived the following key conclusions:

• For preserving the central language’s capabilities, incorporating LoRA-based SFT is indeed more effective at preventing catastrophic forgetting than its full-parameter conterpart. However, it primarily protects multilingual understanding (MU) tasks while multilingual generation (MG) capabilities are also significantly affected.

• For target language enhancement, LoRA-based methods also show a trend for improving MU tasks over MG tasks.

• By contrast, our proposed Lens achieves a more comprehensive performance, simultaneously enhancing understanding and generation for target languages while maintaining both the understanding and generation capabilities of the central language across different base models.

We appreciate your suggestion and will include the above experimental results and discussion in the revised manuscript to provide more empirical insights.

We hope these clarifications address your concerns. Thank you once again for your detailed and thoughtful feedback, which has been invaluable in refining our work.

Thank you for your valuable feedback on our paper. We appreciate the recognition of novelty of our method and its potential impact on the multilingual research community. We address your concerns and questions as follows.

Weakness 1: I am curious about the extent of improvement the method proposed in the paper can offer when the representation space of LLMs for languages within the same language family as English is closer to that of English, such as Es, Fr, and De.

Thank you for this insightful suggestion.

In our work, we deliberately focused on languages that are currently more under-represented in existing LLMs to highlight the robustness of our method in enhancing multilingual capabilities across a diverse range of linguistic characteristics. We agree that evaluating languages within the same family as English, such as Spanish (Es), French (Fr), and German (De), is also valuable.

Here are our supplemented experimental results based on LLaMA-3-8B-Instruct, where these 3 languages are also not supported according to its official model card [1]. For Multilingual Understanding (MU) evaluation, we adopt M-MMLU dataset which covers all 4 languages En, Es, Fr and De. And Multilingual Generation (MG) evaluation is performed on MT-Bench.

The results show that our Lens achieves comparable improvements for these languages as well. We will include these results in the revised version to provide a more comprehensive evaluation.

Reference:

[1] https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

Weakness 3: Some explanatory text needs to be added to explain the author's motivations and make it easier for readers to read. For examples, the function Span() in Equation (2)，the design of Equation (3).

We agree that some equations could benefit from clearer explanations to enhance readability. Specifically:

• For Equation (2), in linear algebra, Span() refers to the columns of a matrix that represents all possible linear combinations of those vectors. This concept is commonly used to describe subspaces. Here the constraints indicates that our language-agnostic and -specific subspace must be orthogonal to each other.

• For Equation (3), it aims to identify a direction of language expression within the language-specific subspace, ensuring each target language could be effectively expressed along this direction.

These additions will make the underlying motivations more transparent to the reader.

Weakness 4: Typo: line 1009， “batchsize” -> “batch size”.

Thank you for pointing out the typo in line 1009. We have corrected batchsize to batch size in the revised version, which is highlighted in orange.

Question 1: How different data sizes affect Lens, and how much improvement can be achieved using more training data.

Thank you for this insightful question. To address this, we investigated the impact of varying training data sizes (from 50 to 1,000) on the performance of LENS.

The results indicate that increasing the amount of training data leads to diminishing returns for LENS, a trend consistent with the observations for xSFT and xSFT-Full. This finding reinforces our claim (lines 361 - 367) that for extensively pre-trained LLMs such as LLaMA-3 (trained on over 15T tokens), over-reliance on more training data falls short of meeting scalability needs. Instead of focusing on larger training datasets, it is more critical to identify supervision signals that are both reliable and scalable. This directly motivates us to seek internal supervision from the central language with the backbone itself. And we hope that LENS inspires future research to explore more efficient, scalable, and automated supervision signals for multilingual enhancement of state-of-the-art LLMs.

Once again, thank you for your thoughtful question. We will further emphasize this point in the revised manuscript.