When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners

Thank you for your thoughtful feedback. Below we address your concerns point by point.

---

W1.1: its additional computation could make it less practical

Thank you for your feedback. Regarding the concern on computational overhead, we conducted a speed test on the MGSM dataset using DeepSeek-R1-Distill-Qwen-7B on a single NVIDIA A100 GPU.

The results show that our method causes minimal slowdown in inference speed.

Importantly, the intervention itself involves only a lightweight projection and addition operation, which is negligible compared to the total cost of running the LLM. The slight speed drop mainly comes from our vLLM-based implementation, not from the method itself.

We are actively working on optimizing the code to further reduce overhead and bring the inference speed closer to the original model, enhancing the method’s practicality.

---

W1.2: post-training integration

We have discussed the potential integration of our language–reasoning decoupling approach into training in Appendix A (Lines 640–646). For example, using the ablated representation as a supervision signal could encourage language-invariant intermediate representations during supervised fine-tuning or reinforcement learning, potentially improving generalization.

However, implementing and validating this at training time is beyond the scope of this paper and is left as future work.

---

W2: Dataset difficulty and translation quality

Thank you for the thoughtful observation. We agree that some of the datasets used, such as MGSM, are relatively easy for modern LLMs.

Translation quality is also a valid concern. To address this, we introduced PolyMath [1], a human-verified multilingual dataset. For each language, we sampled 50 questions from each of three difficulty levels (low, medium, high) and computed a difficulty-weighted score.

• DeepSeek-R1-Distill-Qwen-7B

• Qwen2.5-7b-Instruct

Experiments on DeepSeek-R1-Distill-Qwen-7B and Qwen-2.5-Instruct-7B show that, while higher-quality and balanced datasets reduce the performance gap between English and other high-resource languages (e.g., Spanish, French), mid- and low-resource languages like Swahili and Telugu still underperform—confirming that capability gaps persist.

Our method helps close this gap, notably elevating performance in weaker languages (e.g., Russian in DeepSeek, Thai in Qwen) to levels comparable with stronger languages. This demonstrates the practical value and effectiveness of our intervention.

Reference:

[1] PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts.

---

We sincerely appreciate your comments and will revise the manuscript to better reflect these clarifications.

Thank you for your thoughtful feedback. We address each of your concerns below.

---

W1. Limited number of mid/low-resource languages

Thank you for your comment and suggestion. While we include 11 typologically diverse languages covering high-, mid-, and low-resource cases, we acknowledge that the number of mid/low-resource languages is still limited. This is mainly due to the lack of widely adopted multilingual benchmarks with sufficient coverage of truly low-resource languages. In future work, we aim to incorporate more underrepresented languages, once suitable benchmarks become available.

---

W2: Entanglement of language and knowledge (e.g., INCLUDE benchmark)

Thank you for raising this important point. To address this, we conduct additional experiments on the Regionality subset of the INCLUDE benchmark, using Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Qwen-7B. The results are summarized below:

• Qwen2.5-7B-Instruct

• DeepSeek-R1-Distill-Qwen-7B

We find that even under strong language–knowledge entanglement, our method still yields consistent improvements, suggesting that it maintains robust reasoning while preserving localized knowledge fidelity. We will include this analysis and discussion in the revised version to highlight the generality and practical value of our approach.

---

Q1: What are the higher layers used for re-injection?

Thank you for the question. The specific middle and higher layers used for ablation and re-injection vary by model. The selection is roughly based on dividing the model into three segments (lower, middle, higher), as inspired by recent interpretability studies. The exact layer ranges for each model are as follows:

---

Q2: Unexpected effects near ablation strength 0

Indeed, some models exhibit minor deviations from the overall trend when the ablation strength is near 0. Upon further analysis, we found that this is primarily due to Japanese and Thai, which behave anomalously in both performance and language fidelity in this narrow region.

We believe this is related to the surface-form similarity between Japanese/Thai and Chinese. As illustrated in Figures 4 (a)–9 (a), the lower-layer representations of Japanese and Thai are quite similar, and as shown in Figures 4 (c)–9 (c), at higher layers, both languages become closer to Chinese than to English. In practice, we observed that when slight ablation is applied, the model may even respond in Chinese to Japanese or Thai prompts, in contrast to its usual tendency to reply in English to prompts in other languages.

Our intuition is that these “neighboring” languages do not form fully independent anchors in the multilingual semantic space. As a result, small changes in ablation strength may trigger abrupt shifts in representation, leading to local irregularities in both reasoning and language output.

This highlights the complex internal dynamics of multilingual LLMs, particularly for lower-resource languages that are structurally close to high-resource ones. We plan to investigate this further in future work by analyzing language clustering in representation space and exploring language-specific or adaptive interventions to improve stability and fairness.

---

Q3: Why is preserving reasoning fidelity important?

Thank you for the question. We believe that preserving language fidelity is crucial for multilingual LLMs.

On one hand, it ensures that users can interact with the model fluently in their native language, which is essential for usability and accessibility. On the other hand, language consistency reflects the model’s ability to understand and maintain context, supporting accurate and coherent information exchange.

Therefore, we view maintaining alignment between input and output languages as a necessary goal for practical multilingual applications.

---

Thank you again for your valuable feedback and we will incorporate your suggestions to improve the paper.

Thank you for your thoughtful and detailed review. We appreciate your feedback and address each point below.

---

W1: Impact on multilingual language generation quality

Thank you for the suggestion. To address your concern, we conducted additional machine translation experiments by translating English sentences into target languages, in order to assess the impact of our method on language generation quality.

Specifically, we used the Flores-101 dataset and sampled 200 sentences per language. We evaluated ROUGE and BLEU scores on Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Qwen-7B.

• DeepSeek-R1-Distill-Qwen-7B

• Qwen2.5-7b-Instruct

Results show that our method does not degrade language generation quality in the prompted language. We will include these results in the revision.

---

W2: Causal effect of projection intervention

Thank you for this important point. We fully agree that without a random subspace control, it is difficult to establish the specificity of the language subspace used in our method.

To address this, we generated random subspaces of the same rank as our extracted multilingual subspace and replaced the original subspace in the intervention. We then evaluated the performance on MGSM using Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Qwen-7B.

• DeepSeek-R1-Distill-Qwen-7B

• Qwen2.5-7b-Instruct

The results show that with random subspaces, both performance and fidelity degrade, whereas our method yields consistent improvements. This supports a causal link between the extracted language subspace and the observed reasoning gains. We will include these control results in the revision.

---

W3: Implications for training

Thank you for the suggestion.

We have discussed the potential integration of our language–reasoning decoupling approach into training in Appendix A (Lines 640–646). For example, using the ablated representation as a supervision signal could encourage language-invariant intermediate representations during supervised fine-tuning or reinforcement learning, potentially improving generalization.

However, implementing and validating this at training time is beyond the scope of this paper and is left as future work.

---

W4: Limited scale of post-training comparisons

Thank you for the suggestion. We agree that comparing against post-training methods at larger scales would strengthen the paper. Due to limited computational resources, we were unable to run such large-scale post-training baselines.

However, as shown in Tables 1 and 2, our training-free intervention consistently improves multilingual reasoning across a wide range of model sizes, from 1.7B to 32B. These results demonstrate the scalability and effectiveness of our method, even without additional training.

---

W5: Clarification on open-source terminology

Thank you for pointing this out. We acknowledge the misuse of the term “open-source models.” In the revision, we have corrected all instances to “open-weight models” to accurately reflect that these models only release weights, not training data or pretraining details.

---

W6: Reproducibility of code

Thank you for the feedback. We apologize for the incomplete documentation.

The released code covers all core components, including vLLM-based intervention, language subspace extraction, and main evaluation scripts, and supports reproducing key results.

We are committed to improving reproducibility and will release a fully documented version shortly. Thank you again—we appreciate the suggestion and will address it promptly.

---

For each of the question you raised, we have provided point-by-point responses above. Please kindly refer to our replies to Weaknesses 1–6 for detailed clarifications and corresponding experimental or revision plans.

We truly appreciate your constructive feedback and hope our responses address your concerns.

Thank you very much for your thoughtful reply and your willingness to consider increasing the score. We truly appreciate your continued engagement. Below, we address the remaining concerns with new experiments and clarifications.

---

1. Evaluation on standard reasoning benchmarks (AIME24/25, MATH500, GPQA-Diamond)

We fully agree that evaluating on widely-used English reasoning benchmarks is important to support the general claim that our intervention improves reasoning capabilities.

To this end, we conducted additional experiments using DeepSeek-R1-Distill-Qwen-7B, applying our intervention on AIME 2024, AIME 2025, MATH500, and GPQA-Diamond. Results are shown below:

We observe that our method consistently improves performance even on these English-only benchmarks, further supporting our core claim:

Reasoning and language processing can be explicitly decoupled, and reasoning mechanisms in LLMs are largely language-agnostic.

The reason for improved performance in English, is that the language-agnostic representation space is often anchored and implicitly aligned to English, due to its dominance in training data. Our intervention encourages models to operate more directly in this aligned space, benefiting even English reasoning tasks.

We also extend our evaluation using PolyMath, PolyMath [1], a human-verified multilingual dataset. For each language, we sampled 50 questions from each of three difficulty levels (low, medium, high) and computed a difficulty-weighted score. Results again show consistent improvements across difficulty levels and languages, reinforcing the robustness and effectiveness of our approach.

• DeepSeek-R1-Distill-Qwen-7B

Reference:

[1] PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts.

---

2. Sensitivity of the mean embedding matrix to sample choice

Thank you for raising this important point. In our current setup (Appendix F, Lines 743–746), we construct the mean embedding matrix using the 7,500 training examples from MATH, translated into 10 languages via Google Translate. These samples were not handpicked, and we did not manually verify translation quality, suggesting that our method is already reasonably robust to sample noise and translation imperfections.

To directly test sensitivity, we conducted additional experiments by varying the number of samples used to construct the embedding matrix. We tested 5%, 20% and 50% examples.

The results show that performance remains relatively stable across different sample sizes, indicating that our approach is not highly sensitive to the exact sample set, and the constructed language subspace is robust and generalizable. We will include this discussion and the supporting results in the revised paper.

---

We thank you again for the thoughtful feedback—your suggestions have greatly strengthened our paper, both in scope and in evidence.

Thank you for your thoughtful and constructive review. We sincerely appreciate your feedback and address each of your concerns point by point below.

---

W1: Unclear methodology

Thank you for the valuable feedback, and we apologize for the confusion.

Your understanding is correct: ablation suppresses language signals in mid layers, while decoupling further re-injects them in higher layers. All reported main experimental results are based on decoupling, as stated in Implementation Details (Lines 172–173).

Regarding re-injection, we use a similar layer-wise rule as in Equation (3), which is used for ablation; the only difference is that we add back the language component instead of subtracting it. The exact layers for ablation and re-injection per model are listed below:

---

W2: Unsupported claim

Thank you for your detailed comments. We would like to clarify the experimental settings and improve our descriptions accordingly.

In our experiments, we roughly divide the model layers into lower, middle, and higher layers.

• In Figure 3, we apply ablation only in the middle layers.
• In Figure 4, we apply ablation only on small groups of consecutive layers.
• In Table 1 and Table 2, we apply ablation in the middle layers followed by re-injection in the higher layers—i.e., full decoupling. The exact ablation and re-injection layers for each model are listed in our response to W1.

As you correctly observed in Figure 4, ablating different layers does not always improve reasoning accuracy. Our wording here was not sufficiently rigorous. Compared to language fidelity, ablation at certain isolated layers can have unstable effects on reasoning. While Figure 3 shows that middle-layer ablation consistently boosts reasoning, Figure 4 focuses on the impact of ablation on language fidelity—especially the sharp drop when ablation is applied in higher layers.

This is precisely why our main setup performs middle-layer ablation and top-layer re-injection: to improve reasoning while preserving output fluency and consistency.

We fully agree with your point about the reasoning–fidelity trade-off. This trade-off is indeed present under ablation alone, and motivates our decoupling strategy.

Thank you again for the constructive feedback. We will revise the manuscript to make these distinctions and justifications clearer.

---

W3: Poor visualization of results

Thank you for the helpful suggestion.

We fully agree that the main tables could be made more readable and the overall effect clearer through visualizations. In the revision, we will include bar/line plots for Tables 1–3 and improve the figure captions by explicitly defining the dashed lines (which indicate baseline performance).

We appreciate your feedback and will revise accordingly.

---

W4: Misc

We agree that moving the Limitation section into the main body will improve clarity and completeness, and we will make this change in the revision.

Regarding the checklist: you are right—Appendix F does not currently include statistical significance results. This was an oversight on our part. We have conducted significance tests on the multilingual results, showing that our method yields statistically significant gains (p < 0.05) on most benchmarks. We believe this further supports the effectiveness of our approach, especially given its low cost and wide language coverage.

We will include the statistical analysis and correct the checklist in the revised version. Thank you again for your careful review.

---

Q1: choice of layers for ablation

For Tables 1, 2, and 3, we apply ablation in the middle layers and re-injection in the higher layers. The specific layer ranges used for each model are provided in our response to W1. This choice is inspired by recent interpretability studies on multilingual representations in LLMs (see Lines 241–242).

We will include this information directly in the revised paper for clarity.

---

Q2: Regarding Re-Injection

Yes, language was re-injected in the higher layers for all main results—i.e., we used decoupling, not pure ablation. The specific re-injection layers for each model are listed in our response to W1.

The implementation is nearly identical to ablation, except that we add the language component back instead of subtracting it.

---

Q3: only ablate/suppress (idiosyncrasies of) other languages

Thank you for the insightful suggestion.

We have tried using a language subspace that excludes the prompt language. However, in this case, the prompt has very little projection onto the subspace, so the ablation has minimal effect—leading to negligible changes in reasoning performance.

We agree this is an interesting direction and worth further exploration.

---

Once again, thank you for your constructive feedback. We hope our response has addressed your concern.

Dear Reviewer trBL,

Thank you again for your thoughtful feedback and valuable comments on our paper. We just wanted to kindly follow up and check if our rebuttal and the additional experiments have sufficiently addressed your concerns.

If there are any remaining questions or clarifications we can provide, we’d be more than happy to assist.

We truly appreciate your time and consideration.