MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages

Thank you for your encouragement of our work. We reply to your questions as follows.

W1: From my perspective, the motivation of MergeMinds is extremely similar with LangBridge, both of which utilize the mapping layer to map multilingual encoder to existing MLLM.

• In related works about MLLM, LLM does not have visual capabilities at all. Compared with this, the most significant difference in our research is that LLM has built-in multilingual capabilities, .
• Based on this difference, we need to explore whether there is a more efficient training solution than MLLM to effectively activate the multilingual capabilities built-in LLM.
• LangBridge is a solution like the works in MLLM that only considers integrating encoder to obtain capabilities that are not available in LLM. LangBridge does not consider boosting the built-in multilingual capabilities of LLM, which results in its performance being significantly lower than MindMerger. It lags behind MindMerger by 8.0% and 7.1% in low-resource languages and all languages in the MGSM dataset. Therefore, different from the existing work of MLLM, with MindMerger as a starting point, there are still many challenges to explore in the future to stimulate the multilingual as well as other bulit-in capabilities of LLM.

W2, Q3: The work of this paper has already been explored on multi-modal large models. Does this work adapt the paradigm from InstructBlip's Qformer and LLaVA's two-stage alignment training to a multilingual setting?

• Because the focus of our work is on how to combine multilingual capabilities from external model and LLM itself, we face the challenge of avoiding over-reliance on the capabilities of the external multilingual model during training. Similarly, we also face the challenge of avoiding over-reliance on the multilingual capabilities built into LLM during training.
• Moreover, our goal is to activate the multilingual capabilities of the LLM rather than to increase new capabilities it does not have, it is possible to explore a more efficient training solution than the MLLM. It can be observed from Table 6 in the paper that MindMerger has surpassed all baselines under the using of only 5,000 training samples per language at the second stage.
• In addition, the research on integrating multilingual models into LLM is still under-explored and different from the existing work on MLLM. For example, what data to use for training, what structure to use for integration, etc. still need more insightful discussions.

Q1: It appears that the Te language is not included in the MGSM. Could you explain the reason for this omission?

• This is not a technical issue, since the QAlign and MathOctopus work we followed did not compare Te, so we ignored Te as well.
• We added the Te experiment in the table above. MindMerger is still the best model on Te, with a 9.2% improvement over xCoT.

Q2: I noticed that some work similar to xCoT was referenced but not included in the comparative table. What was the rationale behind this decision?

• We selected the two strongest methods for each of the two categories of baselines: relearning-based and replacement-based.
• xCoT is a relearning-based method, but its performance is lower than MultiReason (MultiReason-Lora) and QAlign. Considering the page limit of the paper, we did not include it in the comparison.
• We show the performance of xCoT in the table above, and we consider adding it to the appendix.

Q4: Missing Reference.

• Thank you for the reviewers’ kind reminder, we will cite them in the next version of paper.

Best wishes.

Thanks for your great response. I have no additional problems.

Thank you for the valuable reviews. We have added some experiments and will include them in the next version of paper.

W1. In Section 3.2, the authors mention that they use translation data and query translation task data generated from public translation models for the two-stage training; while in lines 182-192, it shows that they also use several English task datasets, such as MetaMathQA. It is unclear how they use these data.

• For all models, we first use English task datasets to fully fine-tune LLM in order to obtain the reason capabilities, and then use other data to train.
• Taking the mathematical reasoning task as an example, both our model and all baselines are based on MetaMath-Llama (denote as MonoReason in our paper) as the base model, and MetaMath-Llama is fully fine-tuning Llama on the MetaMathQA dataset.

W2. Although the author mentioned that only a small number of parameters are trained during two-stage training, the paper lacks a comparison with baselines in this apsect. Particularly, the proposed method requires the external multilingual model, which is expected to affect the inference efficiency.

• Compared with the parallel encoding input query, the serial generation time is much longer.
• As report in the follow table, compared with the single LLM model QAlign under batch size 1, the average inference speed for MindMerger even faster than QAlign, where MindMerger spends an average of 3.65s inferring a sample while QAlign uses 3.81.

• This phenomenon is caused by the length of the text generated by QAlign is slightly larger than that of MindMerger as reported as belows:

• As for the cost of the external multilingual model, it is very insignificant. As reported in the below table, due to its small number of parameters, the time used to encode the query of external model is only half of QAlign (0.02s vs 0.04s).

W3. I know asking for new results might be unrealistic at this stage, but I am curious what would happen if you only used two-stage training for LLM without including an external multilingual model.

• There are three types of training data in the paper: (a) translation data, (b) English task data, and (c) query translation task data.
• We experimented with four dataset usage settings (b, c, b->c, c->b) to fully fine-tune Llama and compare the influence of using translation data for training in the first stage (a->b, a->c, a->b->c, a->c->b).
• The experimental results are shown in the table below. Regardless of the way the training data is used, MindMerger has an average accuracy that is at least 6.2 higher than the fully fine-tuned method.
• Using translation data as the first stage training data can help improve performance on 3 out of 4 fully fine-tuning settings (a->c, a->b, and a->c->b).
• The gain of translation data for MindMerger is significantly greater than that of fully fine-tuning Llama. With the help of translation data, MindMerger improved the average accuracy rate by 5.5%, while the improvements for fully fine-tuning Llama were 0.8%, 3.8% and 1.0% respectively.

Thanks again to the reviewer for the experimental suggestions. We believe that these additional experiments will help further enhance the impact of our work.

Best wishes.

Dear authors,

Thanks for your response and update with the results! I decided to raise the soundness score.

W3. Confusing word choices, such as "mapping stage", "merging" and "augmentation stage". These words often lead to different understanding rather than what are described in the paper.

• Thank you for your feedback, we will fine-tune the wording and description in the next version based on your comments.

W4. "Query translation task" and "query translation data" and many parts of the methodology are not explained clearly. It would be better if there are explicit visual examples or diagrams.

• Thank you for your feedback again. We followed the settings of QAlign to translate the query from English to non-English automatically as training data. Below are three examples of query translation task data:

If you have any questions about our work, we look forward to discussing with you and, if possible, hope that you can improve our score appropriately.

Best wishes.

W2. Missing baseline comparison XLT.

• We have discussed XLT in related work, lines 80-81, which is a prompt-based approach that brings limited improvements over open source models such as Llama through carefully crafted prompts.
• Our model and the baselines we compared are all supervised fine-tuning methods. Although the work of XLT is inspiring, prompt-based methods are not in the same category works as our model.
• We added a comparison between XLT and our model. The experimental results are shown in the following two tables. Under the same base model, MindMerger has a huge lead over XLT.
• Due to page limitations, we consider adding the experimental results of XLT to the appendix.

Thank you for your reviews.

The main concern is the performance of our model on the generation task. We believe that adding experiments on the generation task will help further expand the scope of our work.

We are grateful for the feedbacks on some writing improvement and we will revise them in the next version.

The specific responses are as follows:

W1. The method is incomplete and ambiguous on the generation side. It makes sense that injecting multilingual embeddings at the bottom for the input may help the LLM understand the context better, but that does not necessarily make the LLM generate in those native language better, especially if the LLMs are weak in producing content in native languages.

• Our model can be a key step towards improving LLM multilingual generation capabilities, since language understanding is the basis of language generation.

• To response this concern, we added an experimental comparison on the translation task based on Flores-101 dataset. We implemented four settings: X-En, En-X, X-Zh, and Zh-X. In each setting, we compare fully fine-tuned Llama2 (Llama-SFT) with MindMerger using 100 K and 200 K training samples per language, respectively.

• The experimental results are shown in the following four tables. MindMerger consistently outperforms Llama-SFT in translation quality under all comparison settings.

• Considering that the top of MindMerger is frozen and not trained, the improvement in generation quality comes from the enhancement of query understanding, which may be a valuable insight for future research on multilingual generation tasks.

Dear reviewer -- could you see whether the author response addressed your concern, especially about your comment on how their method can improve generation of LLMs? they provided new experiments, as well as a comparison with baseline you proposed. If you can modify the scores based on the response, that'd be very helpful!

Q1. The details of the two training stages lack clarity, specifically regarding the utilization of translation and parallel data, as well as the precise training tasks involved.

• Thank you for your feedback, we will improve the description of training data in the next version of our paper.
• In the first stage (mapping stage), all tasks including Math, Commonsense, and NLI are trained using non-English to English translation data. The following is an example:

• In the second stage (augmentation stage), the input query of each task is translated into non-English to construct multilingual training samples, both English and non-English samples are used to train our model. Below are examples of training samples for three tasks:

Thank you again for your comments. We hope our responses can resolve your concerns. We hope you can improve our score appropriately and are willing to have more discussions with you.

Best wishes.

W3. This is further proved by Figure (4b), which illustrates that the mapping layer tends to obtain a unified representation for different languages and this is the same as QAlign. Therefore, it is confusing why this method can outperform QAlign.

• This reflects that our model has better capability to obtain unified representation than QAlign.
• In Table 9 of the paper, we calculated the cosine similarity and top-1 retrieval recall between representations of the same query expressed in different languages to quantify the model’s capability to obtain a unified representation. The results show that our model significantly improves LLM’s capability to obtain a unified representation.
• We further compare the last layer pooling representation of QAlign and our method in the following table. It can be observed that although QAlign slightly outperforms MonoReason, the score of cosine similarity and recall@1 are still far behind our model, which proves that our model is better able to obtain unified representations.

W4. The paper uses Llama2 as the base model. However, its multilingual performance is far from sota. Until submission, I believe Mistral, Gemma, and Llama3 have been released, whose multilingual performance is much better.

• In fact, we have compared Mistral as well as the larger MetaMath-Llama-13B in Table 5. The experimental results show that our method is also consistently effective on LLM with stronger multilingual capabilities.
• We further compare our method with baselines based on Llama3 in the following table. Our method still outperforms all baselines.

We thank the reviewer for taking the time to provide reviews. However, we would like to clarify some misunderstandings as follows:

W1. To my understanding, the Embedding Layer of the model merely converts the query from text space to the token/embedding space. This transformation may not be interpreted as "understanding the input" since the information hasn't been operated and combined by the attention layer. However, after the multilingual model has been "understood" by the model. Therefore, it does not make much sense to concatenate these two embeddings.

• Our model is based on the finding that providing LLM with inputs from two representation spaces can improve LLM's multilingual reasoning capability, where one representation comes from the native space of LLM's embedding layer, and the other representation is the mapping of multilingual queries to unified space provided by the multilingual model.

• The two different types of representations are the key to boost better understanding of the query in subsequent layers of LLM.

W2. As contains the information of query q, includes duplicated information. This may be why the method can surpass baselines such as QAlign and Translate-En. It would be more convincing if baselines also input duplicated information.

• We duplicated the input for all baselines during training and inference. As shown in the below table, the performance of 4 out of 6 baselines decreased with duplicated input.
• Note that our model outperforms all baselines by at least 6.7% and 8.0% across all languages and low-resource languages, which far exceeds the improvement of the baseline by duplicated input. Therefore, it can be considered that duplicate input is not the mechanism for improving the performance of our model.

Dear reviewer, could you check at the authors new experimental results and see whether it address your concerns, and update the score accordingly if they do?

Thanks so much for so detailed reply in rebuttal to address my concern. Thay addressed my concerns, and I have raised my score accordingly.