Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts

Thank you for your kind and insightful comments. This paper aims to democratize medical AI for a broader linguistic community. By collecting data, analyzing and leveraging the model's multilingual mechanisms, and integrating linguistic priors, it provides the community with high-quality data, efficient, generalizable, and scalable technical solutions, as well as high-performing models. Below, we will address your questions one by one.

Q1: Evaluation is performed on translated test sets, generated by other language models.

Regarding the evaluation, we used peer-reviewed medical evaluation datasets, which are detailed in Appendix A.3 and A.5 and referenced in lines 126-127 and 226-227 of the revised version. For languages without existing medical evaluation datasets, we utilized the medical-related sections of MMLU to create evaluation sets for those languages. In total, we integrated 10 medical evaluation datasets for the corresponding languages and supplemented others with the medical section of MMLU for other languages. The specific datasets are described in detail in Table 7.

To further verify the accuracy of the translated datasets as suggested, we sampled 100 items from the evaluation sets of two minor languages, Croatian (Hr) and Danish (Da). With the assistance of two Chinese translation professionals in the respective languages, we enlisted three licensed Chinese doctors to annotate the accuracy of the items. The average accuracy rates were 90% and 92%, respectively, partially validating the quality of the translated datasets. Additionally, a prior study [1] demonstrated that Google Translate achieves an average accuracy of 82.5% in the complex and rigorous context of emergency medicine.

Meanwhile, for the languages evaluated using MMLU medical parts for evaluation and within MMMLU, we use the accuracy comparation of GPT-4o and Apollo-MoE (develop from Qwen2-7B) on the MMMLU medical part (proofread by OpenAI) and the MMLU translated medical part, and compared the results. As shown in the Table below, the models achieved nearly identical scores on these two datasets, with the largest accuracy difference being only 1.3%. Further verified the effectiveness of Google Translate.

The results are mentioned in the lines 127-128 and datailed in the Appendix A.6 of the revised version.

However, we also acknowledge that translation is a stopgap measure to showcase the medical multilingual capabilities of Models the community. In future work, we plan to leverage the influence of this study to seek support from local medical professionals and focus on building a multilingual medical evaluation set with localized characteristics. This will be a tedious but meaningful work.

Q2: Little discussion is made of the risk of test set contamination in the training set of the model in languages other than the source language.

For other minor languages, we extracted 2,000 samples from the English data and used Google Translate to create training corpora for each language. Since the source of the translation data inherently eliminates the risk of test set leakage, the risk of leakage in the translated dataset is minimal.

Moreover, to address your concerns, we conducted data leakage detection on Croatian (Hr) and Danish (Da) using a 16-token overlap as the measurement criterion as suggested. This process identified 2 (0.1%) and 0 (0.0%) overlapping data items out of 2000, respectively, further supporting the above statement.

[1] A Pragmatic Assessment of Google Translate for Emergency Department Instructions

Q3: Most techniques presented in the paper do not appear to be very novel, even though their combination seems to be, and their choices are well-justified.

Complexity is not the goal. Towards addressing real practical problems, simpler methods are often more elegant and can provide valuable insights to the community. We use Hybrid Routing as a tool to explore model mechanisms. By employing Post-MoE and introducing language-family experts, we effectively utilize the discovered mechanisms to efficiently democratize medical AI for a broader linguistic community.

Q4: In general, could you help me perceive the theorical aspects of your other contributions which I may have missed?

As noted in the paper, we have observed and drawn inspiration from studies on neural analysis that explore large-model multilingual mechanisms. From an application perspective, our method leverages the modular characteristics of MoE to efficiently utilize the discovered mechanisms. It can be extended to accommodate any number of languages while enabling the combination and disassembly of modules.

Theoretically, while many studies have highlighted the presence of multilingual neurons at the bottom layers of the model, we further observed that activation parameters at bottom layers are concentrated in the Chinese and English language modules rather than being distributed across various language modules. We hypothesize that this may reflect a model strategy that relies on core language experts at the bottom layers for multilingual translation. As this finding is not directly relevant to the main focus of the paper and is a broader multilingual question, we chose not to include it explicitly and plan to explore it in future work.

This observation also demonstrates that our method provides an additional dimension (language experts) for analyzing multilingual mechanisms beyond the depth perspective. It invites further investigation into questions such as which language experts should be utilized at specific depths and how language experts at different depths interact with one another.

Moreover, by treating different modalities as distinct languages, the code framework provided in this paper can also be applied to study how modalities interact during the multimodal adaptation process.

Q5: What role does relative importance of languages play in the linguistic family grouping? English is likely overshadowing other languages using the same expert.

As shown in Table 1, different languages generally enhance each other, with English having a positive impact on other languages. Your perspective is certainly valuable, and we plan to conduct more fine-grained analyses in future works.

Q6: How did you choose your 50 languages? Moreover, why did you include Latin in your experiments? It's no longer an active language of science since many centuries, and Google Translate is unlikely to do a good job on modern medical documents in this case.

We selected representative languages based on geographical distribution. Latin was included because much traditional Western medical knowledge is recorded in Latin. Additionally, Latin is widely used in biology, anatomy, and medical law. Teaching the model to understand Latin can assist the community in interpreting and drawing inspiration from the knowledge of predecessors.

Q7: About Open source

Open sourcing is actually a core motivation of our work. We have released all datasets, evaluation sets, code, and models from the paper, which have received significant downloads.

Q8: Why do you want to present this work at ICLR?

Frankly speaking, ICLR has a broader influence. Given the need for the work to reach a larger audience and advance related research fields by engaging more like-minded researchers, we chose this conference.

Additionally, ICLR has featured significant related work in recent years on topics such as multilingual models [1], MoE architectures [2][3], and medical applications [4].

[1] Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

[2] Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

[3] Sparse MoE with Language Guided Routing for Multilingual Machine Translation

[4] Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study

Q3: employing Google Translate to translate the questions and answers of MMLU - why not use mmmlu from OpenAI?

OpenAI released MMMLU around September 23, 2024, very close to the conference deadline of October 1, 2024. Unfortunately we were not able to use it. Thank you for giving us the information.

Q4: Have any experiments been conducted to assess the robustness of translations used for low-resource languages? If not, how do you propose to validate the accuracy of these translations, given the potential consequences in a medical context?

For evaluation, we utilized a peer-reviewed medical evaluation dataset, detailed in Appendix A.3 and A.5. For languages lacking existing medical evaluation datasets, we followed the LLaMA3 multilingual evaluation framework and used the medical-related sections of Google Translate MMLU to construct evaluation sets for these languages.

To verify the accuracy of the translated datasets as suggested, we sampled 100 items from the evaluation sets of two rare languages, Croatian (Hr) and Danish (Da). With the assistance of two Chinese translation professionals in the respective languages, we enlisted three licensed Chinese doctors to annotate the accuracy of the items. The average accuracy rates were 90% and 92%, respectively, partially validating the quality of the translated datasets. Additionally, a prior study [1] demonstrated that Google Translate achieves an average accuracy of 82.5% in the complex and rigorous context of emergency medicine.

However, we also acknowledge that translation is a stopgap measure to showcase the medical multilingual capabilities of Models the community. In future work, we plan to leverage the influence of this study to seek support from local medical professionals and focus on building a multilingual medical evaluation set with localized characteristics. This will be a tedious but meaningful work.

Typo: " 3b is unclear" We have enhanced the resolution of the image. The corresponding explanation of the image can be found in lines 250-252 and 284-289 of the revised version. We hope this will be helpful to you.

[1] A Pragmatic Assessment of Google Translate for Emergency Department Instructions

Thank you for your kind and insightful reviews. Below, we will address your questions one by one.

Q1: Although the MoE model's efficiency is emphasized, a more extensive comparison with the same computation cost dense models (especially regarding computational cost and training efficiency) could strengthen the claims.

Thank you for your suggestion, which will indeed enhance the robustness of the paper. We have conducted additional experiments based on your recommendations and will present them below.

Experiment Settings: To accurately construct a dense model of equivalent size, we replicated the MLP following the approach used in MoE Upcycling and initialized the routing with an average distribution. Unlike MoE, the initialized dense model employs full activation instead of sparse activation.

• Table 2 Related Supplementary Experiments:

  The purpose of Table 2 is to demonstrate the generalization advantage of the Hybrid Routing method over other approaches. We have provided supplementary baseline results for dense models with parameters matching either the total parameters and the activation parameters of the MoE models. As shown in the table, compared to a Dense model with the same total parameters, the MoE model exhibits better generalization on minor languages. Additionally, compared to a Dense model with the same activation parameters, the MoE model performs better on both major and minor languages. For related content, please refer to lines 195 and 222 in new version paper.

  Method/Model	Active Param.	Param.	Avg. Major	Avg. Minor
  Qwen2-0.5B	0.49B	0.49B	29.7	31.5
  Dense	0.49B	0.49B	37.8	24.6
  Dense with Same Active Param.	0.81B	0.81B	38.4	26.2
  Dense with Same Total Param.	3.95B	3.95B	42.0	30.9
  MoE with Hybrid-k Routing	0.81B	3.95B	40.0	32.0

• Table 5 Related Supplementary Experiments:

  The purpose of Table 5 is to evaluate the scalability of the primary method proposed in this paper (PostMoE with language family experts). Due to time constraints, we only supplemented the dense baselines with the same activation parameters. As shown in the Table, the MoE models outperform the Dense model with the same number of activation parameters. For related content, please refer to line 394 in new version paper.

  Method/BaseModel	Active Param.	Avg. Major	Avg. Minor
  Qwen2-0.5B	0.49B	29.7	31.5
  Dense	0.49B	39.2	32.2
  Dense with Same Active Param.	0.52B	39.4	34.0
  MoE with Hybrid-k Routing	0.52B	40.5	34.6
  Qwen2-1.5B	1.54B	42.9	38.4
  Dense	1.54B	52.2	43.7
  Dense with Same Active Param.	1.63B	52.8	44.1
  MoE with Hybrid-k Routing	1.63B	54.8	44.9
  Qwen2-7B	7.62B	55.2	49.2
  Dense	7.62B	69.0	55.7
  Dense with Same Active Param.	8.02B	68.5	56.3
  MoE with Hybrid-k Routing	8.02B	69.9	58.3

Q2: The improvement mostly appears in Italian and French, also decreased in many more low-resource languages. This seems to lead to some overclaim... how is the data-mixture determined?

In Tables 2 and 4, the model performance improved across all languages after training, with an average increase exceeding 10%. In Table 5, all minor languages showed improvement except for Serbian (Sr), and nearly one-third achieved an average improvement of over 10%. For Serbian, we observed the performance decline also within Dense models, which may be attributed to the unique characteristics of its alphabet and linguistic logic.

Regarding data mixing, we collected, processed, and utilized as much high-quality medical data as possible without deliberately explore the data-mixture influence. However, we agree that this is a highly promising research direction, and we appreciate your insightful suggestion.

Thank you for your suggestion. Following your recommendation, for the languages evaluated using MMLU medical parts for evaluation and within MMMLU, we added the accuracy of GPT-4o and Apollo-MoE (develop from Qwen2-7B) on the MMMLU medical part and the MMLU translated medical part , and compared the results. As shown in the table, the models achieved nearly identical scores on these two datasets, with the largest accuracy difference being only 1.3%. Additionally, as mentioned earlier, in our rigorous manual verification, 90% and 92% of the 100 instances were completely consistent. The results are mentioned in the lines 127-128 and datailed in the Appendix A.6 of the revised version.

Thank you for your kind and insightful comments. This paper aims to democratize medical AI for a broader linguistic community. By collecting data, analyzing and leveraging the model's multilingual mechanisms, and integrating linguistic priors, it provides the community with high-quality data, efficient, generalizable, and scalable technical solutions, as well as high-performing models. Below, we will address your questions one by one.

Q1: The paper focused on the medical LLMs, while the evaluation is based on MMLU; although it includes part of the medical-related tasks, the evaluation on this general domain dataset will bring additional noise on evaluating LLMs in medical tasks.

We sincerely apologize for any misunderstanding caused by unclear writing. Regarding the evaluation, we only utilized the medical-related sections of MMLU to create evaluation sets for languages without existing medical evaluation datasets. For other languages, we used peer-reviewed medical evaluation datasets, which are detailed in Appendix A.3 and A.5 and referenced in lines 127-128 and 225-226 of the original paper. In total, we integrated 10 medical evaluation datasets for the corresponding languages and supplemented others with the medical section of MMLU for other languages. The specific datasets are described in detail in Table 7 of the original paper.

Q2: Much relevant research on extending pre-trained models to different languages is overlooked in this paper (related works).

GreenPLM is an impressive contribution, sharing the same motivation as our work—to efficiently expand the model's multilingual capabilities. Since many similar outstanding works focus on optimizations based on BERT, they were not included in the related work section. Following your suggestion, we have supplement the related work to provide readers with a broader perspective and highlight the substantial groundwork and innovative ideas within the multilingual community. Please refer to lines 451-458 in the revised version for details.

Typo: "shwows" We have fixed it in the new version. Thank you for carefully reviewing our work.

Dear Reviewer somb,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10250 Authors

Q4: I wonder if the differences of 0.9 and 0.6 for major and minor languages, respectively, are meaningful enough to consider.

First, for the 7B model, the performance improved of major and minor languages are 0.9 and 2.6, respectively, with a competitive increase in the average performance of minor languages.

More importantly, the primary method proposed in this paper (PostMoE with language family experts) offers the advantages of generalizability, enabling adding other languages efficiently and effectively. To provide a more intuitive understanding, we selected two additional languages not included in the 50 languages, "Bengali" (Bn) and "Amharic" (Am), to demonstrate the model's efficient generalization capability.

Specifically, we processed 2,000 data samples for "Bengali" (Bn) and "Amharic" (Am) and continued fine-tuning the Dense models and ApolloMoE models. The results of different scales are shown in the tables below.

The experimental results demonstrate the clear advantages of the proposed method in adapting to additional languages. For performance improvement in newly added languages, the ApolloMoE model outperforms the Dense model across all scales, with training-related gains also showing advantages in most cases. For preserving the performance of original languages, the ApolloMoE model maintains or even improves performance across nearly all scales, whereas the Dense model generally experiences a decline in performance. Please refer to lines 363-365 and App.E of revised version for more details.

Typo. Figure2: We have corrected the image. For further details, please refer to lines 170-179 of the revised version. Thank you for your careful review.

Thank you for your kind and insightful comments. This paper aims to democratize medical AI for a broader linguistic community. By collecting data, analyzing and leveraging the model's multilingual mechanisms, and integrating linguistic priors, it provides the community with high-quality data, efficient, generalizable, and scalable technical solutions, as well as high-performing models. Below, we will address your questions one by one.

Q1: The core idea of the proposed method does not appear particularly novel

From the perspective of the Hybrid Routing method, this approach addresses the issue in the BTX method where the initialization of the MoE model's Attention layer cannot directly align with the Attention layers of expert models trained on specific tasks. Additionally, it provides interpretability that Sparse Upcycling lacks.

More importantly, from the methodological perspective of the paper, we view this approach as a tool for exploring model mechanisms. By employing Post-MoE and introducing language-family experts, we fully leverage the discovered mechanisms to efficiently achieve the original objectives.

Q2: It may be unfair to directly compare the results between the baseline dense model and the MoE model with the same base model size.

Thank you for your suggestion, which will indeed enhance the robustness of the paper. We have conducted additional experiments based on your recommendations and will present them below.

Experiment Settings: To accurately construct a dense model of equivalent size, we replicated the MLP following the approach used in MoE Upcycling and initialized the routing with an average distribution. Unlike MoE, the initialized dense model employs full activation instead of sparse activation.

• Supplementary Experiments Related to Table 2:

  The purpose of Table 2 is to demonstrate the generalization advantage of the Hybrid Routing method over other approaches. We have provided supplementary baseline results for dense models with parameters matching either the total parameters and the activation parameters of the MoE models. As shown in the table, compared to a Dense model with the same total parameters, the MoE model exhibits better generalization on minor languages. Additionally, compared to a Dense model with the same activation parameters, the MoE model performs better on both major and minor languages. Please refer to lines 195, 222-224 in the revised version for details.

  Method/Model	Active Param.	Param.	Avg. Major	Avg. Minor
  Qwen2-0.5B	0.49B	0.49B	29.7	31.5
  Dense	0.49B	0.49B	37.8	24.6
  Dense with Same Active Param.	0.81B	0.81B	38.4	26.2
  Dense with Same Total Param.	3.95B	3.95B	42.0	30.9
  MoE with Hybrid-k Routing	0.81B	3.95B	40.0	32.0

• Supplementary Experiments related to Table 5:

  The purpose of Table 5 is to evaluate the scalability of the primary method proposed in this paper (PostMoE with language family experts). Due to time constraints, we only supplemented the dense baselines with the same activation parameters. As shown in the Table, the MoE models outperform the Dense models with the same number of activation parameters. Please refer to lines 394-410 in the revised version for more details.

  Method/BaseModel	Active Param.	Avg. Major	Avg. Minor
  Qwen2-0.5B	0.49B	29.7	31.5
  Dense	0.49B	39.2	32.2
  Dense with Same Active Param.	0.52B	39.4	34.0
  MoE with Hybrid-k Routing	0.52B	40.5	34.6
  Qwen2-1.5B	1.54B	42.9	38.4
  Dense	1.54B	52.2	43.7
  Dense with Same Active Param.	1.63B	52.8	44.1
  MoE with Hybrid-k Routing	1.63B	54.8	44.9
  Qwen2-7B	7.62B	55.2	49.2
  Dense	7.62B	69.0	55.7
  Dense with Same Active Param.	8.02B	68.5	56.3
  MoE with Hybrid-k Routing	8.02B	69.9	58.3

Q3: It appears that the performance of minor languages is not significantly worse compared to major languages in recent top models. Do the authors provide any other explanation for the motivation behind developing the proposed method?

Contribution to the Open Source Community: For open-source models, the accuracy gap between major and minor languages exceeds 10% and the ACC is under 50, underscoring a significant disparity in this domain within the open-source community. Our work open-sources all data, code, and models, providing the community with an efficient and practical solution to address this issue.

Additional Motivations:

• Privacy Considerations: In medical services, a locally deployed medical AI model is essential to eliminate the risk of personal medical data leakage. Current closed-source AI service cannot address this issue.
• Community Accessibility: In many minor language communities, particularly in regions with limited medical resources, Internet access is often challenging. A locally operable medical AI model can mitigate this issue at a very low cost. Current closed-source AI service cannot address this issue.

Dear Reviewer kMZh,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10250 Authors

Dear Reviewer kMZh,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10250 Authors

(I sincerely apologize for the delay in my response.) I would like to thank the authors for their thorough efforts in addressing my questions and concerns, including conducting additional experiments. Since most of my concerns have been sufficiently addressed, I have increased my score and voted to accept the submission to the conference.