Sparse MoE with Language Guided Routing for Multilingual Machine Translation

[CONS6 Zero-shot Translation Evaluation] Applying the MMT model to conduct zero-shot translation is an interesting suggestion and application. Due to the limited time of the rebuttal period, we collected some non-English centric test sets from the Tatoeba challenge [6] and evaluated $*Dense*$ and $*Lingual-SMoE*$ on them. Although our $*Lingual-SMoE*$ is not specialized for multi-to-multi translation, it outperforms its Dense counterpart, as shown in Table S5.

[6] https://github.com/Helsinki-NLP/Tatoeba-Challenge

Table S5 Zero-shot translation performance

[CONS7 Bibliography and Citation] Thanks for the detailed review. We have addressed them in our updated draft.

• We have changed "Nllbteam" to "NLLB Team" in the bibliography.
• We have updated the information about the two published versions of the paper.
• We adopt the original Adam in [7]. We have added it to the bibliography and cited it in P6. The hyperparameters of Adam are changed from $\alpha$, $\beta$ to $\beta_1$, $\beta_2$, $\epsilon$, which are set to $0.9, 0.98, 10^{-8}$, respectively. We will check all the bibliographies to make sure the information is up to date in the final version.

[7] Adam: A Method for Stochastic Optimization

[Question: Code Availability] The core implementation codes are provided in the supplement with training and evaluation scripts. URLs for additional code will be published in the final version.

[CONS4 Analysis of LS-SMoE]

[Clarification on the analysis of $*LS-SMoE*$] We respectfully point out that the intention of the argument in P7 2(1) is that the fixed routing of $*LS-SMoE*$ leads to an imbalanced training of experts, where experts corresponding to English are activated much more often, due to its quantitative superiority and the English-centric nature of the OPUS dataset. Specifically for the SMoE model, the results in [3,4] show that the routing decision in the decoder is more language-specific. This is also consistent with the design in LSL [5] which the decoder is completely language-specific. Thus, the effect of imbalance activation of $*LS-SMoE*$ is more severe in the en-xx direction, which requires higher routing diversity to explore language-specific parameters. Regarding the score per language pair, since the number of language pairs is relatively large ($96$ for OPUS-50, $188$ for OPUS-100), we will find a better format to present the detailed result in the final version.

[3] Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

[4] Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model

[5] Learning Language-Specific Layers for Multilingual Machine Translation

[Decoder language-specific SMoE baseline] We have implemented an SMoE baseline $*Hybrid-SMoE*$ where only the decoder is target language-specific routing, while the encoder is only dependent on the input token. The only difference to $*LS-SMoE*$ is that the router of $*Hybrid-SMoE*$ is learnable. From the results in Table 1 and Table 4, we can conclude that learnable routing is a useful technique for improving SMoE for MMT. Therefore, we consider $*Hybrid-SMoE*$ as an improved version of the reviewer's proposal. The performance of $*Hybrid-SMoE*$ shows that language-level routing alone is too coarse and justifies combining language, language family, and input information in a hierarchical router.

[CONS5 Implementation Details]

[Model efficiency] Thanks for the interesting question. We measure model efficiency by parameter size, the number of tera floating point operations (TFLOPs), and training and inference time. in Table S3. In particular, we record the average tokens processed per second (token/s) on the same test set to measure inference efficiency. For training efficiency, we report the average second cost per step (s/step). We compare our $`Lingual-SMoE`$ to one of the current SOTA SMoE models $*GS-SMoE*$. The result shows that with significant improvement in translation performance, our design demands only marginal additional parameters.

Table S3 Comparison of Model efficiency of $`Lingual-SMoE`$ and $*GS-SMoE*$

[Data epoch] As for the details of the data epochs, we show the training steps and corresponding data epoch on OPUS-16, OPUS-50, OPUS-100 in Table S4. We conducted a preliminary experiment to determine the training steps of our models to ensure that they are well-trained. Specifically, we train $*GS-SMoE*$ and $*Dense*$ on OPUS-100 for 10 epochs (about 918k steps). We find that the losses for both models converge quickly at about 100K steps and then decrease at a much slower rate. Similar experiments are performed on all datasets so that we can finally determine the training steps that ensure model convergence.

Table S4 Training step and epoch details.

[Checkpoint selection] Since the validation and test sets are separate, we use the overall language perplexity on the validation set as the metric to select checkpoints during training. This setting applies to all methods to ensure a fair comparison.

We sincerely appreciate your careful review and a great summary of our contributions. And thank you for the constructive comments. Please see our point-by-point response below.

[CONS1 Evaluation Reproductivity] We appreciate this constructive suggestion. The Sacrebleu Hash is as below. It has been added in the updated draft as a footnote on page 6, highlighted in orange.
BLEU Signature: nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.3.1

[CONS2 Evaluation on Other Metrics]

Thanks for your comments. We have provided the ChrF, COMET, ROUGE-L, and METEOR scores of $*Dense*$, $*GS-SMoE*$, and $`Lingual-SMoE`$ trained on OPUS-50 datasets in Table S1, which show that our $*Lingual-SMoE*$ is competitive across different metrics. The ChrF score is computed using Sacre-BLEU. The COMET score is calculated using the COMET framework. The ROUGE-L, and METEOR scores are calculated with the HuggingFace evaluate library. Table S1 is included in the updated draft, in Appendix.

Table S1 Multilingual machine translation performance on OPUS-50 dataset.

[CONS3 Improving LS-SMoE]

Thank you for the helpful suggestion. We understand your concerns about building a stronger baseline that better aligns with the original study. However, we decided not to include $*LS-SMoE*$ with source and target language-specific routing in the encoder, and dense pre-training as baselines due to the following reasons:

• We have previously experimented with the $*LS-SMoE*$ encoder with both source and target language-specific parameters. However, the performance of such a combination is suboptimal and inferior to our current choice;
• While dense pre-training has been verified to benefit $*LS-SMoE*$, training baseline models from scratch better cope with our settings, which can exclude possible interference for examining the effect of language-guided routing, and whether it learns language-specific and language-family-specific knowledge. Therefore, $*LS-SMoE*$ with dense pre-training is not included in our baselines for fairness of comparison.

[Extra experiment on $*LS-SMoE*$] We have reported the performance of the two $*LS-SMoE*$ variants in Table S2. The first variant we have examined is $*LS-SMoE*^{+hybrid}$, where the encoder is converted from using only source language routing to a mixture of source and target language routing (i.e., source language routing in the two lower MoE layers and target language routing in the upper). As the ratio of target-level routing increases, $*LS-SMoE*^{+hybrid}$ performs better on the more diverse setting of $`en-xx`$ with an average 2.53 increase in BLEU, yet suffering from a 7.23 decrease in the BLEU score on the setting of $`xx-en`$. Another variant we studied is $*LS-SMoE*^{+dense}$, whose weights are initialized from $*Dense*$. More specifically, the expert feed-forward networks in $*LS-SMoE*^{+dense}$ are initialized from the same feed-forward networks in $*Dense*$ in the corresponding layers. The result of $*LS-SMoE*^{+dense}$ is much better than $*LS-SMoE*$, but still falls behind our $*LGR-SMoE*$ without dense pre-training.

[Possible reasons] The possible reason for the reverse effect of optimizing $*LS-SMoE*$ model with both source and target language routing in the encoder could be attributed to the complexity of SMoE optimization by nature [1,2]. Single type routing policy is already challenging. With a fused routing policy, the difficulty level of optimization will further increase. In summary, it will be interesting to discover different routing policies of the SMoE model; however, it falls beyond the scope of our paper and we will leave it in the future.

[1] Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers

[2] TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training

Table S2 Average BLEU score of multilingual machine translation performance on OPUS-16 dataset

[CONS3 Inconsistent Improvement Across Langauges]

[Detailed observation] Thanks for pointing this out. (1) From Tables 1 and 2, the translation direction $`xx-en`$ benefits less from our method than $`en-xx`$ to a certain extent. (2) Taking the improvement of $`Lingual-SMoE`$ from its SMoE counterpart $*GS-SMoE*$ as an example: the improvements are greater in OPUS-100 ($2.84\%$) than in OPUS-16/50 ($2.42\%$ and $1.39\%$); (3) among low/medium/high-resource language groups (in OPUS-16 as an example), the improvements are more obvious in low and medium resource languages ($3.66\%$ and $3.66\%$) compared to high resource languages ($1.33\%$).

[Detailed analysis] Several conclusions can be drawn from the results: (1) $`xx-en`$ direction translation shows less improvement compared to $`en-xx`$. (2) Low-resource languages benefit more from LGR. (3) When the number of languages increases, the performance gap between $`en-xx`$ and $`xx-en`$ decreases, which indicates that more languages help our method improve en-xx and xx-en directions simultaneously.

[Possible reasons] The possible explanations for these results are: (1) in $`en-xx`$, we design routing with more exploration capacity because the "$`xx`$" side covers more language families, which is the reason for the higher performance gain. (2) In low-resource scenarios, a linguistic prior drawn from closer family languages is more helpful, which is consistent with the design intuition of the prior. (3) More languages and data allow for more exploration.

[Conclusion and future work] Our method is advantageous in most cases across language numbers and data abundance. We are pioneering in the design and incorporation of linguistic priors in MMT with the SMoE model. We acknowledge that there is a lot of follow-up research that can be done in these promising directions, which we will explore in future work. For example, we can further investigate how to improve the capacity and performance of $`xx-en`$ routing exploration.

[CONS4 Writing Details] Thanks for your careful reading. We have addressed the following issues in our updated draft, highlighted in orange:

• Citation formats and grammatical consistency in the last paragraph of the "Related Work" section.
• In Section 4.2, "Table 1" is cited as "Table 4.2".
• The confusion of alpha and beta values is because they represent different hyperparameters. Eq. (2) is the ratio of load balancing and language grouping loss in the total loss. while in the implementation are the hyperparameters for Adam. To avoid confusion, we have changed the coefficients of the auxiliary losses to $c_1$ and $c_2$. For the Adam hyperparameters in the implementation details, $\alpha$ and $\beta$ are changed to $\beta_1$ and $\beta_2$.

Thanks for acknowledging our methodology and experiment as meaningful and sufficient. We provide pointwise responses to your concerns about our experiments as below.

[CONS1 Dynamic Routing According to Language Difficulty]

Thanks for pointing this out. We examine language difficulty from two perspectives: grammatical complexity and resource availability, and combine them in dynamic routing. Here, we (1) report the results of the additional dynamic routing experiments that integrate data abundance and grammatical complexity; (2) explain why we consider the data abundance factor; and (3) appropriately refine paper writing to avoid potential confusion.

• [Additional experiment results] First, we compute language difficulty scores by adding grammar difficulty (rated by GPT-4 from 1-5) and data abundance scores (0,1,2 for high/medium/low resource) as the metrics to divide languages into easy, medium, and difficult groups, following the ratio of high/medium/low resource language numbers. Then we train a $`Lingual-SMoE`^{+grammar}$ with the language difficulty groups and compare it to the dense and SMoE baselines. All models are trained in OPUS-16 settings. As shown in Table S1, $`Lingual-SMoE`^{+grammar}$ outperforms the baseline models with the incorporation of both grammar and data information.

Table S1 Performance comparisons of $`Lingual-SMoE`^{+grammar}$ with dynamic routing according to language difficulty defined by grammar complexity and data abundance, compared to $*Dense*$ and vanilla SMoE $*GS-SMoE*$. Average BLEU scores for each direction are reported.

• [Reason why we mainly consider data availability] The reason we do not include grammar complexity is: First, the data amount is an objective value, but grammar complexity is subjective for people with different first languages and education. Also, the degree of grammar complexity varies from GPT to human. So we decided not to consider grammar complexity in our $`Lingual-SMoE`$.

[CONS2 Model Efficiency]

Thanks for the comments. We report model size, the number of tera floating point operations (TFLOPs), inference, and training efficiency in Table S2. In particular, we record the average tokens processed per second (token/s) on the same test set to measure inference efficiency. For training efficiency, we report the average second cost per step (s/step). We compare our $`Lingual-SMoE`$ to one of the current SOTA SMoE models $*GS-SMoE*$. With significant translation performance, our design requires only marginal additional parameters.

Table S2 Comparison of Model efficiency of $`Lingual-SMoE`$ and $*GS-SMoE*$

We clarify that the improvements are not driven by the increased number of experts, since all the baseline SMoE models and our designs (except $*ST-SMoE*$ with top-$1$ routing) activate only the top 2 experts during the training.

Thank you for your comprehensive classification and the additional information. The modifications made in the manuscript have addressed all of my concerns, and I have revised my revaluation accordingly.

Dear Reviewer xU8U,

We thank reviewer xU8U for the time of reviewing and the constructive comments. We really hope to have a further discussion with reviewer xU8U to see if our response solves the concerns.

In response, we have (1) presented additional experimental results and analysis of dynamic routing; (2) provided details about model efficiency; (3) clarified other analysis and writing confusions;

We genuinely hope reviewer xU8U could kindly check our response. Thank you!

Best wishes,

Authors

[CONS3 Benchmark Selection]

[Extra experiment with domain consistent dataset] As the reviewer acknowledges, it is challenging to train models on WMT, since the data size of its usual setting [1] is about 400M combining en-xx and xx-en directions, which is four times larger than OPUS-100. So, we re-examine language guided routing $*LGR-SMoE*$ on a subset of WMT to exclude possible interference from data domain similarity. Specifically, we collect all WMT16 data, lv-en from WMT17, ro-en from WMT18, and kk-en from WMT19, in total 9 language pairs. We implement data pre-processing and model training with the same setting as OPUS-16. The average BLEU scores on each language pair are presented in Table S1, which shows that language-guided routing consistently outperforms.

Table S1 Performance comparisons of $*LGR-SMoE*$ compared to $*Dense*$ and vanilla SMoE $*GS-SMoE*$ on WMT dataset. Average BLEU scores for all language pairs are reported.

[Extra experiment with complex language difficulty metric] We address the confusion by (1) experimenting with dynamic routing including both grammar complexity and data abundance, and (2) explaining the reason we do not include grammar complexity in the main content.

• we have developed a variant of dynamic routing that accounts for both grammar complexity and data volume. As shown in Table S2, it consistently outperforms the dense and SMoE baselines.
• Specifically, we group languages according to the same ratio of high/medium/low resource language numbers, with the language difficulty metrics that add grammar complexity scores (rated by GPT-4 from 1-5) and data abundance scores (0,1,2 for high/medium/low resource). Then we train $`Lingual-SMoE`^{+grammar}$ with language difficulty groups and compare it to the dense and SMoE baselines. All models are trained in OPUS-16 settings.
• Due to the subjective characteristic of grammar complexity, we choose to use only data abundance as a metric of language difficulty. To avoid misunderstanding, we revise the paper draft by explicitly defining language difficulty in the abstract and introduction, and detailing the implementation of the method. We also update the probe test in the Appendix.

Table S2 Performance comparisons of $`Lingual-SMoE`$ with dynamic routing according to language difficulty defined by grammar complexity and data abundance, compared to $*Dense*$ and vanilla SMoE $*GS-SMoE*$. Average BLEU scores for each direction are reported.

[CONS4 visualization] For better visualization, we provide the heatmap of cosine similarity of routing decisions in the final encoder and decoder SMoE layer, where the observation is consistent with the LGR design: closer languages have higher similarity in routing. We also enlarge the original visualizations and move them to Appendix. Please refer to Figure A6-8 in the updated draft.

Many thanks to reviewer mayu for acknowledging that our designs and experiments are "convincing" and "thorough". We sincerely appreciate all constructive suggestions that help us to further improve our work. To address reviewer mayu's questions, we provide point-by-point responses below.

[CONS1 Baseline Score] We respectfully argue that our setting is fair and reasonable from a training cost, training supervision, and implementation detail perspective, with further analysis of possible reasons for the difference in baseline score.

[High computation cost and model convergence observation] Our training setting is carefully designed by calculating the training cost and experiment to examine model convergence. For example, it costs about $1093.93$ A600 GPU hours to train a vanilla transformer-based SMoE model with 32 experts on the full OPUS-100 dataset (107M sentence pairs) for 500k steps (as in the CLSR [1]) with 8192 max tokens per batch (in our setting). Meanwhile, we observe that the models converge much faster than at the end of training. First, we perform a preliminary experiment training $*GS-SMoE*$ and $*Dense*$ in the same setting as above for 10 epochs (about 918k steps). We find that both the dense and SMoE baselines converge quickly at 100K steps, and then the loss decreases at a much slower rate. Thus, we ultimately design the 200K training steps for OPUS-100 datasets to reduce the training cost while obtaining a relatively well-trained model. A similar observation applies to OPUS-16/50 as well.

[Implementation details and comparison] To ensure a fair comparison, we reimplement all baselines in our code base, which is based on the MoE branch of fairseq (https://github.com/facebookresearch/fairseq/tree/moe). All models are trained and evaluated using the same settings. This also answers the reviewer's question about our implementation details.

[Other possible reason] Aside from training time, there are a few possible reasons for the difference in BLEU score, such as training framework, tokenization, and evaluation metric. For example, CLSR [1] uses tensorflow as the training framework. Some other works use different tokenizers and BLEU score implementation [2]. In particular, we adopt the same evaluation settings as in SCoMoE [3].

[1] Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation

[2] No Language Left Behind: Scaling Human-Centered Machine Translation

[3] SCoMoE: Efficient Mixtures of Experts with Structured Communication

[CONS2 More Baseline Experiments in OPUS-50/100 Settings] Thanks for the question. This could be explained by the results of OPUS-16 and our experimental design. We will also implement $*ST-SMoE*$ in OPUS-50/100 and include it in the final version of the paper.

[Experiment design] First, we consider a powerful baseline in the experiments with more languages. From Table 1, the higher performance of $*Residual-SMoE*$ over dense and other SMoE baselines proves the effectiveness of adding a shared expert for the token-routing SMoE model. Although our $*LGR-SMoE*$ performs better, we consider it as a backbone to incorporate and testify the residual technique on language-guided routing mechanism, which is named $*LGR*^{res}*-SMoE*$. Then, we experiment with the two language-guided routing baselines ($*LGR-SMoE*$, $*LGR*^{res}*-SMoE*$) as shown in Table 2.

[SMoE baseline selection] From Table 1, the average BLEU score of $*ST-SMoE*$ falls behind $*Residual-SMoE*$, so we focus on examining the most competitive baseline technique, Residual expert, in more language settings.

[Further experiments.] To address the reviewer's concerns, we will train and evaluate $*ST-SMoE*$ on more OPUS-50/100. However, running the full data training costs about $436.25$ A6000 GPU hours. Due to the limited time in the rebuttal period, we will include it in the final version.

Dear Reviewer mayu,

We thank reviewer mayu for the time of reviewing and the constructive comments. We really hope to have a further discussion with reviewer mayu to see if our response solves the concerns.

In response, we have (1) clarified our baseline implementations; (2) presented additional experimental results and analysis of dynamic routing; (3) improved visualization results;

We genuinely hope reviewer mayu could kindly check our response. Thank you!

Best wishes,

Authors

Dear Reviewer mayu,

We thank reviewer mayu for the time of reviewing and the constructive comments. We really hope to have a further discussion with reviewer mayu to see if our response solves the concerns.

In response, we have (1) clarified our baseline implementations; (2) presented additional experiments about more benchmark and dynamic routing; (3) improved visualization results;

We genuinely hope reviewer mayu could kindly check our response. Thank you!

Best wishes,

Authors

We are very glad and appreciate that reviewer BJsk had a positive first impression. To address reviewer BJsk's questions, we provide point-by-point responses to your concerns as follows.

[CONS1 Baseline Notation] Thank you for the helpful suggestions. Due to space limitations, we add a table comparing the difference between baseline models and our methods in Appendix Table A7. Also, we will find a better abbreviation for each baseline and unify the expression throughout the paper in the final version.

[CONS2 The Definition of Language Complexity].

Thanks for the feedback. We revise and explicitly mention that we use data availability to indicate language difficulties in the methodology section of the modified draft. Revisions are highlighted in orange.

In addition, we implement dynamic routing considering both grammar complexity and data amount, whose performance on OPUS-16 is shown in Table S1. Specifically, we group languages according to the same ratio of high/medium/low resource language numbers, with the language difficulty metrics that add grammar complexity scores (rated by GPT-4 from 1-5) and data abundance scores (0,1,2 for high/medium/low resource). Then we train $`Lingual-SMoE`^{+grammar}$ with language difficulty groups and compare it to the dense and SMoE baselines in OPUS-16 settings. As shown in Table S1, it consistently outperforms the dense and SMoE baselines. However, due to the subjective nature of grammar complexity, we choose not to include it in our main content.

Table S1 Performance comparisons of $`Lingual-SMoE`$ with dynamic routing according to language difficulty defined by grammar complexity and data abundance, compared to $*Dense*$ and vanilla SMoE $*GS-SMoE*$. Average BLEU scores for each direction are reported.

[More References] Thanks for the helpful references. We have included these papers in the revised draft. Specifically, we cite these papers in the "Routing Designs and Expert Capacity in SMoE" paragraph in the Related Work section, highlighted in orange.

[CONS3 Dynamic Capacity for Hard and Medium Languages] Thanks for the detailed review. This is a plotting issue. We have updated Figure 5 in the modified draft and switched the medium and low in the legend.

[QUESTION1 Effect of Dynamic Expert Allocation (DEA) on Different Datasets]

We respectfully point out that the DEA has consistently improved in scale across OPUS-16/50/100. The reviewer's observation could be attributed to the implementation difference between DEA in OPUS-16 and OPUS-50/100, and the table layout.

[How to calculate the performance gain of DEA] First, DEA is introduced to further improve the Language Guided Routing (LGR) design. Since we find that adding a shared expert is helpful for SMoE MMT, we implement a variant of the original $*LGR-SMoE*$ model called $*LGR*^{res}*-SMoE*$ by replacing one of the input-specific experts with a shared expert. Then, DEA is added to the best model with LGR, which is $*LGR*^{res}*-SMoE*$ and $*LGR-SMoE*$ in OPUS-50/100 setting. Thus, the improvement in DEA could be calculated by subtracting the BLEU score of $*LGR*^{res}*-SMoE*$ from $`Lingual-SMoE`$ in OPUS-16, and that of $*LGR*$ from $`Lingual-SMoE`$ in OPUS-50/100.

[Specific performance gain of DEA on OPUS-16/50/100] Thus, the gain of DEA in average BLEU is $0.1\%$ for OPUS-16, $0.09\%$ for OPUS-50, and $0.17\%$ for OPUS-100, which is $0.12\%$ on average.

[QUESTION2 Overfitting Problem on Low-resource Language for Dense Model] Yes, the dense MMT model also suffers from overfitting on low-resource language, as shown by many papers on MMT with dense model [1,2]. In particular, [3] compares the dense and SMoE model overfitting existing in dense models, showing that this problem is more acute in SMoE model than the dense model, which is also the motivation of our DEA design.

[1] Neural Machine Translation for Low-resource Languages: A Survey

[2] Meta-Learning for Low-Resource Neural Machine Translation

[3] Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

[QUESTION3 Model Efficiency]

We wonder whether the "inference implication" mentioned by the reviewer means inference efficiency, or the trade-off between speed and performance, as discussed in many studies of SMoE [4-6]. If the "inference implication" is inference efficiency, we record the inference average tokens processed per second (token/s) of $*GS-SMoE*$ and $`Lingual-SMoE`$ on the same test set. The inference token/s for $*GS-SMoE*$ is $469.84$, and that of $`Lingual-SMoE`$ is $456.83$. Compared to vanilla SMoE, our model achieves considerable improvement with marginal computational costs.

[4] Task-level Mixture-of-Experts for Efficient Inference

[5] MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

[6] Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers