On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

• Limited novelty when stacked against recent related work The main contribution of this work - convergence analysis for three HMoE variants - is highly incremental, as the main components used in this analyses (such as Voronoi losses) were already introduced in [1], while the Laplace gating for MoEs was introduced in [2]. Application of apparatus from [1] and [2] to HMoE instead of MoE is the only novel contribution of this work.
• Empirical evaluation is not entirely aligned with the theoretical contribution of the paper I find most of the experiments in the empirical part of the paper confusing when placed in context of the theoretical contribution. The theoretical part demonstrates the benefits of using Laplace-Laplace gating over the other variants, and I would expect the empirical part to be focused around supporting the theoretical findings. Instead, it appears the authors focus on showcasing the performence of their final HMoE-based model. Only Figure 2 compares the HMoE SS/LS/LL variants, while the rest of the empirical evaluation compares a single HMoE-based model to different multi-modal methods.
• Unconvincing evaluation As such, I am not entirely convinced that Laplace-Laplace gating offers consistent advantages also in the real-world, deep model setting. In line 521 the authors write: "best-performing Softmax-Laplace gating combination of HMoE in latent domain discovery". Does this mean that in some cases Laplace-Laplace has inferior performance? The authors should report the score of every variant also in Table 2 and Table 3.
• Unclear purpose of empirical analysis Similarly, the analysis from Figure 3 has the same issue. How does the choice of gating functions affect the token distribution? Would token distribution of a non-hierarchical MoE be significantly different? Without such a comparison the analysis is somewhat informative, but not relevant to the theoretical contribution of the paper.
• Limited impact In the introduction the authors mention the rising use of MoEs in modern foundational models. However, none of these use hierarchical routing, and the last work to tackle non-Gaussian HMoE was [3]. Given the community's lack of interest in HMoEs, the impact of this work may be limited.
• Incomplete description of empirical evaluation Every training hyperparameter should be provided for each experiment performed in the paper. The appendix describes the datasets used and data preprocessing only. The provided code is missing some of the experiments, e.g. for ImageNet. How many seeds were run for calculating the reported standard deviations?
• Unfair comparison to MoE models In some of the experiments the score of HMoE-based model is compared to non-hierarchical/standard MoE-based model. However, two levels of routing means two routers are used, resulting with HMoE having an effectively double the computational and memory resources spend on routing compared to MoE. How can be anyone sure that any gains in performance are due to the hierarchical nature of routing as opposed to additional computation and memory allocated for routing? Some recent works does utilize deeper routers, hinting that increased computation allocated for routing does help performance [4, 5].

References:

[1] Nguyen, Huy, TrungTin Nguyen, and Nhat Ho. "Demystifying softmax gating function in Gaussian mixture of experts." Advances in Neural Information Processing Systems 36 (2024).

[2] Han, Xing, et al. "Fusemoe: Mixture-of-experts transformers for fleximodal fusion." arXiv preprint arXiv:2402.03226 (2024).

[3] Shazeer, Noam, et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." International Conference on Learning Representations. 2016.

[4] Chi, Zewen, et al. "On the representation collapse of sparse mixture of experts." Advances in Neural Information Processing Systems 35 (2022): 34600-34613.

[5] Zhang, Zhengyan, et al. "MoEfication: Transformer Feed-forward Layers are Mixtures of Experts." Findings of the Association for Computational Linguistics: ACL 2022. 2022.

• While the paper focuses on enhancing HMoE’s performance with alternative gating mechanisms, it lacks a detailed discussion on the associated computational complexity. A deeper analysis of the model’s resource demands and potential methods to mitigate these challenges would add valuable context, particularly for large-scale or resource-constrained applications.

While the theoretical work is very detailed, and has an interesting end result, I wasn't able to follow the arguments in detail, though I am not an expert in this sort of convergence theory. To me, the main text of the theory section is a little too verbose and overly technical, especially since there is already a large appendix with details. I've listed a number of questions below, but in particular:

• I didn't see a clear summary sentence of what the results mean, along the lines of "This shows that n samples are required in order for the best MLE params to be within $O(n^{-1/2})$ of the true distribution's parameters, up to log factors".
• Some of the equations seem unnecessary for a high-level understanding, e.g. I don't see a need to list the entire vornonoi loss function L in the main text, and also don't see a need for the "mixing measure" $G_*$. The notation is also pretty busy, with lots of subscripts and hats/tildes/etc. This makes it more difficult to read, and would be good to simplify as well.

For the experimental application, I didn't see an estimate of model size or computation ops in any of the comparisons, so it's hard to know if the model performed better or is simply larger than the baselines. In particular, comparing against non-hierarchical MoE, all levels of gating should be included in op and param counts.

I also didn't see a section that corroborates the theoretical convergence rates experimentally on toy gaussian data used in the theorems. Seeing measurements that converge at the described rates would make the theory more convincing and relatable to a wider audience.

• The main weakness of the paper is the writing being very unclear. I'll list some examples below:

  • Jumbled transitions between sections. Example in line 126, where we suddenly talk about the "convergence of expert specialisation"(which is only again mentioned line 260) before transitioning to an entirely different topic. Another example is line 272 "moving from this section" while it is not clear at all what we (the reader) should take away from this section.

  • The mathematical notations are cumbersome and only briefly introduced

    • some notations could be omitted for the sake of readability (e.g. the $\ast$ notations in Equation 1)
    • Some notations could be simplified. For instance introducing random variables as $X$ and $Y$ in lines 108 but then only referring to their density $x$ and $y$
    • Overall, I find that the methods section could be shortened by (i) moving some descriptions in appendix, (ii) only underlying the key take-away and (iii) making notations more lightweight.

  • Some statements/motivation are not substantiated:

    • line 71: "there is a growing demand for models that can deliver accurate and individualized predictions for each subgroup. Therefore, it is worthwhile to study. HMoE, which can leverage the intrinsic information within complex input structures and achieve superior performance on corresponding tasks." -> I do not see.a good reason why HMoE are a better solution to that problem than e.g. increasing the number of experts in MoE, or a deeper structure, or any other techniques to make the decision space "more complex"

  • Limitations in experimental setting:

    • Some details are missing: such as how many experts are used, or even better, considering different number of experts. For the image classification benchmarks, it's very unclear how a "one-layer MoE module" are placed in the sparse MoE backbone: e.g. what is being fine-tuned ?
    • Since the propose HMoE has a deeper routing mechanism, the paper should also discuss efficiency of the method vs MoE when integrating the new routers.
    • I find the description of the experimental section in the abstract/intro quite misleading. First most of the experiments are on clinical data, which is different from diverse real-world scenarios (line 19 + 535). As for image classification, CIFAR or tiny-ImageNet are interesting benchmarks, but a bit far from real world tasks.