MOEfication by Experts as Masks

• The proposed method requires additional training samples for fine-tuning, with the data collection process being non-trivial and involving the incorporation of multiple datasets.
• The main comparison focuses solely on MoEfication, which may be insufficient to fully highlight the advantages of the proposed method. There are numerous expert pruning and merging techniques that could be adapted to your setup. Including more recent baselines would strengthen the experimental comparisons.

• Activation ratio as a metric used for comparing the proposed method with baselines is inappropriate. FLOPs of the forward pass of the entire model would be more appropriate, as they take into the account the costs like routing networks. As such, the comparison between pruning and the proposed method seems highly unfair and misleading to me.
• The paper is lacking any wall-clock time measurements of forward pass execution of the model. The reduction of number of non-zero weights or activations does not necessarily translate to speedup of inference time [8, 12]. The advantage of MoE-based methods was that all experts were homogeneous and thus relatively easily parallelizable on contemporary hardware like GPUs. By making each expert of different size the authors are giving up the hardware effectiveness. This is especially true for FH, FW and FHW variants of the proposed method, as they resemble unstructured pruning. Unless the authors present an efficient implementation for GPUs that works on batched inputs, the method will appear as unpractical and may be of little interest to the community.
• The evaluation is extremely weak as the proposed method is compared to only a single baseline (MoEfication). Since there is no theoretical contribution and this is mostly an empirical work, one would expect a thorough empirical evaluation, e.g. at least 3 recent baselines. There are multiple works that may be appropriate for this [4,9, 10, 11].
• The evaluation on larger models is limited to loss only. Does the performance on downstream tasks collapse when compared to a dense model?
• The paper's main contribution is not explained in enough detail. The authors write "For independent experts, we introduce a routing mechanism that selectively activates experts, following the standard MoE routing strategy.", but other than that the "standard MoE routing strategy" is not described anywhere. Are routers trained end-to-end, or like in MoEfication? Are they trained simultaneously with the masks $v$? What is the architecture of the router (depth, width)? Is Top-$k$ used? If yes, how is $k$ set? How is a MoE layer defined - are the outputs of each expert weighted by the output of the router (like in [5]) or are they simply added together (like in [6])?
• Similarly, the description of the experimental part is also sparse and lacks details. What were the hyperparameters used for the proposed methods, and what were the hyperparameters used for MoEfication? Which variant of MoEfication ([6] proposed multiple) was used? E.g. since granularity is crucial for the performace of MoE-based models [7], what was the expert size for MoEfication?
• Code for the experiments has not been provided, limiting the ability to verify and reproduce the findings of the paper, or to enhance the reader's understanding of the method and of the experimental setup.
• The proposed method, similarly to [6], transforms a dense model into a sparsly activated model. Since a crucial component of the method is the $L_0$ regularization from [13], the novelty of this work appears limited.
• Authors claim to "propose the concept of “activation pruning” (line 139)". However, similar terms like "activation sparsity" and "dynamic pruning" - that refer to basically the same concept - have been used since at least 2019, and a vast amount of literature on this topic exists [1,2,3]. The authors discuss weight pruning (and MoE) in the related work section, but do not cover activation sparsity literature, which may be even more relevant to this work than weight pruning. Similarly, the authors fail to compare their method to any activation sparsity method, e.g. the work of Mirzadeh et al. [4].
• The writing, spelling, and the grammar in the paper should be significantly improved. For example in line 70: "through this mechanism, we can: (1) Adaptively learn which dimensions to share, token-specific, or prune." ("which dimensions to token-specific"). Another example from line 508: "scaline law". Line 157: "basic masking method by selecting expand intermediate dimensions in the FFN". Line 190: "Here, the goal is to learn mask matrices that select sub-dimensions corresponding to specific tokens, while still maintaining overall model performance.". Line 257: "For continue pre-training process". Line 296: "separatedly". There are many more examples thoroughout the paper. As a result the text is difficult to read.

Minor:

• The authors write "the optimization objective for each layer is defined as..", while Eq 5 sums over every layer.
• There is no reference to Figure 1 anywhere in the text.
• Y-axis label and title in Figure 2c appears to be incorrect.
• The "Mem"/"memory usage" in Table 1 is completely vague and should be clarified.
• Line 366: "As the compression rate decreases, maintaining model performance becomes increasingly difficult." I think the authors meant "increases" here.

1. Clarity and Description Issues

   • The experimental results lack detailed and clear descriptions, e.g., the model used in Table 2 is not specified.

   • Some expressions are unclear and hard to follow:

     • Line 158: “expand intermediate dimension”

     • Line 507: "reducing the total parameters which may not against the spirit of scaling law" contains a grammar error, and does not make a clear point.

2. Methodology Design Concerns

   • Training Scheme of Learnable Masks: The MoE model uses different experts (masks in this paper) for each token. The learnable channel-wise mask is parameterized and trained. Does this mean we need to train a mask for each token in the vocabulary? This would require training on a vast number of tokens, consuming significant resources.

   • Generalization across Datasets: According to Section 3.1, the masks are trained on a collection of datasets. However, it is unclear whether these masks can generalize to unseen tasks or prompt sets.

3. Insufficient Experimental Results for Larger Models

   • The author did not specify the model used in Table 2. According to the description, the LLaMA3 model is only mentioned in Figure 3, not Table 2. Performance comparisons with the original model and baseline methods should be included to ensure performance preservation.

The primary issue I see in the paper is the matter of applicability of this method in any practical setting. Mixture of Experts is widely used, and multiple properties of MoE design are there for good reasons. The paper does not refer to the multiple properties of MoE that were sacrificed when designing MoM.

1. Wall-clock time versus activated parameters/FLOPS. Mixture of Experts usually uses experts of equal size. While this limitation can be lifted and the quality of the model may be improved with great gains in terms of theoretical FLOPs, the speed on real-world hardware may suffer tremendously with varied-sized experts. The authors don't show any results measuring wall-clock performance of their model. Going by my experiences, the actual inference time may be worse than even the dense baseline (on average), let alone the MoE model. For the method that focuses on faster inference with the same number of parameters, that is a critical matter, but no measurements are available in the paper. I think both the wall-clock time of batched training and the wall-clock time of inference (batched or unbatched, preferably both) should be included in such a paper. The expectation is that inference time will be lower than other techniques (higher training cost could be acceptable if inference numbers were good).
2. Activated models in the whole model versus just FF. Usually, in MoE, the Feed-Forward layer is kept at the same number of FLOPs with an expanding number of parameters. MoM is doing something different, keeping the number of parameters while reducing FLOPs. However, this approach generally has a low ceiling for potential improvement, as the attention block in Transformer usually accounts for around a third of model FLOPs. Following that, removing even the whole FeedForward network may result in a maximum speed-up of 3x. Authors, however, report the number of activated parameters (or compression rate) just for FeedForward layer, not the whole model. This issue will further reduce the applicability of MoM in the real world, especially in connection with issue number 1.

Other weaknesses of the paper:

3. MoM versus baseline of pruning. The baseline of pruning the model, shown in Figure 4, seems to achieve better performance than MoM, although with a worse activated parameters ratio. While it is possible that MoM is still better at a Pareto-frontier, this result is not very useful. I would suggest training possibly a couple of MoM and pruning models with a variable weight of L0/L1 loss (that is, modify the final loss to $L_{lm} + \alpha * L_{mask}$, and vary the alpha). Then scatterplot those experiments with axes "final activated ratio" and "final loss", to show if pruning or MoM seems to be better at a Pareto-frontier. While I understand that this may require more experiments, the current comparison of MoM to pruning is of little value.

Minor point: the paper could benefit from improved writing in all sections. However, while some sentences gave me a pause, it is generally understandable. A few excerpts to show what I mean, just from the very first page of the paper: lines 34-37 "(...) Mixture-of-experts approach, which designs multiple expert structures with extensive parameters but activates only a subset during computation." - MoE does not design, "expert structure" is kind of an awkward phrase, "extensive number of parameters", "subset of them during processing" (or "forward computation"). The same lines in the caption of Table 1 - "Mem indicates memory usage" column should have values "high/low", not "check"; or it should be renamed to "low memory usage." Also, I believe the paper's title should start with "MoEfication" with a lowercase "o". While the whole paper could be improved in terms of writing, the text is understandable nonetheless, and it has had little impact on my (at the moment) negative recommendation.

See questions below.