PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

Weakness 5 about More Experiments:

Thank you for pointing out the possibility of introducing more experiments.

1. Regarding the comparisons, we’ve already taken 2 PEFT solutions in their popular settings (LoRA for QV matrices in attention and parallel adapters) as our baselines. All these baselines serves as “MoE-Agnostic PEFT” methods as the design choices within our unified framework of PEFT for MoE (See Section 3.1.2 and Figure 2).
2. We admit more experiments could be carried out on more LoRA variants. We are running experiments with more settings on router or/and expert weight matrices, and results will be provided soon in our next revised version.
3. Regarding testing on more MoE LLMs we have pointed out in our response to your Weakness #3 that rather than exhaustively testing every emerging variant, we believe our current focus better serves the field's current needs. Our choice of base models reflects careful consideration of the current research landscape. The reason we implemented our experiments on OLMoE-1B-7B and Mixtral-8$\times$7B is because they are the state-of-the-art MoE models with the best performance/cost ratio (according to MMLU performance reported in OLMoE’s report [2]) under the activated parameter scales of ~1B and ~10B. We believe these two models can be reasonably selected as adequate “standard” representative on the current landscape of mainstream MoE designs.
4. Implementing our framework on all mentioned models requires several magnitudes more significant computational budget for scaling and getting observations along the design dimensions. Without an infinite amount of budget for experiments, we believe that instead of exhaustively experimenting every MoE base models, focusing on more thorough and deeper exploration of design choices within our own frameworks with several selected representative models is better for providing empirical findings that is more useful to the community.

Weakness 6 about Lack of Theory:

We appreciate your feedback. However, we have to respectfully refrain from your assessment regarding theoretical requirements, and address that from our perspective, ICLR has never historically been, and indeed shall not be, a platform where only works with substantial theoretical analysis and support could be appreciated.

• Among the list of previously accepted submissions from ICLR’24, only ~24% mentioned “theory / theoretical” in their title and metadata, a ratio similar from those mentioning “framework” (~28%) or “empirical” (~20%). This clearly indicates that the machine learning community has long embraced a more inclusive view of meaningful contributions than what you suggested.
• In fact, in rapidly evolving areas like LLMs, it's worth noting that purely empirical technical reports have consistently driven the field's most significant advances. While theoretical understanding certainly has its place, it has typically followed — rather than led — the practical breakthroughs that have actually moved the field forward.
• Your opinion also seems to overlook established precedents in our field. He et al. (ICLR'22) [1], for instance, have successfully introduced frameworks in a similar way, by exploring design dimensions of PEFT through comprehensive empirical analysis. Their methodology - proposing a framework and methodically exploring design combinations - has proven remarkably effective at uncovering valuable insights, much like our approach.

As clearly articulated in our introduction (lines 90-97), our work delivers concrete contributions through a systematic unified framework (Figure 2), novel insights into previously unexplored dynamics between (Figure 3), and all the empirically-validated design principles across multiple dimension (Section 4.2 & 4.3). Therefore, we maintain our position on this matter, that the practical insights and empirical rigor we provide are enough to make meaningful contributions to the understanding of MoE and PEFT, and align precisely with what the community has consistently valued.

Dear Reviewer T3WA,

Thank you for your response! We appreciate it and would like to further clarify some points in our revised paper to help address your concerns.

About novelty:

The novelty of our work actually extends far beyond just being the first unified framework for PEFT in MoE. We systematically explore possible designs for PEFT for MoE, and we especially discover the previously underexplored dynamics between vectors in routers for PEFT and FFN experts.

1. The dynamics between vectors in FFN experts, FFN expert router and PEFT expert router (visualized in Figure 3) provides a novel and intriguing understanding of how the vectors in the router for PEFT experts can interact with the vectors in the original router and the FFN experts [line 209-238]. This brings the count of possible routing patterns to combinatorial numbers, which create much more exploration of hidden state patterns to be stored as vectors in routers. This insight suggests that routing mechanisms can be leveraged not just for FFN expert selection, but also as a important integral part of the PEFT process itself, an insight that motivated our design and could also have significant implications for future MoE and PEFT research.
2. We provide a systematic study of how the PEFT modules can interact with MoE’s routing mechanisms. We break down the detailed designs within PEFT and MoE, re-frame the possible solutions under functional and compositional strategies, and define a complete set of design dimensions along which different methods can vary. Any future method in this research direction which operates along these dimensions identified in our framework should benefit from our practical results in Section 4, which helps explore these design dimensions.

About the applicability of our findings:

Thank you for your concerns about applicability of our findings. We’d like to point out that in our revised version, we reorganized our core results to better demonstrate our observations and findings that consistently generalize across all settings, domains, and models we’ve experimented.

We would also like to further clarify the previous misunderstanding about task-dependent configurations, which refers only for the sensitiveness of configurations under our observation 3 (see below) and calls for need of hyperparameter tuning in this case, instead of addressing all our results.

In our [Section 4.2 Experiment Results], it is consistently observed that

• Observation 1. PERFT-R emerges as the best strategy in terms of being able to achieve the best performance with extreme activated parameter efficiency.
• Observation 2. PERFT-R and PERFT-E generally benefit from scaling up. For each configuration, the optimal settings can be found via scaling along the activated parameter efficiency.
• Observation 3 (3 to 2). PERFT-R is more sensitive to the overall number of PEFT experts rather than the activated. Changing the total number of PEFT experts leads to more significant variation in performance, over changing the activated number of PEFT experts.

We further analyze these observations, and summarize in [Section 4.3 Result Analyses] that:

• Finding 1: Routing is (very!) important in scaling the number of PEFT experts. Without routing for PEFT experts leads to severe degradation of performance.
• Finding 2: Routing contributes more from its weight distribution, rather than sparse activation (expert selection). Even PERFT-R with no sparsity can perform well, and sometimes even better.
• Finding 3: With more PEFT experts, PERFT-E can become favored over PERFT-R, as PERFT-E sacrifices the flexibility (learning a independent routing for PEFT experts) for better stability (using the original routing pattern well-pretrained for FFN experts).

These observations and analyses are not merely based on the selected results presented in Section 4, but also from the comprehensive results in tables (Table 4,5,6,7,8) provided in Appendix C, with >160 PEFT settings being independently fine-tuned and >1100 evaluation scores reported across 14 benchmark datasets, which are enough for drawing to our conclusion. All the aforementioned empirical findings are able to generalize across all our experiments, and are directly applicable for the community in their designing similar PEFT and MoE approaches.

We hope these clarifications can help address your concerns and better demonstrate the value of our contributions. Thank you for your time and effort!

Thanks for the reply. The provided trial and error approach by overfitting to the specific test data does not sound a solid method. Thus, even though there are lots of experiments conducted, I am sticking to my original rating of 5 due to lack of insights.

Weakness about Novelty:

We would like to clarify that it is exactly because the adapters and routing mechanisms have been independently validated that motivated us to propose our unified framework. We aim to contribute to the community by proposing our unified framework covering ALL specific design dimensions (functional / compositional strategies) and possible choices on them that anyone would face when designing PEFT for MoE.

• There hasn’t been any previous work applying PEFT to MoE models, and we try to not only be the first methods studying this, but to do so systematically.
• Our work is beyond a simple combination of MoE and PEFT, as we seek to build a framework for all possible design choices, and when doing so we also managed to reveal the intriguing dynamics behind key memory vectors in experts and expert vectors in routers for both FFN & PEFT experts (as shown in Fig. 3 & 6) which provides insights for fine-tuning MoEs that have rarely been discussed in such detail in previous MoE studies.
• By combining representative designs under this framework, we proposed our PERFT family. We thoroughly experimented our PERFT variants with an exhaustive list of possible designs, and presented results with empirical observations we identified in our experiments across settings, domains, and models. No MoE fine-tuning has been experimented at this scale before, hence we believe our findings could facilitate the community understanding PEFT and MoE.

Our approach mirrors how He et al. (ICLR’22) [7] have established their unified view of PEFT, where they also explored possible design dimensions and carried out their experiments by combining different design choices. In our work we examined our design choices in a much larger scale and a much more thorough way.

Weakness about Increased Complexity:

The reason that motivated us to propose and study PERFT-R with a routing among introduced PEFT modules in addition to the original routing between the FFN experts is based on several observations:

1. In Section 3.1.1 Functional Strategy, our framework has identified this design of introducing routing among PEFT experts as a possible choice in designing PEFT for MoE models.
2. Recent works on PEFT for dense models (Section 2.2 PEFT with MoE-like Structures) have studied numerous similar MoE-like PEFT methods which has shown improvements, suggesting that in MoE models the design of routing among PEFT experts might also be a highly competitive choice to consider.
3. We demonstrated a potential of the benefits from the dynamics between key-memory vectors of FFN experts and expert vectors in routers for both PEFT experts and FFN experts in our Figure 3.
4. In our experiments we verified that PERFT-R with its routing among PEFT modules is, as expected, generally able to achieve the best performance across domains and different activated parameter settings.

You mentioned that “a peft model can be learnt for an individual expert and these modified experts be combined in a new mixture”, and this is analyzed in Section 3.2 as the equivalence between PERFT-E and PERFT-R, by which we can view PERFT-E as a simplified special case of PERFT-R that utilizes the original router’s routing patterns instead of learning a independent router (See Eq. (8) at line 288) . This can only be achieved when the number of introduced PEFT experts $M$ matches the number of FFN experts $N$, and the activated PEFT experts matches the number of activated FFN experts as well, which restricts its flexibility as leaving the bottleneck size $D_B$ as the only scaling factor available. In contrast, PERFT-R allows arbitrary settings for selecting Top $K$/$M$ PEFT experts to activate. We compared and analyzed the experiment results of PERFT-R and PERFT-E in Section 4.3 line 479-514.

The difference in the computational cost during training and inference is controlled in our experiments via “# Act.” and “% Act.” in Table 1, and “activated parameter efficiency” in Fig. 4 & 5. We compared different methods with similar scale of activated parameters, which directly translate to FLOPs incurred during both fine-tuning and inference.

For the memory cost, as all our methods are within the scope of Parameter-Efficient Fine-Tuning methods, the total amount of parameters introduced should be marginal in comparison to the total amount of parameters in the MoE base model. In our experiments we kept our largest experiments under the scale of <1%, which means no special consideration of memory advantage is needed for detailed discussion.

Dear Reviewer hrjf,

Thank you for your feedback. We appreciate it and would like to further clarify some points in our revised paper to help address your concerns.

• Regrading the previous Line 31:

  Regarding your original claim about MoE model’s efficiency

  Mixture of Experts models do not significantly or necessarily reduce computation costs since routing methods may be complex and may not optimize and generalize trivially, and also significantly increase memory costs for both training and inference.

  We’ve already removed these parts in our revised version, since they only served as background introduction rather than an integral part in our framework, and it is not us who made these claims. This part is currently modified as

  With so many new MoE LLMs available, how to effectively fine-tune them for downstream tasks has become an area of considerable value.

  We still like to point out that your previous opinion in this matter seems in the contrary to the current understandings in both academia and industry. The claim of efficiency we previously cited has been explicitly demonstrated across numerous works [1-9], including technical reports of almost EVERY MoE LLMs ever released, published conference papers, and well-validated works with >1k citations, even in early studies you provided, like Sparsely-Gate MoE [1] and GShard [2]:

  Sparsely-Gate MoE [1]: For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total. For our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU … Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA.

  GShard [2]: We … demonstrated a 600B parameter multilingual neural machine translation model can efficiently be trained in 4 days achieving superior performance and quality compared to prior art when translating 100 languages to English with a single model. In addition to the far better translation quality, MoE Transformer models trained with GShard also excel at training efficiency, with a training cost of 22 TPU v3 core years compared to 29 TPU years used for training all 100 bilingual Transformer baseline models.

  Mixtral [3]: Mixtral only uses 13B active parameters for each token. With 5x lower active parameters, Mixtral is able to outperform Llama 2-70B across most categories … The memory costs for serving Mixtral are proportional to its sparse parameter count, 47B, which is still smaller than Llama 2 70B.

  DeepSeekMoE [4]: Evaluation results reveal that with only about 40% of computations, DeepSeekMoE-16B achieves comparable performance with DeepSeek-7B, … DeepSeekMoE-16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2-7B, which has approximately 2.5 times the activated parameters.

  QwenMoE [5]: … we have observed a remarkable reduction of 75% in training costs when using Qwen1.5-MoE-A2.7B in comparison to Qwen1.5-7B … Qwen1.5-MoE-A2.7B model exhibits an impressive improvement in speed, being approximately 1.74 times faster compared to the Qwen1.5-7B model. This acceleration is primarily attributed to the fact that the MoE architecture activates a notably smaller portion of its total parameters, thereby reducing computational demands.

  Grok-1 [6]: … Grok-1 displayed strong results, surpassing all other models in its compute class, including ChatGPT-3.5 and Inflection-1. It is only surpassed by models that were trained with a significantly larger amount of training data and compute resources like GPT-4. This showcases the rapid progress we are making at xAI in training LLMs with exceptional efficiency.

  JetMoE [7]: This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, … JetMoE-8B outperforms the Llama2-7B model, and JetMoE-8B-Chat outperforms the Llama2-13B-Chat model, demonstrating that LLM training can be much more cost-effective than generally thought. In addition, JetMoE-8B has 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B.