FloE: On-the-Fly MoE Inference on Memory-constrained GPU

Dear reviewer ZXAh,

We sincerely appreciate your recognition and valuable suggestions. Below, we summarize and respond to the issues, suggestions, and questions you raised.

[Minor Issue 1] The absence of Mixtral-GPU performance results in Section 3.2. While HQQ Int2 seems to reflect the same setting, the authors should clarify this explicitly.

[Response] We sincerely appreciate you pointing out the issue with our inconsistent terminology. Indeed, HQQ-INT2 refers to the performance of Mixtral-GPU on downstream tasks. We utilized HQQ to quantize the experts to INT2, enabling the entire Mixtral-8×7B model to fit within GPU memory. In future versions of the paper, we will ensure the consistency of the configurations for HQQ-INT2 and Mixtral-GPU by clearly stating their alignment.

[Suggestion 1] I strongly encourage the authors to include code assets during the rebuttal period.

[Response] We genuinely hope that our efforts can contribute to the community by facilitating the deployment of MoE models on consumer-grade GPUs and edge devices. We have provided the current version of our code via an anonymous link below Code Link of FloE, and we plan to open-source it after completing the adaptation for more models.

[Suggestion 2] Figures 6–8 use color schemes that are difficult to read. Additionally, Figure 8's caption contains a missing space.

[Response] For later versions of the paper, we will refine the color schemes and captions for Figures 6-8.

[Question 1] What is the latency of the predictor described in Section 2.3?

[Response] We measured the latency of the predictor over 500 forward passes and calculated the average values. The results show that the inter-expert sparse predictor has a latency of 0.11 milliseconds, while the intra-expert predictor's latency is 0.27 milliseconds. In comparison, the average execution time for a single FloE Transformer block is 5.74 milliseconds. These latencies account for only 1.9% and 4.7% of the total execution time, respectively, indicating that they have almost no significant impact on the model's generation speed.

[Question 2] Given that the proposed quantization method outperforms existing approaches, could it also be applied to dense models, particularly due to the similarities with the SwiGLU module?

[Response] The evaluation methods for quantization and sparsification sensitivity are equally applicable to dense models. The theoretical proof presented in Appendix B is also generalizable and can be applied to MoE models. These are our preliminary results on Mistral. Moving forward, we plan to further explore how to compress the size of MLPs in dense models to improve inference speed and reduce deployment costs. This will be one of the key directions for our future work.

Looking forward to hearing from you!

Best regards,

Authors

Code Link of FloE:

https://anonymous.4open.science/r/floe-5843

Results Table:

Quantization Sensitivity of Mistral-7B

Sparsification Sensitivity of Mistral-7B

Dear reviewer bhnT：

Thank you for recognizing the importance of the research problem, as well as the soundness of our methodology and experiments. Below, we address the key concerns regarding MMLU performance and comparisons with non-compressed baselines:

1. FloE achieves competitive performance on MMLU.

As you noted, MMLU performance is relatively sensitive to sparsity, which is reasonable given its complexity as a large-scale, multi-task NLP benchmark covering 57 subjects. Similar sparsity-sensitive trends are observed in other challenging tasks such as GSM8K (math reasoning) and HumanEval (code generation), which aligns with the observations in Sirius [1].

However, FloE demonstrates best performance retention compared to other baselines (see Figure 10 of original manuscript). Specifically, as shown in the table below, at 90% sparsity, CATS almost completely fails on these three tasks, achieving an average score of only 16.2, while FLoE maintains a score of 43.8. At 80% sparsity, FLoE retains over 93% of the base model's capabilities.

2. Non-compressed baselines exhibit limited performance.

To address your concerns, we have used Mistral-7B and Llama-3.2-3B as baselines for non-compressed methods.

(1) Clarification of Fair Evaluation Setup.

To ensure that the activation parameter count of FLoE aligns with that of similarly sized non-compressed models, we applied INT8 quantization to the attention layers of FLoE at 90% sparsity. This adjustment ensures that the memory footprint of FLoE is comparable to that of Llama-3.2-3B in its FP16 configuration. For FLoE at 80% sparsity, the memory usage remains consistent with the original setup and matches that of Mistral-7B. Furthermore, we have supplemented the experimental results for MMLU (5-shot), GSM8K (8-shot), and HumanEval, providing a comprehensive comparison between FLoE and the baseline models.

(2) Analysis of New Results.

Not only on MMLU, but also on complex tasks such as GSM8K and HumanEval, various sparse methods face significant challenges, with performance tending to degrade more rapidly. However, our method, FLoE, demonstrates the best performance retention. At 90% sparsity, CATS almost completely fails on these three tasks, achieving an average score of only 16.2, while FLoE maintains a score of 43.8. At 80% sparsity, FLoE retains over 93% of the base model's capabilities.

Even smaller non-compressed models struggle to match the performance of large compressed models on complex reasoning tasks. On MMLU, GSM8K, and HumanEval, FloE at 90% sparsity outperforms both Mistral-7B and Llama-3.2-3B. Moreover, FloE at 80% sparsity preserves performance closest to the base model, demonstrating its clear advantage in these tasks. These results highlight that compressing large models can be more beneficial than relying on smaller non-compressed models, particularly for demanding reasoning tasks.

In summary, FloE not only achieves superior performance retention under high sparsity but also surpasses smaller non-compressed models across multiple benchmarks. This underscores the effectiveness of our approach in balancing model efficiency and task performance, especially in scenarios where computational resources are limited.

We believe these findings strongly reinforce FloE’s robustness and practicality, making it a compelling choice for deploying MoE-based LLMs in resource-constrained environments. For users with limited resources who cannot deploy MoE models, we highly recommend FloE as an efficient and practical alternative, delivering high performance while significantly reducing computational costs.

Looking forward to hearing from you!

Best,

Authors

Results Table:

[1] Zhou, Y., Chen, Z., Xu, Z., Lin, V., & Chen, B. Sirius: Contextual Sparsity with Correction for Efficient LLMs. NeurIPS, 2024.

Dear reviewer oREx,

Thanks for your careful review. We summarize and address your key concerns as follows:

[Suggestion 1] A theoretical analysis of how expert sparsification and quantization affect routing distributions and model expressiveness.

[Response] In Appendix B, we provide a detailed theoretical proof explaining the sparsity sensitivity differences observed in models with SiLU activations. Based on the distribution characteristics of the three matrices, we demonstrate why the output activations of the up projection exhibit lower sparsity sensitivity. Additionally, in lines 190 - 201 (Section 2.2.2), we offer qualitative analysis regarding the quantization sensitivity differences.

The impact of compression methods on routing was evaluated by performing 500 forward passes on the ShareGPT corpus. We measured the shift in routing logits caused by compression and found the results to be consistent with downstream task performance. Our method induces significantly less distortion to the routing distribution compared to other baselines with the same sparsity ratio. This observation aligns with the theoretical proof provided in Appendix B, further validating our approach.

[Suggestion 2] A deeper investigation into failure cases, such as whether certain compressed experts become underutilized or degrade faster than others.

[Response] In our current compression scheme that applies a uniform sparsity ratio to all experts, we have supplemented additional tests measuring the L2 norm of output errors before and after expert compression to investigate failure cases (as shown in expert sparsity loss figure). Our findings revealed that compression of experts in the 30th and 31st layers induces the most pronounced performance degradation. When preserving these critical layers as non-sparse, the model showed improvements of 1.3 on the MMLU benchmark and an average gain of 1.66 across six CSR tasks (BoolQ, SciQ, ARC-C, ARC-E, QA and WG).

These empirical results substantiate our hypothesis regarding layer-specific compression sensitivity. Accordingly, we develop to implement more fine-grained compression by selectively reducing the sparsity ratio for these critical experts, which can better preserve model performance while maintaining compression benefits.

[Suggestion 3] Additional studies on compression effects across different MoE models and more diverse tasks, to validate generalization beyond Mixtral-8×7B. For instance, how does DeepSeek-v2 or v3 behave in your system? Other open-source MoE models also include the Arctic model from SnowFlake.

[Response] In the Appendix E and F, we evaluate the sparsity sensitivity of Deepseek V2 and Phi 3.5, as well as the quantization sensitivity of Phi 3.5, Qwen 1.5A 2.7B, and Deepseek MoE. These results are consistent with the conclusions on Mixtral 8×7B, further validating our findings. Furthermore, we have supplemented the experimental results for MMLU (5-shot), GSM8K (8-shot), and HumanEval, providing a comprehensive comparison between FloE and the baseline models. These results demonstrate the strong generalizability of our method across different models and downstream tasks.

[Experiment Issue] Is the accuracy loss caused by FloE's compression sufficiently competitive compared to model distillation or direct deployment of smaller models?

Comparing with smaller non-compressed models is indeed crucial. However, due to time and resource constraints, we were unable to distill smaller models and instead chose the more advanced Llama3.2-3B and Mistral-7B as baselines for comparison. More details refer to the response to Reviewer bhnT.

We evaluated performance on three complex downstream tasks: MMLU, GSM8K, and HumanEval, which pose significant challenges for both small distilled models and sparse methods. Our method not only exhibits the least performance degradation among sparse baselines but also outperforms smaller models of similar scale across tasks. Despite retaining only 93% of the base model's accuracy on these tasks, it significantly surpasses other baselines. (Resualt Table in response to Reviewer bhnT)

Thus, we argue that appropriately compressing larger models is more competitive than directly deploying or distilling smaller models, especially in scenarios involving resource-constrained environments and complex reasoning tasks.

Cheers,

Authors

Results Table:

Router Logits Shift

Downstream Task Performance of Pin Higher Layer

Dear Reviewer u26i,

We sincerely thank you for your thorough and thoughtful review of our submission. Your feedback is invaluable, and we will provide a concise response to the knowledge background and GPU utilization concerns you raised.

[Minor issue] The paper is very difficult to read as it skips proper description of the terminologies used in the paper. Although it is understandable that it is challenging to put a lot of content into short 8 pages, I believe a little section to describe some background would be very helpful for readers.

[Responce] Thank you for your understanding regarding the difficulty of including rich content within the 8 pages. We agree that a more thorough background would be helpful for readers. As such, we have placed the related work on expert offloading and sparsity in LLMs in Appendix A, with a concise introduction and positioning provided in Lines 105-108, Section 2, Page 2 of the main text. We would be happy to move Appendix A to Section 2 if given any opportunity to revise.

In Appendix A, we introduce experts offloading and sparsity in LLMs, two key areas critical to the efficient deployment of large language models. Experts offloading addresses the memory bottlenecks caused by the massive parameter counts in Mixture-of-Experts (MoE) models, with frameworks like Llama.cpp and DeepSpeed Inference transferring expert weights between VRAM and DRAM. However, limited PCIe bandwidth leads to communication delays, and existing prefetching strategies face tradeoffs in accuracy, latency, and scalability, especially under multi-expert activation. Sparsity in LLMs, on the other hand, minimizes computational and memory overhead through techniques like weight pruning and activation sparsity. While these methods show promise, they are often hindered by performance degradation, hardware inefficiency, or limited adaptability to modern architectures like SwiGLU.

Our work builds on these foundations by addressing the gaps in both areas. We propose an approach for on-the-fly inference in MoE models and explore contextual sparsity techniques tailored for modern architectures. By optimizing parameter utilization and reducing hardware bottlenecks, our solutions aim to improve the efficiency, accuracy, and practicality of deploying MoE models in resource-constrained environments.

[Key Issue] Key question I have about the paper is on the GPU utilization w/ and w/o this work. Although memory is a big bottleneck in MoE adoption is "the large size of activated experts overburdens the limited PCIe bandwidth", another big contributor is the low GPU utilization that spikes up the cost of running services using MoE. I think it would be great to understand how GPU utilization is impacted.

[Responce] Two challenges stemming from distinct scenarios. Your analysis of the bottlenecks in MoE deployment is insightful. Indeed, PCIe transmission bandwidth and GPU utilization represent two critical challenges that arise from distinct scenarios and are driven by different underlying causes in the current deployment of MoE models.

The first challenge, which occurs in scenarios with limited GPU memory resources, compels the heterogeneous storage of model weights. Consequently, substantial latency occurs during inference due to the transfer of parameters between DRAM and VRAM.

The second challenge arises from an imbalance between computational load and memory usage in GPU-deployed MoE inference scenarios. This frequently leads to suboptimal GPU utilization, especially when processing small-batch requests, leaving GPU cores underutilized.

Our work primarily focuses on addressing the first challenge. Our work reduces the latency of model inference caused by weight transfer while maximizing the preservation of model accuracy. This is achieved by designing an ingenious hybrid compression strategy and pairing it with a corresponding prediction mechanism. Though our solution does not directly resolve GPU utilization concerns, we would be happy to share additional insights into how our approach interacts with GPU performance. We tested on the ShareGPT corpus and measured the GPU utilization rate at 63.2% during 50 forward passes with 20 tokens each. Given the relatively large matrix dimensions of MoE models compared to consumer-grade GPUs, we have observed that GPU utilization remains reasonably high in practice.

The research on the second challenge is orthogonal to our work. The challenge of GPU utilization that you raised is indeed one of the major challenges in MoE deployment. It is also a key direction for our future research and exploration. We deeply appreciate your feedback and hope to continue advancing our understanding as we work toward optimizing MoE deployment.

We hope the above clarifications address your concerns and would greatly appreciate your further feedback.

Cheers,

Authors