Thank you for your valuable feedback. We would like to address your comments and questions as follows.

Questions For Authors:

1. The authors propose a method to predict expert activations during decoding based on historical activation traces. Could you please give more details on how to manage the cache at the system level, especially when data needs to be transferred between GPUs and CPUs?

We manage the expert cache at the granularity of individual experts, with each expert's parameters stored as a contiguous memory block. Our system incorporates several techniques to ensure efficient CPU–GPU transfers:

• Avoiding repetitive expert fetching: Once token-to-expert routing is determined at each layer, the experts that are already resident in GPU memory are "locked" to prevent eviction until their inference tasks are completed. While common frameworks such as Hugging Face launch selected experts strictly in ascending index order (e.g., executing expert 3, then 4, then 5 if those are selected), MoE-Infinity prioritizes experts already in cache, enabling faster inference by overlapping their execution with the fetching of cache-missed experts.
• Non-blocking Eviction: When an expert is evicted from the GPU cache, its corresponding parameters remain in CPU memory. This design choice eliminates the need for blocking transfers back to the CPU.
• Memory Pooling and Reuse: Separate expert memory pools are maintained on both CPU and GPU. For the GPU pool, evicted experts do not immediately trigger memory deallocation; instead, their slots are simply marked as available. This avoids the unpredictable and sometimes costly latency (up to tens of milliseconds) associated with memory free operations.
• Optimized Data Transfers: CPU–GPU data transfers use CUDA’s memory management APIs (e.g., cudaMemcpy) and are accelerated through pinned memory to enable direct memory access (DMA), reducing transfer latency.

Others:

N/A

Thank you for the detailed feedback—we’ll revise the figures and writing for clarity, and address key concerns below.

Questions For Authors:

1. Clarification to vLLM MoE Offloading implementation to ensure apple-to-apple comparison.

Thanks for raising this point. We include vLLM as it is a widely used SOTA LLM inference engine that supports CPU offloading, and many reviewers and users requested its inclusion in our evaluation.

We are aware that vLLM achieves efficient on-demand parameter fetching, sometimes outperforming baselines with simple expert prefetching (such as Mixtral-Offloading), which can waste expensive PCIe bandwidth by prefetching unused experts. This is shown in Table 1, and we will clarify it in the revised version.
In fact, sparsity-aware expert caching (MoE-Infinity) is conceptually complementary to vLLM’s offloading engine, and we are investigating how to integrate MoE-Infinity into vLLM to further improve its performance on MoE models.

2. The reasoning behind the decision to restrict bsz = 1. Expanding the analysis to include multi-request scenarios.

In practice, the beam width used in models is relatively small (typically ≤ 4), and batch sizes in ToT-style reasoning (e.g., Tree of Thoughts, NIPS 2023) is 5. The result of batch size (1-32) is shown in Reviewer S1MY Others Section Q3.

Others:

1. Contribution 1 asserts that caching can be leveraged even when the batch size is set to 1. While the claim is supported by empirical traces collected by the author, it lacks novelty.

Our system, started in September 2022, was among the first to explore sparsity-aware expert caching. Since its open-source release, it has gained significant traction and has been continuously improved to deliver state-of-the-art MoE inference on memory-constrained machines.

Despite this, our contributions are fundamentally different from [1, 2, 3, 4], which also explains why our system significantly outperforms [1]—the only open-source library among them.
First, Contribution 1 goes beyond highlighting expert locality (i.e., skewed expert reuse). It focuses on identifying when and how such reuse patterns can be robustly observed and exploited. Our core finding—distinct from [1–4]—is that skewed expert reuse emerges only at the request level during decoding, rather than across requests [3, 4], tokens [1], or tasks [2]. Capturing these patterns goes beyond frequency counting or Markov models; we use request-level tracing with continuous matching—an approach not explored and used in prior works [1–4].
Specifically:

• Mixtral-Offloading [1] assumes a correlation between individual tokens and expert activation. However, in modern MoE-based LLMs using attention, token activation depends on context, so a token may activate different experts in different settings, making this correlation unreliable and thus resulting in poor performance in practice.
• [2] suggests that task-specific inputs lead to activation skewness. However, in practice, as the number of prompts increases, expert usage tends to become uniform at the task level. With this uniformity, the expert cache cannot effectively prioritize which experts to retain, which explains why this method is not widely adopted in real-world deployments.
• EdgeMoE [3] claims reuse patterns exist across requests, but this fails to hold when expert usage is aggregated, which trends toward uniformity. In fact, this was one of the early designs implemented in MoE-Infinity (as a parallel research work), and we soon realized that this claim does not work over long periods of LLM serving.
• SwapMoE [4] assumes semantically similar consecutive prompts to maintain reuse patterns, making it hard to deploy in practice since this assumption does not robustly hold. In contrast, MoE-Infinity does not rely on this assumption and can robustly capture reuse patterns across diverse datasets and models.

In our collaboration with a partner deploying DeepSeek-R1, none of the above methods were used due to poor performance; they’re now working with us to integrate MoE-Infinity in clusters potentially hosting thousands of GPUs.

2. While the paper describes the methodology, further discussion on its distinct advantages over existing approaches would strengthen this contribution.

We carry out further micro benchmarks to distinguish our approaches from baselines. The result of the cache hit rate can be found in Reviewer S1MY Others Section Q4, and the results of the ablation studies of MoE-Infinity system components can be found in Reviewer HkeA Others Section Q2.

Thank you for your valuable comments and suggestions. We address your concerns and comments as follows.

Questions for Authors:

No questions

Others:

1. Accessibility for General ML Audience: The paper can be difficult to follow for a general ML audience due to its engineering-heavy focus. While the speedups are impressive, the ML novelty of the work is rather limited.

Our paper targets the ICML “Machine Learning Systems” track, with a focus on system-level novelty. We intentionally avoid modifying ML algorithms or MoE architectures (i.e., ML novelty). Based on our interactions with the ML community, there is a strong preference for systems that serve unmodified full models, thus preserving model accuracy. This approach also ensures MoE-Infinity remains compatible with emerging ML techniques like multi-token prediction. We will revise the writing of our paper to suit the general ML audience.

2. Clarity on Contributions to Speedup: Are these gains primarily from optimized caching and prefetching, the enhanced eviction policy, or the expert location prior?

Thank you for this suggestion. We did have the ablation studies while designing MoE-Infinity to understand how much benefits each component contributed, but we could not include them in the submitted version due to the page limit.
The results of the ablation studies can be found in the table below. We disabled one component at a time and measured the resulting performance degradation.
The most significant degradation occurs when disabling expert-wise offloading, as MoE layers are inherently sparse—fewer than 25% of experts are activated per token per layer. Without fine-grained offloading, the system unnecessarily fetches inactive experts, leading to a 3-4x increase in latency.
Other system-level optimizations also play crucial roles. pinned memory and NUMA-aware placement affect PCIe transfer speed, which is critical in offloading-enabled inference with small batch sizes. Finally, the benefit of our caching strategy is strongly tied to the cache hit rate, which is directly impacted by expert access patterns and the eviction policy.

3. Lack of Detail in Claims: Several claims in the paper lack sufficient grounding or detail.

Thank you for your comments. Figure 2 is based on 1,000 requests uniformly sampled from a mixed dataset comprising BigBench, MMLU, and FLAN. We observe the same phenomenon consistently across each individual dataset. These results indicate that state-of-the-art MoE models, such as DeepSeek and Mixtral, show limited or no clear evidence of task specialization.
We analyzed expert activation across MoE layers during long-context decoding and found that activation is both highly skewed within each layer and non-uniform across layers—the most frequently activated expert IDs differ from one layer to another.

4. Comparisons Across Frameworks: While comparisons across various frameworks are valuable, the vastly different approaches and implementation details make it challenging to pinpoint what drives the improved speedup.

We have results comparing mispredict between MoE-Infinity and baselines: caching the top k% frequently activated experts (Top-K), LRU, and Belady. We present the results in the table below. MoE-Infinity achieves 11–34% higher hit rates than baselines. Notably, for models like DeepSeek and Arctic, which exhibit complex and sparse expert activation (i.e., 100s experts per layer and low expert selection), MoE-Infinity significantly outperforms all alternatives. Among baselines, Top-K performs better than LRU (by 3–7%) except DeepSeek. These results underscore the need for our designs: selecting representative traces and continuously predicting activation during decoding.

We have also conducted a detailed breakdown analysis; please refer to Others Section Q2 for further information.

Thank you for the helpful suggestions. Below, we address the remaining concerns.

Questions:

1. 75% of experts are never activated?

We clarify this does not mean 75% of experts are never activated. The 25% activation rate is inherent to Mixtral’s design, activating the top 2 out of 8 experts per layer per token at each decode step. When tracking expert activation across many requests, all experts are utilized, with usage following a uniform distribution.

2. Expert activation changes depending on dataset?

Yes. We explored this during the design of MoE-Infinity by mixing datasets and switching them during inference. Table 3 presents partial results.

Our experiments show that MoE-Infinity is robust across datasets. When mixing numerous tasks from BigBench, tail decoding latency remains stable. In a more extreme test(MMLU->BigBench ->C4->MATH), we observed that latency briefly spikes when switching to BigBench but recovers within ~20 prompts. Latency stays stable from C4 to MATH, as BigBench already covers their activation patterns. Results are similar across models(DeepSeek, Switch, NLLB, Mixtral).

Others:

1. Expert activation varies across different layers?

Expert activation is skewed within layers and non-uniform across layers, with top expert IDs varying by layer. The heatmap with layer-wise normalized activation count(Mixtral-8x7B on BIG-Bench, decode 128) shows that some layers focus activation on a small subset of experts, while others distribute activation more evenly. Similar patterns were observed in DeepSeek, Arctic across datasets like BigBench and SharedGPT.

2. Multi-GPU not fully explored.

MoE-Infinity supports multi-GPU via expert parallelism.

The table below shows our multi-GPU results. Scaling to more GPUs lets MoE-Infinity cache more experts, greatly reducing latency esp. for large models like Mixtral-8x7B, NLLB, and DeepSeek-V2-Lite. DeepSeek’s 4/8GPU results omitted(fits in 2 GPUs).

3. Batch size > 1 with baselines?

We are aware that small batch sizes >1(e.g., in Tree of Thought prompting) are relevant on personal machines, and MoE-Infinity supports such use cases. On GSM8K (prompt 512, decode 256), MoE-Infinity achieves 2–6× higher TPOT for batch sizes 1–32, with similar trends on other datasets. As batch size grows, reduced expert sparsity narrows the gap. At batch size 32 (already large for personal machines), MoE-Infinity outperforms most baselines and is only ~10% slower than vLLM, despite vLLM’s heavy kernel and engineering optimizations. This gap is expected to shrink as MoE-Infinity matures.

4. How often does MoE-Infinity mispredict?

We have results comparing mispredict between MoE-Infinity and baselines: caching the top k% frequently activated experts(Top-K), LRU, and Belady. We present the results in the table below. MoE-Infinity achieves 11–34% higher hit rates than baselines. Notably, for models like DeepSeek and Arctic, which exhibit complex and sparse expert activation(i.e., 100s experts per layer and low expert selection), MoE-Infinity significantly outperforms all alternatives. Among baselines, Top-K performs better than LRU(by 3–7%) except DeepSeek. These results underscore the need for our designs: selecting representative traces and continuously predicting activation during decoding.

The above results highlight the importance of capturing request-level skewness using our advanced tracing and prediction mechanism, which selects representative activation patterns from past traces and continuously predicts expert usage as decoding progresses—a capability uniquely offered by MoE-Infinity among SOTA MoE systems.

5. Have the authors found that some experts are useless?

When handling large many-task datasets(e.g., BigBench), all experts are eventually activated, with activation ratios converging toward uniformity over time. While some experts may be underused in short intervals,we do not prune them; instead, we trace these skewed patterns to guide future activation prediction, aiming for robust long-term serving.

6. Combine with compression techniques?

MoE-Infinity is designed to serve unmodified models to preserve accuracy, with lossy optimizations considered complementary. Some users have integrated quantization methods like GPTQ, and MoE-Infinity still shows robust performance.