MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

paper does not provide direct experimental or theoretical proof that the poor MoE quantization effect is caused by load imbalance

We provide additional theoretical and experimental justifications below:

• Inter-Expert Imbalance
  • Theoretical Basis: Ours derivation proves that expert balance is helpful for accurate Hessian.
  • Experimental: Table R1 shows a positive correlation between expert balance and accuracy. EBSS(Expert-Balanced Self-Sampling) achieves 10x imbalance reduction and best accuracy. Table 5 shows EBSS reduces the accuracy gap by 49%.
• Intra-Expert Imbalance
  • Theoretical: Eq 15 formalizes expert aggregation, showing token-expert affinity varies. Eq 17 leverages linear equivalence for layer-wise quantization to anticipate affinity shifts.
  • Experimental: Table 5 shows that on DeepSeek-MoE-16B, AGQ(Affinity-Guided Quantization) alone reduces the 'Quarot+GPTQ' vs. FP gap by 30%, and combined with EBSS, by 60%.

Supplementary Material

We provide pseudocode for EBSS and AGQ in Algorithm 1-2.

Experiments cover code generation(HumanEval) and complex math reasoning(GSM8K). Table 2 shows we improve HumanEval over others by 44%-63%.

Figure 2 shows that EBSS significantly reduces inter-expert imbalance.

GW-MoE is not explicitly discussed in the paper but relevant to its context

We appreciate GW-MoE's insights in identifying routing uncertainty, which highlights load balancing. We've incorporated this into our revision for greater scholarly depth.

EBSS may introduce additional computational burden

We appreciate your concern, but EBSS reduces complexity significantly through three innovations:

• Probability-Guided Path Pruning eliminates unlikely branches using cumulative probability and expert balance, reducing candidates from $V^n$(exponential) to $mn$(linear).
• Cumulative Probability stores only $m$(e.g., 4) scalars, cutting storage overhead from $mV$ to $m$, enabling efficient PPL computation via Eq 12.
• Deferred Imbalance Calculation avoids iterating over all tokens (e.g., 150K) by leveraging known expert distributions from the current sequence.

Experiments in Table R6 shows EBSS's efficency(just 17–42 mins).

How does EBSS adjust the trade-off between PPL and expert balance? is there a risk of over-balancing experts at the cost of PPL?

EBSS optimizes: $D^* = \arg\min_{D}\{ \overbrace{\text{PPL}(M,D)}^{\in[0,1]} + \overbrace{\frac{\sigma(M,D)}{\tau}}^{\in[1,+\infty)} \}$where $\tau$ balances data distribution alignment (via PPL) and expert balance (via $\sigma$). Smaller τ prioritizes expert balance but may increase PPL. $\tau=1.2$ is determined through ablation in the manuscript. Table R2 further validate its generalizability in more models.

There is no risk of over-balancing as the cost of PPL because PPL is bounded, we prove it below:

• when ignoring $\sigma$，we denote $\text{PPL}=p$, $p$ typically remains close to 1 due to self-sampling(see Fig 4). We have:$D^* = \text{PPL}({M,D})+ \frac{\sigma(M,D)}{\tau}\leq p+\frac{1}{\tau}$ since $\sigma \geq 0$, we have: $\text{PPL}(M,D)\leq p+\frac{1}{\tau} - \frac{\sigma}{\tau}\leq p+\frac{1}{\tau}$

Can you provide a sensitivity analysis of all EBSS hyperparameters

Additional ablations in Table R2-R3, which shows：

• τ = 1.2 is also generalizable and effective on other models.
• Accuracy improves with increasing $w$, though marginal gains diminish significantly beyond w=4, offering an effective trade-off between performance and efficiency.

non-linear activation functions and residual connections in expert networks may break this assumption

For nonlinear SiLU in MoE LLMs, Eq 17 and AGQ remain effective. Table R4 and R5 verify them from respectively:

• Numerical Error. The reformulated gate output:$y_i\approx\left((xW^{up})\odot f(c_ixW^{gate})\right)W^{down}$ remains high cosine similarity(higher c close to 1) with true outputs. Moreover, top-k selection inherently prioritizes high-affinity tokens, further minimizing approximate error.
• Task Performance. Applying AGQ to gate layers yields significant gains in both PPL and accuracy(1.2%+ gain), confirming its effectiveness.

AGQ and residual connections (outside MoE layers) are decoupled. We agree that affinity masks hold promise and will explore it as MoE evolves.

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

The proposed method shows marginal improvements against GPTQ

When applying both proposed modules to QWENMOE-14B, the performance does not show a clear improvement

While Table 1 shows modest overall gains (1.2–4.1%), MoEQuant delivers substantial improvements in instruction-tuned models and complex multistep reasoning tasks(sensitive to quantization and critical for real applications). As shown in Table R1:

• On code generation(HumanEval) and mathe reasoning(GSM8K), MoEQuant achieves 4.7–9.8% relative accuracy gains in base models.
• In instruction-tuned models—more representative of deployment scenarios—it achieves 17.2–23.6% gains.

Additionly, we're sorry for potentially misleading readers about GPTQ in manuscript. As Line 353 states, it actually uses the initialization, similar to the competitive QuaRot.

However, I have concerns regarding its scalability to larger MoE models

Table 1 already demonstrates MoEQuant's effectiveness on Mixtral-8x7B (56B params). To further address your concern, we provide results for DeepSeekV2 in Table R2, where MoEQuant achieves a consistent 3.75%+ relative gain.

the speedup observed in Table 4 is not significant

The modest speedup for Qwen-MoE-14B and DeepSeek-MoE-14B stems from lower bandwidth pressure with fewer activated parameters, a trend also observed in [OSTQuant]. For Qwen-MoE-14B and DeepSeek-MoE-16B, where 2.7B and 2.8B params are active per step, acceleration remains low. In contrast, Mixtral-8×7B achieves a 2.1× speedup. Notably, after quantization, Qwen-MoE-14B requires only 8.51 GB of memory, enabling MoE-LLM deployment on edge devices (e.g., RTX 3060).

[OSTQuant]:Refining Large Language Model Quantization with Orthogonal and Scaling Transformations

when combining MoEQuant with GPTQ, the paper does not report the corresponding latency

We appreciate the your suggestion and add a comparison of time overhead in Table R3. Since EBSS reduces search complexity from exponential to linear through Probability-Guided Pruning and AGQ remains lightweight with negligible overhead, the additional time increase is minimal(only 11 minutes for DeepSeek).

it would be beneficial to give more analyses about the synthetic data

We appreciate your constructive feedback and provide synthetic cases in Table R4, which highlight two key characteristics:

1. Coherence and Distributional Alignment: Generated sentences maintain coherence and align with the original models' distribution, also evidenced by consistently low PPL in Figure 4.
2. Diversity: The data spans diverse domains, including reasoning and code generation. This diversity alleviates expert imbalance and drives significant performance gains (relative 5–24%) on complex reasoning tasks(Table 2).

Paper only demonstrates the existence of imbalance among experts, however, it lacks empirical or theoretical analysis on how this imbalance affects the quantization results

Thanks for your insightful feedback. we provide both empirical and theoretical analysis below:

• As shown in Table R5, expert balance is positively correlated with final accuracy. MoEQuant reduces imbalance by 90% compared to others, achieving the best accuracy by encouraging even token distribution among experts.
• In GPTQ, the Hessian matrix quality directly impacts quantization precision. We estimate it via the finite difference method: $H_{ij} \approx \frac{\partial^2{L}}{\partial w_i \partial w_j} \approx \frac{L(w+\epsilon e_i+ \epsilon e_j)-L(w + \epsilon e_i)-L(w+\epsilon e_j)+L(w)}{\epsilon^2}$ $\epsilon$ is the tiny perturbation, $e_i$ is the i-th standard basis vector, $L$ is the loss. A fourth-order Taylor expansion yields: $\hat{H_{ij}} \approx H_{ij} + \frac{\epsilon^2}{12}(\nabla_{iiii}L+6\nabla_{iijj}L+\nabla_{jjjj}L)+O(\epsilon^4)$ Assume the estimated variance of the loss is $Var(L)=\frac{\sigma^2}{n}$ , $\sigma^2$ is the population variance, and $n$ is the sample size. Then the expectation of the Hessian estimation: $E[\hat{H_{ij}}] = H_{ij}+O(\epsilon^2)+O(\frac{\sigma^2}{n\epsilon^2})$ Then, the bias of the Hessian estimation is: $Bias(\hat{H_{ij}}) = E[\hat{H_{ij}}]-H_{ij} \approx O(\epsilon^2)+O(\frac{\sigma^2}{n\epsilon^2})$ The formula's second term is inversely proportional to sample size n: larger n means smaller statistical deviation, and vice versa. Thus, maintaining expert balance ensures all experts are well-calibrated, leading to better performance.

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

The study focuses solely on weight quantization and does not address activation quantization.

Memory constraints are the decisive factor for deploying MoE LLMs, which require orders of magnitude more memory than dense models (e.g., DeepSeek-MoE-14B retains 14B params in memory despite activating only 2.7B). Therefore, we prioritize weight quantization because it significantly alleviates memory pressure, achieving a 3.74× memory saving and a 2× speedup (see Table 4) for Mixtral-8x7B, thereby enabling deployment on consumer GPUs.

To further address your concern, we add W4A4 experiments. Table R1 show that MoEQuant improves accuracy by over 3% compared to QuaRot, demonstrating its versatility across diverse quantization scenarios.

MoEQuant is primarily an enhancement of existing quantization methods like AWQ and GPTQ. While it introduces novel techniques

MoEQuant is a novel PTQ framework tailored for MoE LLMs rather than an enhancement, addressing key limitations of existing methods through three contributions:

• Problem Identification: As shown in Table 1, previous PTQ methods to MoE LLMs causes severe accuracy drops(up to 17%), We attribute it to 2 key issues: inadequate calibration across experts(inter-expert imbalance) and inaccurate quantization error modeling(intra-expert imbalance) inherent in MoE quantization.
• Technical Innovation: EBSS(Expert-Balanced Self-Sampling) ensures adequate expert calibration via probability-guided pruning and deferred imbalance evaluation, maintaining alignment with fp models while reducing search complexity (Sec. 4.2). AGQ(Affinity-Guided Quantization) incorporates token-expert affinities into quantization loss and Hessian statistics, enabling precise sensitivity analysis for MoE’s dynamic routing (Sec. 4.3).
• Empirical Validation: MoEQuant seamlessly integrates with existing PTQ methods(AWQ,GPTQ,QuaRot), significantly improving accuracy. As validated in Table R7, it outperforms QuaRot by 5–23% on complex tasks like HumanEval and GSM8K, effectively closing prior performance gaps.

The baselines are not the latest and AWQ+GPTQ would be much better

We would like to clarify that, as described in Line 383, our GPTQ implementation equal 'QuaROt+GPTQ', a current SOTA training-free PTQ method, this ensures rigorous baseline.

To further address your concern, we have added a comparison with Omniquant in Table R4-6 and MoEQuant still works better. We also manually implemented GPTQ+AWQ but obtained worse performance.

the ablation study in Table 5 shows that EBSS and AGQ improve the accuracy very little

While Table 5 shows modest overall gains(1.2–4.1% relative), we emphasize that MoEQuant delivers significant gains in instruction-tuned models and complex reasoning tasks(sensitive to quantization and critical for real applications). As shown in Table R2:

• On code generation(HumanEval) and mathematical reasoning(GSM8K) , MoEQuant achieves 4.69–9.84% relative accuracy gains in non-instruction-tuned models.
• For instruction-tuned models, which better reflect deployment scenarios, MoEQuant demonstrates 17.22–23.57% relative gains.

Could you provide comparison of the time cost for MoEQuant and baselines?

We add a comparison of the time costs for MoEQuant and GPTQ. As shown in Table R3, Due to the incorporation of three cost-reducing techniques in EBSS—namely Probability-Guided Path Pruning, Cumulative Probability, and Deferred Expert Imbalance Calculation—MoEQuant only increases a small amount of time overhead compared to GPTQ.

Are the router layers of MoE quantized? how do these layers affect the performance of the quantized model?

Router layers are quantized. We did not find an obvious impact on the final performance. This may be due to the robustness of gating and the redundancy of expert knowledge.

Could you show the latest weight-only quantization methods on MoE models, such as OmniQuant and EfficientQAT

In Table R4-R6, we provide additional results comparing MoEQuant with OmniQuant. While a direct comparison may be unfair since MoEQuant is designed for PTQ and OmniQuant involves training, MoEQuant consistently outperforms OmniQuant across all tested models. This advantage arises because OmniQuant relies on a fixed calibration set, disregarding expert balance. EfficientQAT, being a global QAT algorithm, is beyond the scope of our comparison.

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

Why do some tasks (like HumanEval) benefit significantly from MoEQuant, while others show minimal or no improvement?

Thanks for your insightful observation. We classify tasks into:

• Simple Retrieval Tasks (e.g., BOOLQ HellaSwag) rely on direct answer based on surface-level knowledge and exhibit limited benefits from multi-expert collaboration. However, we still reduces the FP accuracy gap by up to 54%(e.g., on DeepSeek-MoE-16B).

• Challenging Resoning Tasks (e.g., HumanEval GSM8K) involve continuous multi-step logical/mathematical reasoning, code generation, or structured problem-solving. MoE architectures excel here by assigning specialized experts to sub-problems, enhancing performance through collaborative reasoning. However, Applying PTQ to MoE LLMs for these tasks introduces two key challenges:

  • Each expert must be sufficiently calibrated to ensure both functional accuracy and diversity.
  • Mitigating error accumulation during multi-step decoding, which severely degrades accuracy.

  MoEQuant addresses them via EBSS(Expert-Balanced Self-Sampling) for comprehensive calibration of each expert and AGQ(Affinity-Guided Quantizatin) for improved quantization error modeling, achieving 5–24% relative accuracy gains.

Why does AGQ alone underperform GPTQ on certain tasks, but perform better when combined with EBSS?

AGQ optimizes quantization error via token-expert affinity weighting. However, when used alone, imbalanced sample distribution among experts leads experts to receive fewer low-confidence samples, increasing noise and variance in Hessian and causes evaluation variance. Despite this, AGQ still improves overall performance(1.2% relative gain on Mixtral).

Combining AGQ with EBSS ensures sufficient calibration for each expert, yielding a more stable Hessian with low variance, enabling 4% relative gain on Mixtral. This synergy outperforms either method alone.

The paper notes dramatic accuracy drops with methods like RTN at 3-bit quantization. Could you discuss the reasons behind it and explain how MoEQuant effectively mitigates these issues?

As shown in Table R1, 3-bit quantization, limited to just 8 discrete values, constrains RTN performance. Moreover, MoE LLMs suffer from inter- and intra-expert imbalances, exacerbating degradation. MoEQuant addresses these imbalances via:

1. EBSS: Enables comprehensive calibration of each expert, mitigating inter-expert imbalance.
2. AGQ: Incorporates token-expert affinities into quantization loss, correcting intra-expert imbalance.

This dual approach in MoEQuant delivers SOTA 3-bit PTQ results, yielding up to a 10.5% accuracy gain on Mixtral-8x7B (Table 3).

Could you provide additional theoretical or intuitive insights into how the temperature parameter (τ) affects the sampling process in EBSS?

EBSS optimizes the objective: $\mathcal{D}^* = \arg\min_{\mathcal{D}} \{ \overbrace{\text{PPL}(\mathcal{M},\mathcal{D})}^{\in[0,1]} + \overbrace{\frac{\sigma(\mathcal{M},\mathcal{D})}{\tau}}^{\in[1,+\infty)} \}$where $\tau$ balances distribution alignment (via PPL) and expert balance (via $\sigma$). Smaller τ prioritizes expert balance but may slightly increase PPL. However, PPL remains bounded, as proved below:

• When ignoring $\sigma$，we denote $\text{PPL}=p$, $p$ typically remains near 1 due to self-sampling(see Figure 4). We derive:$\mathcal{D}^* = \text{PPL}(\mathcal{M,D})+ \frac{\sigma(\mathcal{M},\mathcal{D})}{\tau}\leq p+\frac{1}{\tau}$ since $\sigma>=0$, we derive: $\text{PPL}(\mathcal{M,D})\leq p+\frac{1}{\tau}$

The hyperparameter $\tau=1.2$ is determined via ablation studies. Further experiments in Table R2 confirm its generalizability.

How might the quantization performance be affected if the activations deviate from this quasi-linear behavior?

For nonlinear function SiLU used in MoE LLMs, Equation 17 and AGQ remain effective. We support it by corresponding experiments：

• Numerical Error. As shown in Table R3, the reformulated gate output: $y_i\approx\left((xW^{up})\odot f(c_ixW^{gate})\right)W^{down}$ remains high cosine similarity with true outputs across c values, approaching 1 for high-c experts . Top-k expert selection inherently prioritizes high-affinity tokens, further minimizing cumulative error.
• Task Performance Validation. As Shown in Table R4, applying AGQ to the gate layers (which governs non-linear interactions) yields significant gains in both perplexity and downstream accuracy(1.2% gain for Qwen-MoE), confirming its effectiveness.