DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation

Q3: How sensitive is the performance of DFRot to the specific choice of the single calibration sample used for optimizing R1?

A3: We performed ablation studies W4A4KV16 on LLaMA3-8B and LLaMA2-7B along two dimensions: the choice of calibration samples and the num of calibration tokens and evaluate results on WikiText. Samples are all sampled from WikiText-2 train. In selecting the number of tokens, we utilize the inputs to the first n transformer blocks as the calibration data source. For example, when 48×2048 tokens are selected, the inputs to transformer blocks 0 to 23 are used for calibration.

Our results indicate that, for tuning the rotation matrices in LLaMA3-8B and LLaMA2-7B, using 16×2048 tokens is often sufficient. We believe that these reasons may all be the causes for the optimization of DFRot being relatively insensitive to the number of tokens in the calibration dataset:

• These special tokens will appear in relatively shallow network layers [1], therefore, a small number of layers are also sufficient to capture these tokens.
• For a model, tokens with massive activations in LLMs tend to exhibit only a few similar data distributions because these tokens are often produced by out_proj or down_proj layers with large weights [2].
• GPTQ will use 128 samples with a length of 2048 to calibrate the weights, which reduces the impact of the sample size during rotation matrix calibration.

Sample without massive activation tokens : We believe that this will indeed lead to a decline in performance, as seen in Figure 5. This is because the optimization of the rotation matrix requires minimizing the quantization error of such tokens as much as possible. Otherwise, the performance of the optimized matrix may tend to be similar to that of the Randomized Orthogonal (RO) transform even worse (Figure 5). In calibration data, we found that, given a length of 2048, these special tokens often exist.

---

Q4: The alternating optimization algorithm lacks discussion of convergence guarantees and the dependence on initialization RH.

A4: The proof of convergence is indeed a missing part in the paper. Although we empirically found that the loss always converged steadily during the optimization of the rotation matrix, we have to admit that due to the discontinuity of the quantization function, we can not rigorously prove the convergence of the algorithm through theoretical analysis.

We believe the dependence on RH may be due to the discontinuity of the quantization function, and the current algorithm is very likely to fall into local minima. It is also something we need to improve in the future.

Q5: The weight gamma for the massive activation loss term is chosen empirically. Have you considered methods to automatically tune or learn this hyperparameter? Additionally, was the test set used as validation for tuning during experiments?

A5: Under the optimization objective of Equation 3, we can’t to turn gamma into a parameter for automatic learning because intuitively, gamma=0 must be optimal.

If we modify the optimization objective, It should be feasible. For example, we can align the input of each block and use the quantization error as a regularization term. In this way, it should be possible to dynamically learn gamma. We tune the rotation matrix using a sample with 2048 length from the WikiText train.

Q6: How to judge massive activations?

A6: By comparing Figure 4 in [3] and Figure 1 in [4], we can intuitively know how to judge outliers and massive activations.

Q6: Please improve the visibility of Figure 2.

A6: Thank you for your suggestion. We will make every effort to improve the visibility of Figure 2.

[1] FlatQuant: Flatness Matters for LLM Quantization

[2] Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform

[3] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

[4] Massive Activations in Large Language Models

We greatly appreciate the reviewer's comments. Indeed, our work is carried out standing on the shoulders of giants.

As the reviewer has suggested, the most interesting aspect of our work lies in our rigorous experiments demonstrating that, after applying the transformations in QuaRot, the tokens with the smallest quantization errors are actually the ones that most affect quantization performance, which is highly counterintuitive. Based on this discovery, we naturally link QuaRot with Massive Activation, helping the community understand why the RH outperforms RO and reasonably applied optimization techniques.

---

Q1: Could you provide a more detailed analysis of the practical latency overhead introduced by the online computation of the R3 and R4 Hadamard transforms during inference?

A1: Since there has been relevant work in the community discussing the overhead of these online rotation matrices, please allow me to directly cite their related work. As seen in Figure7 and Table 9 in [1], the dimension of $R_3$ is generally small (e.g., 64, 128), so kernel fusion with the RoPE operator can be considered to improve memory efficiency without introducing excessive computational overhead. For $R_4$, when the dense model has a large dim, the Kronecker Product can be used to decompose the Hadamard matrix, and kernel fusion techniques can be employed to reduce calculations and enhance memory efficiency respectively. For MoE models, $R_4$ can be small, so kernel fusion techniques can be considered by fusing with the Swish to improve memory efficiency.

For end-to-end inference, we believe that whether INT4 GEMM + Online Hadamard can improve performance compared to INT8 GEMM depends on the specific computing conditions as seen in Figure 10 in [1]. When it comes to computing bound scenarios, we believe INT4 GEMM + Online Hadamard has the potential to improve latency.

---

Q2: Explain Hadamard matrices advantageous behavior with massive activations under 4-bit quantization compared to general random orthogonal matrices.

A2: Before providing an intuitive explanation, we first present some background. Since rotational invariance requires merging the $\alpha$ from RMSNorm into the subsequent Linear layer, we know that the L2-Norm of each token in the calibration set is $\sqrt{D}$, where $D$ is the hidden_state_dim of the model.

Given a token $X\in \mathbb{R}^{1\times D}$ with massive activations, $Q$ is the quantization function, we can know if $\|X\|_\infty \to \sqrt{D}$, $Q(X)\to 0$

$\text{set}(XH)$={$\pm \|X\|_\infty$}, therefore $Q(XH)\approx0$.

$XR\to R_{:,0}$, therefore $Q(XR)>0$

It can be seen that, considering the extreme case, when the massive activation of X is extremely large, we can assume that the quantization errors of both $X$ and the $XH$ tend to 0, while the quantization error of the feature $XR$ is obviously greater than 0. This is the numerical reason behind the phenomenon discovered in our paper, where XR actually increases the quantization error. However, in practice, considering that there may be more than one location of massive activation and the extreme conditions are difficult to meet, although the Hadamard matrix has shown very impressive performance, it is not necessarily the optimal choice for the rotation matrix.

Q1: The authors are encouraged to evaluate methods on more diverse downstream tasks (e.g., MMLU) or model families beyond LLaMA (e.g., Qwen or MoE models).

A1: Results on Wiki and zero-shot results

Results on MMLU

Q2: The alternating optimization algorithm lacks discussion of convergence guarantees.

A2: The proof of convergence is indeed a missing part in the paper. Although we empirically found that the loss always converged steadily during the optimization of the rotation matrix, we have to admit that due to the discontinuity of the quantization function, we can not rigorously prove the convergence of the algorithm through theoretical analysis.

Q3: Insufficient analysis of calibration set sensitivity.

A3: We performed ablation studies W4A4KV16 on LLaMA3-8B and LLaMA2-7B along two dimensions: the choice of calibration samples and the num of calibration tokens and evaluate results on WikiText. Samples are all sampled from WikiText-2 train. In selecting the number of tokens, we utilize the inputs to the first n transformer blocks as the calibration data source. For example, when 48×2048 tokens are selected, the inputs to transformer blocks 0 to 23 are used for calibration.

Our results indicate that, for tuning the rotation matrices in LLaMA3-8B and LLaMA2-7B, using 16×2048 tokens is often sufficient. We believe that these reasons may all be the causes for the optimization of DFRot being relatively insensitive to the number of tokens in the calibration dataset:

• These special tokens will appear in relatively shallow network layers [1], therefore, a small number of layers are also sufficient to capture these tokens.
• For a model, tokens with massive activations in LLMs tend to exhibit only a few similar data distributions because these tokens are often produced by out_proj or down_proj layers with large weights [2].
• GPTQ will use 128 samples with a length of 2048 to calibrate the weights, which reduces the impact of the sample size during rotation matrix calibration.

Q4: No discussion of per-token clipping thresholds for rare tokens.

A4: Although we did not use activation clipping for tokens for the purpose of simplifying the problem in this paper, we acknowledge that clipping techniques are highly effective in minimizing quantization errors. However, in our additional explorations, we discovered some interesting experimental phenomena (although these results were not presented in the paper). We found that when using rotational invariance, GPTQ, and 4-bit activation quantization:

1. LLaMA2-7B: We performed per-token clipping on these rare tokens of LLaMA2-7B, which significantly reduced the quantization error. However, the model performance after GPTQ quantization decreased remarkably. We conjecture that after rotation, the information of massive activation is distributed across each dimension of the token. Although clipping can significantly reduce quantization errors, it also leads to severe loss of this information. Meanwhile, it may undermine the information in the Hessian matrix of GPTQ.

2. LLaMA3-8B: From Figure 4, we can see that for LLaMA3-8B which is difficult to quantize, DFRot has almost eliminated the quantization errors of these rare tokens, making it difficult to further reduce them through activation value clipping for these tokens.

In fact, clipping these rare tokens is a model-specific issue. In the future, we will conduct further research on how to integrate technologies such as rotational invariance, activation clipping, and GPTQ to further enhance the model's performance in 4-bit activation quantization.

[1] Massive Activations in Large Language Models

[2] The Super Weight in Large Language Models

Q1: The paper focuses on optimizing quantization accuracy for outliers and does not address memory reduction or acceleration.

A1: For $R_1$ and $R_2$, as both of them can be fused into network, it will not bring additional cost.

For $R_3$ and $R_4$, since there has been relevant work in the community discussing the overhead of these online rotation matrices, please allow me to directly cite their related work. As seen in Figure 7 and Table 9 in [1], the dimension of $R_3$ is generally small (e.g., 64, 128), so kernel fusion with the RoPE operator can be considered to improve memory efficiency without introducing excessive computational overhead. For $R_4$, when the dense model has a large dim, the Kronecker Product can be used to decompose the Hadamard matrix, and kernel fusion techniques can be employed to reduce calculations and enhance memory efficiency respectively. For MoE models, $R_4$ can be small, so kernel fusion techniques can be considered by fusing with the Swish to improve memory efficiency.

Q2: The limited improvement in zero-shot tasks (e.g., LLaMA2-70B) .

A2: Yes, we have also observed this. We believe that larger models behaves greater robustness. Meanwhile, compared to the LLaMA3 family, the LLaMA2 family performs remarkably well under RH, indicating that RH can effectively handle massive activations in a data-free setting.

In this case, optimizing with DFRot may cause the quantization error of tokens with massive activations to decrease slightly, while the quantization error of some ordinary tokens increases instead, thus affecting the performance of the quantized model.

Q3: Theoretical analysis of large activation tokens (e.g., their association with critical semantic features) supported by literature or derivations.

A3: Our work primarily discovers a counterintuitive phenomenon: in the case of quantization, tokens with massive activation, despite exhibiting the smallest quantization errors, have a significant impact on the performance of quantized models. This is used to explain why RH outperforms RO.

Fortunately, there are already relevant works in the community that have conducted corresponding analyses on the mechanisms and effects of massive activation emergence. The relevant literature is listed below, and we will add these references in the final version.

[1] Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding

[2] Massive Activations in Graph Neural Networks: Decoding Attention for Domain-Dependent Interpretability

[3] House of Cards: Massive Weights in LLMs

[4] MOYU: A Theoretical Study on Massive Over-activation Yielded Uplifts in LLMs

[5] A Refined Analysis of Massive Activations in LLMs

[6] The Super Weight in Large Language Models

For end-to-end inference, we believe that whether INT4 GEMM or INT4 GEMM + Online Hadamard can improve performance compared to INT8 GEMM depends on the specific computing conditions as seen in Figure 10 in [1]. When it comes to computing bound scenarios, we believe INT4 GEMM + Online Hadamard has the potential to improve latency.

For inference, our computations are exactly the same as those of QuaRot, therefore we can see resources consumption from Table 14～Table17 in QuaRot [2] as a reference.

Q4: Details of Rotation matrix and Local optimum risk.

A4: In DFRot, we adopt a complete D×D rotation matrix. Since R1 can be merged into the network, whether to use a diagonal block matrix or a complete matrix has no impact on the inference speed.

In terms of effectiveness, the performance of diagonal block matrices is often inferior to that of complete matrices. Therefore, the former can only smooth the feature distribution within blocks to reduce quantization errors, while the latter can smooth features globally.

From an optimization perspective, the configuration of block diagonal matrices can also be optimized using the DFRot method, and the optimization speed will be faster. However, considering that optimizing the DFRot rotation matrix does not introduce much overhead, we believe that using a complete matrix is more appropriate.

Local optimum risk: Due to the discontinuity of the quantization function, our current optimization method is prone to getting stuck in local minima and often relies on initialization. As can be seen from Figures 5 and 6, the performance of RH is better than that of RO in most cases. Meanwhile, during the optimization of the rotation matrix, we also found that after iterative convergence, the final quantization error optimized with RH initialization is often better than that with RO.

Q5: Calibration-data distribution bias may impact performance.

A5: For GPTQ, the selection of calibration data often significantly impacts performance. For example, when calibrating on WikiText-2 and C4 respectively, there are significant differences in performance on the WikiText-2 test.

We performed ablation studies W4A4KV16 on LLaMA3-8B and LLaMA2-7B and selected a data sample for calibration to optimize rotation matrix. After that, we use GPTQ with 128 samples from WikiText-train as paper did. The results are follows:

We believe that these reasons may all be the causes for the optimization of DFRot being relatively insensitive to the calibration dataset distribution:

• For a model, tokens with massive activations in LLMs tend to exhibit only a few similar data distributions because these tokens are often produced by out_proj or down_proj layers with large weights [2].
• GPTQ will use 128 samples with a length of 2048 to calibrate the weights, which reduces the impact of the sample size during rotation matrix calibration.

[1] FlatQuant: Flatness Matters for LLM Quantization

[2] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs