Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference

Thanks for your comments and suggestions, which are valuable for enhancing our paper. The following are responses to individual concerns:

Novelty: Previous works have untaclked problems for INT4 inference: 1. unmatched smoothing scales would fail for smoothing (SmoothQuant); 2. channel-wise smoothing would bring effect of victim (SmoothQuant); 3. rotated activations may still have channel-wise outliers and be sub-smooth for quantization (QuaRot). In this work, we employ Runtime Smooth to address the sub-smooth issue of rotation, and we also use a rotation technique to address the victim effect. We introduce a seamless and effective integration of the two techniques, which is non-rival.

Moreover, the low efficiency of Runtime Smooth stems from its incompatibility with the hardware computation scheme, which only allows for the computation of one column of the matrix per computation unit. Therefore, we incorporate the reordering technique to collect outliers and normal values. Next, we apply the smoothing scale group-wise, which can approximate smoothness and align with the hardware computation scheme. The reordering technique in ATOM and LLM.int8() serves for mixed precision quantization, which share different motivations with us.

More downstream tasks:

Limited by computation resources, more downstream tasks and models are under testing.

Kernel performance comparison: We have carried out experiments to compare the performance of the kernel, as illustrated in Figure 4. ATOM uses the sub-channel A4W4 scheme, which brings more overhead compared with our method. For QServe, it only supports A8W4 and utilizes the A8W8 GEMM kernel. It is unfair to conduct comparisons between A4W4 and A8W8 since A4W4 lacks the support from hardware.

We thank you for the presentation suggestion and will improve the presentation in the final version.

We appreciate your comments and suggestions that truly enhanced the quality of our paper.

Meaning of Figure 1 (W1, W2): We present Figure 1 to highlight the issue that the existing methods fail to solve: SmoothQuant. SmoothQuant relies on a pre-computed smoothing scale that originates from the calibration set. It would fail when faced with out-of-distribution (OOD) samples, whose channel-wise maximum values cannot match the pre-computed ones.

(b) SmoothQuant still fails for INT4, even when the unmatched problem is addressed by computing the smoothing scale online. The reason for this failure is that SmoothQuant must convert the activation outlier into weights, which makes the weight quantization process extremely challenging.

(c) Addressing the above problems, we find that the channel-wise smoothing method is still bound by the spike outliers. The spike outliers have relatively small elements within the same channel. Hence, applying channel-wise smoothing would make those elements zero and cause the loss of information.

Specifically, the difference between (a) and (b) comes from using the different smoothing scales. The former (SmoothQuant) uses the pre-computed one from the calibration dataset, which could be unmatched with the OOD samples. The latter computes the smoothing scale online and solves the unmatched problem. However, even when solving the unmatched problem, SmoothQuant still fails INT4 due to harder quantization for weights, which is the core idea of (b).

Motivations (W3): We share the same motivation to address outliers during quantization as DuQuant, QuaRot, and SpinQuant do. However, we have observed that the rotated activations may still contain channel-wise outliers and remain sub-smooth for quantization purposes. Moreover, the rotation technique can assist in mitigating the victim effect, which poses a significant threat to the channel-wise smoothing method. Based on these two factors, we propose the Rotated Runtime Smooth method.

Experimental support for claims (W4): Good point! We have validated the importance of matched scales as shown in Figure 3. However, it appears that we neglected to demonstrate the ease of unmatching the fixed smooth scales from the calibration set. To validate the fact, we reparameterize the layer norm and weights according to a fixed smooth scale from the calibration set. Then we collect the activations from WikiText-2-Raw-V1. If the activation matches the smooth scale, u = max(|t|)/RMS(t) tends to be 1. For comparison, we also collect activations without reparameterizing.

The results validate that fixed runtime smooth scales are easy to unmatch, which leads to performance drops.

Provide further explanations for the terms (W2, W5, and W6): Thanks for your suggestions. We will add more explanations for the terms in the paper.

Incomplete ablation study (W7): Here we detail the components. including 1. online smoothing, 2. reordering, and 3. a rotation technique. The reordering serves for the online smoothing from the perspective of efficiency; hence, it could not stand alone. The possible combinations are as follows: 1. (1); 2. (1); 3. (1+2); 4. (3); 5. (1+3); 6. (1+2+3). 1. denotes GPTQ (Table 1); 2. denotes Runtime Smooth (Table 1); 3. denotes Runtime Smooth with reordering (Table 4); 4. denotes QuaRot (Table 1); 5. denotes Rotated Runtime Smooth (Table 4); 6. indicates Rotated Runtime Smooth with reordering (Table 1). Hence, the ablation study is complete.

Additional comparisons with existing methods (W8): The results show that our method outperforms the existing methods, and their variants featured with Runtime Smooth can also achieve better performance.

Dear reviewer. As the rebuttal deadline is approaching, I want to kindly mention that you have not responded to my rebuttal. Timely feedback from yours would be constructive in preparing a comprehensive revision.  I understand everyone has a busy schedule, and I truly appreciate your efforts. Thank you for your time and support.  Best regards.

We appreciate the recognition of our contributions. We are delighted to explain the questions and make corresponding adjustments in the final version.

The meaning of Figure 1: We present Figure 1 to point out the problem not solved by the existing methods - SmoothQuant. (a) SmoothQuant relies on a pre-computed smoothing scale from the calibration set. It would fail when faced with out-of-distribution (OOD) samples, whose channel-wise maximum values cannot match the pre-computed ones.

(b) Even when addressing the unmatched problem by computing the smoothing scale online, SmoothQuant still fails for INT4. The reason is that SmoothQuant has to migrate the outlier of activation to weights, making quantization for weight unbearable.

(c) Addressing the above problems, we find that the channel-wise smoothing method is still bounded by the spike outliers. The spike outliers have relatively small elements within the same channel. Hence, applying channel-wise smoothing would make those elements zero and cause the loss of information.

For “it is unclear what the value of s represents (in relation to the equations in the text) and how exactly it reduces channel-wise outliers”, we have explained it in lines 151–153. s denotes the smoothing scale. It migrates the channel-wise outlier from activation to weight, tending to balance the quantization difficulty between activation and weight. We will make it more clear in Figure 1 for better understanding.

For “effectively resolves the victim issue caused by spike outliers.” We claim that the victim effect is natural for smoothing-based methods (lines 162–164). And it motivates us to integrate rotation operations to address the spike outliers. Thanks for your suggestions. It is valuable to clarify the relationship between the victim issue and smoothing-based methods in the final version.

Description of the reorder process and the meaning of Figure 4: The reorder process is to rearrange the channels of the weight and activation correspondingly, where the final result is equivalent. Specifically, the operation is defined as:

Y = X @ W^T = \sum_{i=1}^{K} X_i @ W_i^T

where X_i \in R^{n \times 1} and W_i \in R^{m \times 1} represent the $i$-th channel of X and W , respectively. K denotes the total number of channels.

The activations and weights are rearranged and represented as X' and W'. Hence, given that X'_i = W'_i, the final result remains consistent. We will provide further clarification in the final version.

For the pipeline of Runtime Smooth, we have discussed it in sections 3.1 and 3.2. And we will make it more clear in the final version.

Why not lower quantization precision: It is a good question. Quantization can be applied to weights and activations for different purposes. For weight-only quantization, it is to reduce the model size and I/O cost. To further accelerate inference, we should apply quantization to both weights and activations and utilize low-precision multiplication kernels, e.g., 8-bit for Hopper and 4-bit for Blackwell. For lower precision activation, it suffers from a heavy accuracy drop and has no gain in acceleration since it is not supported by the hardware. Moreover, the lower precision multiplication kernels are faced with numerical overflow and underflow problems. Hence, we do not apply lower precision quantization to activation. We have discussed related information in Lines 36-40.

For the mentioned low-bit quantization, e.g., one-bit LLM, they focus on weight-only quantization. Whereas our proposed method focuses on activation smoothing.

Hope the above explanations are helpful. We are pleased to have further discussion and are expecting your positive feedback.

We appreciate your suggestions. Lower quantization precision necessitates heavy quantization-aware training, such as one-bit LLM, to ensure accuracy recovery. Most of these works serve for weight-only quantization. To make a fair comparison, we apply activation quantization on top of weight quantization. Based on the quantization-aware training checkpoint (Llama3-8B-1.58-100B-tokens) from one-bit LLM, we conduct activation-weight quantization and testing on WikiText PPL. The result is shown below.

We apply Runtime Smooth, Rotation, and Rotated Runtime Smooth for activation smoothing. The first four rows validate the effectiveness of those validation methods, especially our proposed Rotated Runtime Smooth. Comparing the second row with the fifth row, it shows that quantization-aware training with only 1.58B fails to improve accuracy of  INT4 inference with origin checkpoint.  However, comparing the third row with the last row demonstrates that QAT effectively addresses lower precision (3-bit) inference. Although the current hardware does not support lower precision inference, it shows the potential to gain better accuracy with QAT. Moreover, though there are pioneers working on QAT with both activations and weights (BitNet a4.8)[https://arxiv.org/html/2411.04965v1], it is still worthy to explore the potential of QAT with a rotated weight for better accuracy on extremely low precision inference.

Weight-Acitvation Quantization on LLaMA3-8B-Instruct:

However, as initially claimed, QAT requires heavy retraining. To observe the performance of the proposed method without QAT under lower precision, we conduct experiments on LLaMA3-8B. The result in the following table shows that the proposed methods are easy to fail under lower precision.

Weight-Acitvation Quantization on LLaMA3-8B:

Weight-Only Quantization on LLaMA3-8B:

In summary, due to heavy retraining and unstable performance, it is still inefficient to apply QAT. And we believe our proposed training-free activation smoothing methods can benefit the community. Hope the above information could help you and we are expecting your positive feedback.

Thanks for acknowledging our contribution! We are pleased to explain the overhead brought by Runtime Smooth.

Section 4.5 has discussed the runtime overhead and Figure 6 shows that Runtime Smooth would bring negligible overhead compared with the original A4W4 per-channel setting. The 'Tops' on the y-axis represents the elements processed each second, which can reflect the runtime overhead.

The complete efficiency metrics are listed below:

A4W4 Per channel Quantization (QuaROT)

A4W4 Per channel Quantization + Runtime Smooth (Ours)

A4W4 Sub channel Quantization (ATOM)

Hope the above explanations are helpful. We are pleased to have further discussion and are expecting your positive feedback.

Thanks for your valuable suggestions, which help us enhance the clarity of this paper and make it more understandable for the wider community. We thoroughly examined your suggestions and our manuscript to rectify any typos, ambiguous descriptions, and unclear settings in the revised version.

We have tested methods on WikiText-2 PPL (Table 1) and Common Sense QA (Table 2). To address performance concerns, we have incorporated additional downstream tasks such as MMLU, GSM8k (Math), CEVAL (Chinese), HumanEval (code). The results below indicate that our method surpasses QuaRot in all tasks. The non-trival improvement on GSM8k demonstrates the effectiveness of runtime smoothing for data from diverse domains.

Limited by computation resources, more downstream tasks and models are under testing.

We have also included additional baselines to ensure a fair comparison. Please refer to the global rebuttal for further details.

Moreover, we appreciate the suggestions for testing on other modalities. We will consider this in our future work.

Dear reviewer. As the rebuttal deadline is approaching, I want to kindly mention that you have not responded to my rebuttal. Timely feedback from yours would be constructive in preparing a comprehensive revision.  I understand everyone has a busy schedule, and I truly appreciate your efforts. Thank you for your time and support.  Best regards.