RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization

We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.

Concerned about the setting of hyperparameters.

1. For hyperparameters $\tau_c$ and $\tau_f$

   We have explicitly added the sentence "the performance of RWKVQuant can vary depending on the setting of $\tau_c$ and $\tau_f$, see Appendix A.5" to our latest manuscript after line 274 (first column).

Actually, $\tau_c$ and $\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:

• Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized.
• Set $\tau_c$ to the value at the 50th percentile of all $P_c$.
• Compute the fine-grained proxy $P_f$(Eq.17) for each layer whose $P_c$ < $\tau_c$.
• Set $\tau_f$ to the value at the 20th percentile of all $P_f$.

Although fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. We have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B in Table R1.

2. K represents the order of the Taylor expansion, we set it to 4 for all models.

A larger number of K offers a closer result to the actual value (i.e., K=+∞). Experimentally, we find that K=4 is an ideal trade-off between accuracy and quantization cost under various $\tau_c$ and $\tau_f$. For instance, in RWKV-6-1B Table R2.

Writing Errors.

Thanks for pointing them out. We have fixed them in our latest manuscript.

Concerns about calibrating from the test dataset.

1. RWKVQuant does not include any training process. All samples that are used for calibration are not used to monitor performance. They are only used to generate activations, rather than supervising.

2. For a fair comparison, we randomly sample 128 sequences from Lambda, following previous work like GPTQ [1] and GPTVQ [2]. In addition, we evaluate on various datasets that are not used for calibration, as shown in Table 2 in our manuscript. To further address your concern, we provide Table R3, where calibration is performed using only a small subset (128 samples) of the training dataset under the same conditions. The results indicate that our approach still maintains an advantage.

Does "calibration" here mean the procedure for setting $\tau_c$ and $\tau_f$ such that 90% of the layers use SQ and 10% of the layers use VQ?

Calibration and proportion are two distinct concepts. Calibration is an common practice of PTQ, which samples data and feed it to models to generate activations. Making use of such activations, PTQ can optimize the weight quantization [1,2]. As for the proportion of SQ and VQ, it is fixed to our default configuration in this work corresponding to the first reply.

I would like to see an ablation where this percentage of layers using SQ vs VQ is changed and the performance of RWKVQuant is tracked.

Please refer to the Table R1.

Why is the setting of $\tau_c$ and $\tau_f$ are not the same?

The values of $\tau_c$ and $\tau_f$ can vary across different models. This variation occurs because these values are dictated by the percentile of the proxy specific to each individual model.

References:

[1] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.

[2] GPTVQ: The Blessing of Dimensionality for LLM Quantization.

We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.

Why authors choose to use hybrid quantization instead of only use VQ since the outliers in applying SQ to RWKV is unsolvable.

VQ and SQ are suitable for differently distributed weights. VQ excels in managing outliers but underperforms when dealing with uniform distributions. For weights without outliers, SQ can achieve better results. Thus, we are motivated to hybrid them.

Concerns of Eq. 11.

Thanks for pointing it out. We have fixed this problem in our latest manuscript.

Why the $s_k$ in Eq.15 can be omitted.

Eq.15 is an equal expression of the Taylor expansion, but Eq.17 is obtained by performing an absolute operation to Eq.15. Thereby $s_k$ can be omitted. We explain the reason for this operation in step5 of the fine-grained proxy (line 241-245).

Concerns on the fine-grained proxy.

Fine-grained proxy is not just a Taylor expansion version of coarse-grained one. It extracts the outliers features by performing an absolute function to specific terms of the Taylor expansion, as stated in step5 of the fine-grained proxy (line 241-245).

Authors should add more comparison about different setting of $\tau_c$ and w/ w/o $\tau_f$.

Actually, $\tau_c$ and $\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:

• Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized.
• Set $\tau_c$ to the value at the 50th percentile of all $P_c$.
• Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\tau_c$.
• Set $\tau_f$ to the value at the 20th percentile of all $P_f$.

Although fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. We have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B in Table R1.

It would be better to further explain the novelty and differences between the codebook optimization and previous arts.

Firstly, our approach is designed for element-wise multiplication operations (widely-applied in RWKV), instead of matrix multiplication (widely-applied in Transformer-based LLMs). Compared to matrix level, element-wise poses a challenge of batch integration for calibration.

Secondly, we discover that the activations of RWKV follow the Guassian distribution as shown in Figure 4. Thus, our codebook optimization can be well applied.

Thirdly, directly applying previous arts to RWKV leads to significant accuracy decline as shown in Table 7 of the manuscript. Our method can effectively enhance performance.

How the performance changed with different settings of proportion of SQ and VQ?

1. VQ and SQ are suitable for differently distributed weights. VQ excels in managing outliers but underperforms when dealing with uniform distributions. For weights without outliers, SQ can achieve better results. Thus, we are motivated to hybrid them.

2. It is possible to improve accuracy by adjusting the percentiles for each network. However, this default configuration can already achieve effective results across all RWKV networks and typical tasks as shown in Table 2 of the manuscript.

Why are VQ methods much better than all other previous work?

Generally, SQ is not good at handling outliers for its scaling characterstic, especially in aggressive bitwidth (e.g., lower than 4 bit). In contrast, VQ has the ability to capture the distribution of the original sequence and offers an advantage in terms of compression ratio [1].

Results on other bit-width settings.

Please refer to the Table R1 in the above response.

The effectiveness of codebook optimization is much weaker with larger-scale RWKV.

Generally, larger-scale models are less sensitive to quantization, primarily due to their larger amount of redundant parameters as mention in [2,3]. Thus, the optimal quantization strategy may be less obvious for these models.

Symbol conflict in Eq. 6.

Thanks for pointing it out. We have fixed this problem in our latest manuscript.

The explanation of superscript k in Eq. 12 is missing.

We explain it next to Eq. 12 in line 188-190.

Symbol reuse: m.

Thanks for pointing it out. We have fixed this problem in our latest manuscript.

References:

[1]. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

[2]. A Survey of Quantization Methods for Efficient Neural Network Inference

[3]. LLM Inference Unveiled: Survey and Roofline Model Insights

We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.

The paper would benefit from a broader comparative analysis with other advanced quantization methods.

1. Directly applying SOTA methods on Transformer-based LLMs on RWKV models can lead to significant accuracy degration. For example, applying QuaRot [5] to LLaMA series has only 1% accuracy drop, while that for RWKV series is over 8%.

2. There are few works that apply quantization for RWKV models. We are the first to combine SQ and VQ, achieving effective performance on the RWKV family.

Concerns of Overfitting:

The characteristic of PTQ is that it enables rapid quantization deployment with a small calibration dataset. Current SOTA methods of Transformer-based LLMs do not encounter this issue.

We also care about the overfitting challenges on PTQ for RWKV. We perform experiments on nine datasets, which are not used for calibration (i.e., zero-shot). As shown in Table 2 of the manuscript, RWKVQuant does not overfit.

Lack of pseudo-code:

Thanks for you adivce, we've added an Algorithm to our latest manuscript.

Can this method be applied to activation quantization as well?

Considering that bandwidth is the main bottleneck of LLMs' inference speed, this work is put forward to tackle the challenges associated with weight quantization. Moreover, applying this quantization approach to activations represents a highly promising and valuable area of research. We intend to conduct in-depth explorations of this aspect in our forthcoming research endeavors.

In Table 4, why does the speedup increase with model size? With weight quantization alone, what measures should be taken to achieve actual speedup?

The actual speed up can be effected by various factors, such as instruct latency, memory access, computational time, optimization of de-quantization kernal, and so on.

Generally, memory access takes the most importance across all above factors, since the decode phase is typically memory-bound. With the model size increase, memory access will account for more latency. Considering that weight quantization can reduce this cost, the speed up should also increase with the model size.

What is the actual cost of the codebook optimization step?

The codebook optimization is a lightweight offline process; for instance, it only takes about 15 minutes for a 70-parameter model.

Could you provide specific strategies for tuning the parameters $\tau_c$ and $\tau_f$?

Actually, $\tau_c$ and $\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:

• Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized.
• Set $\tau_c$ to the value at the 50th percentile of all $P_c$.
• Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\tau_c$.
• Set $\tau_f$ to the value at the 20th percentile of all $P_f$.

Although fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. We have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B in Table R1.

References:

[1] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.

[2] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.

[3] GPTVQ: The Blessing of Dimensionality for LLM Quantization.

[4] VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models.

[5] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.

Only empirical setting of $\tau_c$ and $\tau_f$ mentioned in experimental section.

Sorry for causing this confusion. Actually, $\tau_c$ and $\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:

• Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized.
• Set $\tau_c$ to the value at the 50th percentile of all $P_c$.
• Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\tau_c$.
• Set $\tau_f$ to the value at the 20th percentile of all $P_f$.

Although fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. We have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B in Table R1.

Insufficient analysis of optimal quantization strategies across different model scales. No related discussion in experimental section.

In line 347-353 of the manuscript, we have disscussed this point. Here is a takeaway for your convenience. As shown in Table 2 in original paper, on small-scale models, our method reduces the error by 14.5% compared to the SOTA; on larger-scale models, our method performs almost consistently with floating-point precision and reduces the error by 62.34% compared to the SOTA. The following table compares the average zero-shot accuracy between origin models and our quantized models. It can be observed that our method is only of slight accuracy decline across different model sizes .