PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs

Thank you for your comment. To address your concern, we reexplain how prefixquant works, and add the speedup results of 70B models.

Weakness 1 (Question 1): It is not clear how the proposed method works. Specifically, I don't understand why prefixing certain outlier tokens in the KV cache prevents the generation of new outlier tokens. Is there a skipping mechanism in the autoregressive generation, or is the method intended to make the KV cache more quantization-friendly? I doubt the effectiveness if it's the latter.

Answer 1: There seems to be a misunderstanding regarding our paper. In autoregressive inference, prefixing tokens in the input sequence or in the KV cache yields the same results. The advantage of prefixing in the KV cache is that it avoids corresponding computation on linear layers. Our method ensures outliers appear in initial tokens, which we then store in the KV cache to prevent their computation in down_proj layers. As mentioned in line 336, outliers only occupy a few tokens (1-4) per sequence. The positions of these outliers vary with the input sequence (see Figure 4b). By prefixing high-frequency outlier tokens, we confine them to prefixed positions (see Figure 4c). Thus, placing these tokens at the start of the KV cache achieves the desired outcome.

Weakness 2 (Question 2): The performance without fine-tuning is not strong, especially on the 70B models. It would be helpful to add a column in Table 3 indicating whether other methods are fine-tuned, for better clarity.

Answer 2: While the PrefixQuant model without fine-tuning is not exceptionally strong, it excels in efficient static quantization. Other methods suffer significant performance degradation under the same settings. For instance, W4A4KV4 Llama-3-8B with QuaRot in static quantization results in perplexity exceeding 100. In Table 3, Atom and DuQuant do not use fine-tuning, whereas QuaRot relies on additional GPTQ, and SpinQuant uses both fine-tuning and GPTQ. We will clarify this in the final version.

Weakness 3 (Question 3): Do the authors have results with 70B models, and do they still observe a speedup? If not, these limitations should be clearly addressed.

Answer 3: Table 9 in our paper presents the speedup of linear layers with 70B model shapes (8192x8192 and 8192x28672). Static quantization is faster than dynamic quantization. For end-to-end prefilling speedup, since the 4-bit Llama-3-70B can be loaded on an RTX 3090 GPU, we confirmed effectiveness on A100 GPUs only. As shown below, PrefixQuant achieves a 1.2x speedup over QuaRot.

We sincerely appreciate your time and effort in reviewing our paper. Please let us know if you have further inquiries.

Very interesting result, but I don't understand how it can help just to insert outliers into the cache. Does this assume that your 4-bit KV cache has a separate scale factor for the prefixed tokens than for the rest of the entries in the cache? Or are your outlier, prefixed cache entries somehow left in fp16 format? In the latter case, this is not hardware friendly. I would very much like to understand how these prepended tokens in the cache can have a dramatic affect on the subsequent responses without studying the code, as this is the key insight to your work, and thus the key idea to clarify to your readers. Perhaps a bit of pseudocode as to how to implement the cache would help.

I should say that my mental model of KV Cache is that all it does is store, in order, all the tokens generated so far so that it doesn't have to recompute them all when it generates the next token. In this case, having precomputed certain tokens at the beginning doesn't help when you encounter them again in the middle of the prompt or response, such as '\n' or <space> tokens which may appear mutiple times within the sequence.

Our method ensures outliers appear in initial tokens, which we then store in the KV cache to prevent their computation in down_proj layers. As mentioned in line 336, outliers only occupy a few tokens (1-4) per sequence. The positions of these outliers vary with the input sequence (see Figure 4b). By prefixing high-frequency outlier tokens, we confine them to prefixed positions (see Figure 4c). Thus, placing these tokens at the start of the KV cache achieves the desired outcome.

I think I understand this part, my confusion comes from the fact that in the autoregressive generation phase, my model may start to generate new outlier tokens? For example, if I generate a new "\n" in my decoding phase, how do you add this to your KV cache?

Table 9 in our paper presents the speedup of linear layers with 70B model shapes (8192x8192 and 8192x28672)

I don't think single layer profiling helps, do you have whole model perf results.

Thank you for your comment.We reclaim the novelty of our paper, and explain why we need to remove token-wise outliers.

Weakness 1.1: The main concern is the novelty of the proposed method. The idea of adding prefix tokens to mitigate outliers has been explored by previous works: QFeP (Yang et al., arXived May 2024) and CushionCache (Son et al., EMNLP 2024).

Answer 1.1: Prefixed outlier tokens in KV cache have been explored in works like Massive Attention (Sun et al., COLM 2024) and Attention Sink (Xiao et al., ICLR 2024), alongside QFeP and CushionCache. However, PrefixQuant is the first to provide a system analysis of these outliers. As detailed in Section 4, we address upper outlier tokens in inputs and lower outliers in Q/K/V, which were not covered by previous works. Our method identifies outlier tokens within 10 seconds, compared to ~10 hours for CushionCache on Llama-3-8B. Additionally, PrefixQuant is the first to facilitate static quantization in both KV cache and activation.

Weakness 1.2: The advantage of less computation time seems not very critical practically (as these are one-time costs) and does not stem from a technically novel component.

Answer 1.2: We respectfully disagree. Compression time is crucial when deciding whether to adopt a compression technique. This is why methods like GPTQ and AWQ are popular—they complete compression quickly. Although finding the prefixed token is a one-time cost per model, the numerous models make this efficiency significant.

Weakness 1.3: The advantage seems to come mainly from using Hadamard rotation, grid search, and block-wise fine-tuning, which are not original contributions. I recommend comparing the prefix optimization method with CushionCache and QFeP, excluding additional components.

Answer 1.3: In head-to-head comparisons of prefixed tokens, all three papers—PrefixQuant, CushionCache, and QFeP—resolve token-wise outliers effectively, as shown in there activation distribution visualizations. However, PrefixQuant provides more comprehensive system analysis, covering both the KV cache and input of linear layers, unlike the others that focus solely on linear layers. Additionally, PrefixQuant identifies outliers 3600 times faster on Llama-3-8B (10s vs. ~10h) compared to CushionCache. We included Hadamard rotation in our comparisons because it has become a standard component used by several methods, including QuaRot, DuQuant, SpinQuant, and QoQ.

Weakness 2: Removing channel-wise outliers should resolve token-wise outliers logically. I request a more concrete justification.

Answer 2: Let me define outliers: Channel-wise outliers occur at fixed channel indexes, while token-wise outliers appear in specific tokens. Figure 1 in our paper illustrates this. As shown in Figure 1(a), outliers greater than 1,000 are present only in specific tokens, termed token-wise outliers. Figure 1(b) shows that after applying channel-wise alleviation methods like Rotation (QuaRot), outliers are redistributed across token channels. QuaRot reduces outliers to nearly 15 but still struggles with non-uniform distribution. Figure 1(c) demonstrates how our PrefixQuant isolates outlier tokens, reducing maximum values to nearly 0.07.

Weakness 3: The authors could include evaluations on more realistic tasks, such as GSM-8k or MMLU.

Answer 3: The table below shows our comparison results on the MMLU dataset, which is sensitive to quantization. QuaRot's performance collapses on MMLU, but PrefixQuant consistently outperforms previous dynamic quantization methods even without fine-tuning.

Weakness 4: The claim that static per-tensor quantization by PrefixQuant outperforms existing dynamic methods does not seem entirely true.

Answer 4: We detail comparison results in lines 465-480. We primarily claim that PrefixQuant outperforms existing methods in W4A4KV4 and W4A8KV4 settings. In the W8A8KV8 setting, PrefixQuant achieves comparable performance with existing methods, with its main advantage being efficient per-tensor static quantization.

We sincerely appreciate the time dedicated to reviewing our paper. If you have further inquiries, please let us know.

Thank you for your comment. To address your concern, we have added performance comparisons in a long-context setting (8192 context length). This demonstrates that per-tensor quantization of PrefixQuant can outperform previous per-token dynamic quantization methods. We also present the MMLU results and speedup with an 8192 long context.

Weakness 1: Lack of comparison in a long-context setting

Answer 1: The following table shows that PrefixQuant, without fine-tuning, outperforms previous per-token dynamic quantization methods in both 4-bit and 8-bit activations at a context length of 8192 (the maximum length of the original Llama-3-8B). These results demonstrate that PrefixQuant is the first to enable efficient per-tensor static quantization, outperforming the costly per-token dynamic quantization.

Weakness 2: More experiments are needed to assess the effectiveness on challenging subjects like MMLU.

Answer 2: The following table illustrates the comparison results on MMLU datasets, which are more sensitive to quantization. It shows that the performance of QuaRot collapses on the MMLU dataset. However, PrefixQuant consistently outperforms previous dynamic quantization methods, even without fine-tuning.

Question 3: It would be better if the authors measured the real time-to-first-token (pre-filling) speed-up with longer context length (e.g., 8192) than 2048.

Answer 3: We tested the W4A4 Llama-3-8B pre-filling speedup compared to FP16 with a batch size of 1 and a context length of 8192. As shown in the table below, PrefixQuant achieves a 1.83x speedup on the A100 and a 3.02x speedup on the RTX 3090.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know.

Dear Reviewer xJQ6,

We have added performance comparisons in a long-context setting (8192 context length) to address your primary concern. Additionally, we would like to emphasize that static quantization is particularly practical, especially for edge devices such as NPUs, which often do not support dynamic quantization. We look forward to your feedback and appreciate your time and consideration.