LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Q1: More discussions to the previous work.
A1: Thanks for suggesting additional papers for comparison and ablation experiments. It's crucial to note that [1], [3], and [4] represent concurrent works yet to be published by the ICLR submission deadline. As per the ICLR reviewer guidelines, not comparing to these works shouldn't be grounds for rejection. The rapid evolution within the Language Model Models (LLMs) community makes it challenging to track every new development. Regarding [2], it focuses on using generated data for data augmentation in cross-lingual semantic parsing tasks. In contrast, our emphasis lies in showcasing that exclusive utilization of generated data can effectively fine-tune precise quantized LLMs. The primary focuses between the two approaches are distinct. Nonetheless, these papers are good work and we'll incorporate discussions about them in the related works section.

Q2: The 4-6-16 setting of experiments is a little bit strange and unfriendly to hardware. I think the paper should move 4-4-4 experiments that take 4-bit weight and activations from the appendix to the main body.
A2: Thanks for this suggestion. We will move the 4-4-4 experiments to the main table. it's noteworthy that W4A6 quantization can effectively reduce GPU run-time memory usage compared to W4A8. Additionally, it operates at a speed comparable to W4A8 when using dequantization.

Q3: I find the fifth row in Table 1 looks strange. It seems that the performance of GPTQ is really low, even lower than RTN with 4-8-16 bits.
A3: Regarding our previous experiments with GPTQ for LLaMA-7B, we found that the GPTQ is sensitive to calibration data choice. Initially, our use of C4 to calibrate all tasks wasn't optimal. Adjusting the calibration dataset to match the evaluation dataset yielded improved results for W4A16 GPTQ, as shown in the following table and will be reflected in the updated Table 1.

Q4: Experiments can be more convincing if the paper also conducts on other structures like LLaMA-2, and others take the vanilla Transformer structure as LLaMA adopts a different one.
A4: Following your suggestion, we conducted experiments on the OPT-125M model, revealing that the W4A16KV16 LLM-QAT model achieves performance on par with full-precision models. Detailed results for other bit settings will be incorporated into the manuscript.

Q5: Based on my practical experience and discussion with some people in this field, we find quantizing KV cache with per-token quantization on LLaMA models will not incur much accuracy decline, even on a 4-bit case. However, results in the main table seem to give an opposite conclusion. Also, the paper claims that distribution for LLaMA models is usually symmetric. However, we find products of the output of gate and up functions in LLaMA are not so.
A5: Regarding your experiment showing a loss-less accuracy in KV-cache quantization, we're curious if you tested the model on generation tasks that construct sequences token by token. Typically, common evaluation frameworks assess perplexity for datasets like wiki and C4, by predicting the next token based on the ground truth context, instead of the generated tokens, which is similar to the training phase. In that case, the KV cache is not used. Additionally, we confirmed that the gate function output distribution isn't symmetric-like. While symmetric quantization works well for LLaMA models, we'll modify the claim to prevent potential misinterpretation.

Q1: Even though the quantized models still maintain the quality closely, there exist non-trivial gaps especially when the weights are quantized to 4-bits. Especially, with additional training (QAT), it is expected the quality is very close to the floating point unquantized model.
A1: Compared to small models, Language Model Models (LLMs) exhibit distinctive characteristics, where outliers play a significant role. The reviewer's assertion that "additional training (QAT) should yield a quality very close to that of the floating point unquantized model" seems grounded in observations from smaller models, such as BERT or BART. Considering the fact that even for 8-bit quantization, LLMs necessitate special approaches like SmoothQuant [1] or outlier suppression [2], where naive Post-Training Quantization (PTQ) typically suffices for smaller models, it is not surprising that quantizing the LLMs to W4A8 remains a challenging task and still has some gap.
[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.
[2] Dettmers, Tim, et al. "LLM.int8(): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022).

Q2: Also, some PTQ methods such as (Li, Qingyuan, et al. "FPTQ: Fine-grained Post-Training Quantization for Large Language Models." arXiv preprint arXiv:2308.15987 (2023).) claims better quality.
A2: Regarding the FPTQ paper, it represents concurrent work that is not yet to be formally accepted. Per the ICLR reviewer guidelines, this should not be grounds for rejection. Notably, this paper also shows comparisons with our methodology, as FPTQ adopts a higher granularity, this leads to the higher accuracy.

Q3: Typos; Why 3-5 tokens? What exact number is used?
A3: We appreciate your feedback on the typos, all of which will be rectified in our manuscript. To provide specifics, our methodology used {3, 4, 5} deterministic tokens in each round for the first token in 32k vocabulary across three rounds, resulting in a total of 3 x 32k = 96k sentences. Additionally, 4k sentences were generated using random numbers within the range of 3-5. These generated sentences composed our 100k-sample finetuning dataset.

Q1: The techniques utilized in the paper (QAT using StatsQ and LSQ, MinMax quantization, knowledge distillation) have been shown to work in the quantization setting before, so the contributions are not particularly novel. Effectiveness of MinMax for language models due to outliers has also been demonstrated earlier.
A1: The primary focus of our paper lies not in the design of the quantization function but rather in presenting an innovative data generation pipeline facilitating Quantization Aware Training (QAT) for Language Model Models (LLMs) within feasible computational constraints, as detailed in Lines 32-42 of our manuscript. The previous works Vicuna [1] and Alpaca [2] stand as the most relevant works in this domain, while a distinct divergence exists between their methodologies and ours. Vicuna [1] relies on user-uploaded ShareGPT data for GPT instruction fine-tuning, whereas Alpaca [2] necessitates predefined human prompts for data generation, meticulously curating categories for diversity. In contrast, our approach omits human prompts or user data, initiating QAT fine-tuning data generation with just a single random initial token. To our knowledge, this is yet unexplored in current literature. Our quantization method utilizes standard MinMax quantization (Eq.3).
[1] Chiang W L, Li Z, Lin Z, et al. Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
[2] Taori R, Gulrajani I, Zhang T, et al. Alpaca: A strong, replicable instruction-following model[J]. Stanford Center for Research on Foundation Models.

Q2: More ablation studies using a subset of each dataset and comparing that to generated data.
A2: Thanks for the suggestion. While the full-training data of the LLaMA model remains unavailable publicly, we utilized the Red Pajama dataset (a substantial LLM pre-training dataset containing 1 trillion tokens) to address concerns. We sampled 100k sentences from this dataset for a fair comparison to our method. In the following table, we obtained an average score of 61.60 on zero-shot reasoning tasks, slightly surpassing fine-tuning on the C4 dataset but still falling short of the results achieved through finetuning on the generated dataset. This result underscores that sampling a small subset from an extensive pretraining dataset struggles to replicate the dataset's distribution adequately. Conversely, the diversity inherent in data extracted from pretrained LLMs via diverse initial tokens proves more suitable for QAT model finetuning. Nonetheless, this comparison holds value and will be included in our manuscript.

Q3: Have the authors considered using a larger teacher model for the distillation data generation?
A3: Regarding leveraging logits from a superior pre-trained model, we conducted experiments by distilling a LLaMA-7B model using the LLaMA-13B model, shown in the following table. Interestingly, employing the 13B model as the teacher resulted in decreased accuracy compared to using the 7B model. We consider it crucial to employ the same architecture for both teacher and student models. This shared architecture ensures initial weight alignment and promotes the closest distribution match. As a result, the teacher model can effectively guide the student model, aiding in error compensation introduced by quantization.

Q1: Compare LLM-QAT only to training-free methods such as round-to-nearest (RTN) and SmoothQuant [4].
A1: The comparison to the SmoothQuant paper is extensively discussed in our main Tables (Table 1 and 2). We also compared with another SoTA PTQ method for LLMs, GPTQ. These two PTQ methods are the state-of-the-art PTQ methods for LLMs. In the GPTQ paper, thorough comparisons were made between their method and various Post-Training Quantization (PTQ) approaches on different language models, showcasing GPTQ's superior performance. Hence, we select the most effective PTQ method for LLMs as a benchmark for comparison with our approach. Reproducing other PTQ methods on Language Model Models (LLMs) within a short timeframe poses challenges, and we avoid presenting rushed and potentially suboptimal results for other methods.

Q2: The arguments would be more convincing with additional references and explanations for why LLM-QAT primarily focuses on the W4A8 quantization format.
A2: It's worth noting that we did NOT only do experiments on W4A8; rather, we experimented with various bit settings, including W4A6, W4A16, W8A8, and W8A16, etc, detailed in in our main Tables (Table 1 and 2). Therefore, we don’t quite agree with reviewer's comment that 'LLM-QAT primarily focuses on the W4A8 quantization format'.

Q3: Are there experimental results available for other models?
A3: Following your suggestion, we conducted experiments on the OPT-125M model, revealing that the W4A16KV16 LLM-QAT model achieves performance on par with full-precision models. Detailed results for other bit settings will be incorporated into the manuscript.

Q4: ZeroQuant presents data-free quantization that employs random data for generative language models.
A4: We'll include a discussion on ZeroQuant in related works. We would like to emphasize two key distinctions: (1) the random tokens in ZeroQuant is not generated from the first token using LLMs, rather, they employ entirely random tokens for entire sentences, differing from our methodology. (2) the quality requirements for calibration data in PTQ are significantly less demanding compared to the quantity needed for training data in QAT.