1. Deeper theoretical insights

We agree that a theoretical treatment of the effect that each compression scheme has on a model’s outputs would be useful, however, it requires more progress on AI theory research before a rigorous theoretical framework can be created. Hence we leave this to future work. Nevertheless, we conduct a simple experiment where we add gaussian noise to the weights of a model (a strawman model of what quantization does) and observe their flips. As shown in Table 1 in the rebuttal PDF, we find that even this simple noise model retains the accuracy of the baseline while significantly increasing flips. We believe a better modeling of the perturbation caused by the quantized schemes can shed more light on this phenomena.

2. Why flips occur

LLM compression methods approximate weights and activations, thus the outputs produced by compressed models may be viewed as noisy versions of the baseline model’s outputs. Flips occur for a question when the output probability of multiple answers change sufficiently that the highest probability answer from the compressed model is different from the baseline. As explained in Section 5 (Analyzing Flips) of our paper, the number of questions where the compressed model’s answers go from incorrect to correct is roughly equal to the number that go the other way, leading to accuracy of the compressed model remaining nearly the same. To summarize our findings, we empirically showed that this happens because lower confidence answers are more likely to be wrong and flip. The combination of incorrect answers flipping more along with slightly higher odds than random in landing on the correct answer results in incorrect → correct transitions roughly matching correct → incorrect transitions.

3. How flips relate to different compression techniques? Is there any discrepancy between flips in different compression methods?

Our observations are :-

• Quantization in general has fewer flips compared to other compression methods, e.g., WANDA (pruning), Layer Dropping and SliceGPT (sparsification) at comparable sparsity ratios.

• Fixing a sparsity ratio, the order of flips is WANDA < Layer Dropping < SliceGPT. However, in terms of performance gain, this is not an apples-to-apples comparison (for eg. WANDA at 0.3 sparsity won’t be nearly as fast as dropping the last 0.3 fraction of layers). This holds for the above point as well.

• For quantization, in general, order of flips is GPTQ W8A16 < BnB W8A8 < SQ W8A8 < GPTQ W4A16 < AWQ W4A16 < BnB W4A4 (as can be inferred from Figure 1 in the paper).

• Please also see our rebuttal for reviewer #2 above, for details on the consistency of flips (do the same samples flip) across different compression methods.

4. Wider variety of models and tasks

We extend our evaluations to include newer models (Llama 3 8B, Llama 3.1 8B and Gemma 7B) and more tasks (BCFL, Scroll - Quality, MATH, GSM8k).These are in addition to the tasks evaluated in the submitted paper (MMLU, ARC, PIQA, Hellaswag, LAMBADA, GSM8k, TriviaQA). Our general observations about %flips being much larger than difference in %accuracy (thereby indicating significant incorrect$\rightarrow$correct which get cancelled by the opposite transitions) still holds in all the new tasks. We collect all the new evaluations and present them here, for the convenience of the reviewer.

The following table shows the change in accuracy / percentage-flips of the quantized model vs the unquantized 16bit baseline:

BCFL

Scroll - Quality (0-shot)

MATH (4-shot)

GSM8k (8-shot)

Arc-e (0-shot)

Arc-c (0-shot)

MMLU (5-shot)

1. Evaluation of MCQ tasks using second setup

We re-evaluate the MMLU task using the second setup recommended by the reviewer (full answer text probability by summing log probabilities, as described in the HuggingFace blog, this was the EleutherAI Harness implementation as of January 2023). Note that for the ARC and Hellaswag evaluations in the paper (Tables 7-9), the setup 2 was already used, as is the standard in those tasks, and the observed flips are much larger than accuracy change.

The following table shows the change in accuracy / percentage-flips of the quantized model vs the unquantized 16bit baseline:

MMLU

Our observation in the paper that %flips $>>$ change in %accuracy still holds for this evaluation. Comparing these to Table 13 in appendix, we see that the flips percentage using this approach is slightly lower but still significant.

2. Evaluation on generative tasks GSM8k and Hendrycks' MATH

The evaluation on MATH and GSM8k are shown in the tables below. A limited evaluation on GSM8k was also presented in the paper (Table 17). Here we present the evaluation on newer models. Again, the tables show the change in accuracy / percentage-flips of the quantized model vs the unquantized 16bit baseline. We observe that flips is substantially higher in GSM8K even for BitsNBytes 8-bit quantization scheme (accuracy change less than 2%, flips 14-23%).

MATH (4-shot)

GSM8k (8-shot)

3. Error Bars or Statistical Significance Tests

The standard benchmark (ARC, MMLU, PIQA, Hellaswag, LAMBADA, GSM8k, TriviaQA, MATH) results on all models use greedy decoding, making these results fully reproducible without any requirement of error estimates. We will clarify this aspect in the paper.

MT-Bench does involve randomized sampling for generation along with GPT-4 evaluations, both of which are non-deterministic and hence may not be fully reproducible. While having error estimates for this particular benchmark would be useful, it is prohibitively expensive to rerun GPT-4 evaluations.

4. Evaluation on Expanded Model Family (Gemma/Llama 3/Llama 3.1)

We perform additional evaluation on the Llama 3, Llama 3.1 and Gemma models for the ARC and MMLU tasks. More results for qualitative tasks (BCFL, Scroll - Quality) can also be found in the rebuttal for the first reviewer.

Arc-e (0-shot)

Arc-c (0-shot)

MMLU (5-shot)

Observations

Our general observations that %flips $>>$ difference in %accuracy hold in the new evaluations as well.

1. Consistency of flips

The following table shows the % of questions whose answers are changed by 0-6 quantization schemes (for Llama2-70b MMLU task ~15K questions):

The main observations are 1) for most questions (84%), answers are unchanged by all the quantization schemes (and these, as expected, have high top margin). 2) Out of the remaining 16%, roughly half (8%) of the questions are commonly changed by two or more schemes indicating significant overlap between the schemes. Our conclusion is that flips are mostly consistent across schemes.

For a more fine-grained analysis, we also report pairwise fraction of changed examples that are common between two quantization schemes. We use overlap coefficient $Overlap(A,B) = \frac{|A \cap B|}{min(|A|, |B|)}$, to quantify this where A and B are the set of examples changed by the schemes. We again see good overlap, ranging between 40-50% common questions that are changed.

2. Effect of varying degrees of quantization

While the overall %flips do increase monotonically with higher degrees of compression (Tables 5-11 in the Appendix of the paper provide the %flips for multiple degrees of quantization, for BitsNBytes and GPTQ, across multiple datasets), there are cases that only the 16 and 4bit can answer but not 8bit, though not significant (See Table 2 in rebuttal PDF)

3. Top Margin Analysis

A. What does the metric indicate post quantization?

The top margin post quantization is well correlated with the baseline top margin as shown in the scatter plot in Figure 2 of the rebuttal pdf (Llama2-13B, MMLU task; Pearson correlation coefficient of 0.84). Further, high top margin answers are more likely to remain high post quantization. Also, average top margins pre and post quantization are similar (for eg. for Llama2-13b MMLU, the top margins for fp16, BnB W8A8, BnB W4A4 are 0.59, 0.59, 0.58 respectively)

B. Does quantization result in a more uniform probability distribution?

In general, quantization does not make the distribution smoother. However, when quantization extent is high (4bit), the distribution does become slightly more uniform (see Table 3 in PDF where entropy goes up slightly from 0.62/0.64 to 0.66)

C. Are the shifts in predicted likelihood and confidence larger or smaller for examples that were initially higher or lower in confidence?

As can be seen from Figure 1 in the rebuttal PDF, shifts in predicted likelihood are roughly the same irrespective of top margin (except in the cases of very high-top margin, where the shifts are small). From Figure 2, it is apparent that high confidence answers will remain so.

Thus, this fine-grained analysis adds evidence to the hypothesis that quantization introduces varying amount of output noise that flips points close to the decision boundary (low top margin) but does not substantially affect high confidence answers (high top margin).

Addressing Weaknesses

It is expected that model predictions with lower confidence ...

We agree with the reviewer’s point. However, the unexpected result is not that lower confidence answers changed more frequently but that the changes (flips) were balanced (incorrect → correct balancing out correct → incorrect), leading to nearly the same overall accuracy (Section 5: Analyzing Flips). Matching baseline accuracy on some tasks misleads researchers into believing that compressed LLMs are just as good as baseline. We empirically showed that this happens because lower confidence answers are more likely to be wrong and flip. The combination of incorrect answers flipping more often along with slightly higher odds than random in landing on the correct answer results in incorrect → correct transitions roughly matching correct → incorrect transitions.

For example it is not clear if the reason that low confidence...

We clarify that low confidence (low top margin) examples change because they are initially close to the decision boundary. The actual changes in probabilities do not depend on top margin (ie roughly all questions undergo the same amount noise), and top margins are well correlated before/after compression (Figures 1 and 2 in rebuttal PDF).

Change in model prediction has been studied ...

We emphasize that no training or fine-tuning happens during or after compression. This includes the Post-Training-Quantization (PTQ) methods, pruning and layer dropping experiments in our work. Hence there is no learning nor forgetting caused by compression. We will cite the referenced papers on continual learning training dynamics wherever applicable.

In the layer dropping MMLU evaluation (Figure 2a in the paper), we see significant incorrect->correct transitions but clearly, keeping weights of the first N-layers frozen and just dropping the last few layers cannot lead to the compressed model learning anything new. Compression results in approximate values of weight and/or activations, and thus, the outputs from the compressed model can be seen as noise-added versions of the baseline model. In Table 1 in the rebuttal PDF, we show how just by adding gaussian noise to the weights of a model we can get many incorrect->correct transitions both for generative GSM8k and MCQ ARC-challenge.

Additional evaluation

We perform additional evaluations as asked by the reviewer. In particular, we evaluate the following tasks:

• BFCL (Berkeley Function Calling Leaderboard) to test function calling
• SCROLLS (QuALITY split) to test long context understanding which would be useful in retrieval augmented setup
• MATH and GSM8k to test mathematical reasoning
• ARC-easy and ARC-challenge to test general reasoning

The tables below show the change in accuracy / flips-percentage of the quantized model vs the unquantized 16bit baseline:

BCFL

Scroll - Quality (0-shot)

MATH (4-shot)

GSM8k (8-shot)

Arc-e (0-shot)

Arc-c (0-shot)

Observations

In all the above tasks, which are generally harder compared to multiple-choice q&a, our general observations that flip percentage >> difference in accuracy hold. In fact, for many of these tasks, the flip percentages are much higher than observed in the experiments in the paper, even for the 8-bit quantization schemes like GPTQ 8bit and BnB 8bit.

We thank the reviewer for considering the rebuttal. We will include these newer evaluations in the paper. Also, yes, greedy sampling was used.

Qualitative Analysis of flips

1. MATH

This requires the model to just output the correct answer (and not the intermediate steps). We find it hard to clearly characterize the type of mistake made by the LLMs (especially because the range of subjects is very diverse). But , on many questions we find there to be simple mistakes (for eq. off by one errors, off by a factor of 2/10, etc).

Additionally, in the MATH dataset, each question is accompanied with a tag that indicates difficulty ($\in [1,2,3,4,5]$ ) where a higher number indicates that the question is more difficult. This lets us validate our intuition that the questions that are answered by both the baseline and quantized models are the easiest (average difficulty 2.49), and the ones answered by exactly one of them are moderate (this is where correct $\rightarrow$ incorrect transitions and vice versa happens, average difficulty 2.98) and the ones answered by neither are the hardest (average difficulty 3.56). The above numbers were obtained using Llama-3 8b and it's BnB W4A4 quantized version.

2. GSM8k

This task requires the LLM to generate the answer with step-by-step explanations. Here also, we find it hard to clearly characterize the type of mistakes made (though many of them can be vaguely classified as simple calculation errors) by baseline/quantized models or pin down the specific type of question in which such phenomenon happens.

In this task specifically, we observe that at times the model gets stuck in a loop, resulting in no output at all (rather than an incorrect one). For example:-

*Question*:  Travis had 61 apps on his tablet. He deleted 9 apps he didn't use anymore and downloaded 18 more. How many apps are on his tablet now?

*Llama-3-8b BnB W4A4 Answer* : Travis had 61 apps on his tablet. 

He deleted 9 apps he didn't use anymore and downloaded 18 more.

Travis had 61 apps on his tablet. He deleted 9 apps he didn't use anymore and downloaded 18 more.

Travis had 61 apps on his tablet. He deleted 9 apps he didn't use anymore and downloaded 18 more. 

Travis had 61 apps on his tablet. He deleted 9 apps he didn't use anymore and downloaded 18 more. 
(… and so on ...)

We measure % of such cases as %invalid. We observe that quantization makes %invalid worse in general. For example :- %invalid for Llama-3-8b, its BnB 8bit and BnB 4bit versions are 2.00%, 2.43% and 3.72% respectively.

3. BFCL

This task requires the LLM to call an API with the correct parameters to solve a given problem. The available API function definitions are given as a part of the prompt. Given the setup, we feel all problems can be considered to be of similar difficulty.

After analyzing the results, we categorize the errors into the following groups :-

1. incorrect API usage - this involves cases where the all the parameters are not initialized with their correct values or supplying more/less than the required number of parameters, etc. A very common type of error in this category is supplying parameters as a dictionary. For eg. we observed that the same function is called in two ways - sum(a=5,b=7) and sum({'a':5, 'b':7}), but only the first version is correct.

2. calling the wrong API - this involves calling an altogether different function than the one actually needed to solve the problem.

3. misunderstanding the question - this involves solving an orthogonal problem to the one required.

4. adding/omitting characters that make the API call non-parsable - self explanatory

In general, we observe that most of the errors are in the first group itself.

4. Scroll-Quality, ARC

These questions are of MCQ type and it is easier to characterize flips in such tasks (dealt with in the paper already). For such, tasks, top margin which measures the confidence of the baseline model in its answer, is a good predictor of whether the answer will flip or not.

Representative examples of Flips

Note that all the experiments use Llama3-8B and it's BnB W4A4 version

1. MATH

• TABLE 1 - correct -> incorrect cases (baseline correct, quantized wrong)

• TABLE 2 - incorrect -> correct cases (baseline wrong, quantized correct)

---

2. BFCL

• TABLE 1 - correct -> incorrect cases (baseline correct, quantized wrong)

• TABLE 2 - incorrect -> correct cases (baseline wrong, quantized correct)

• TABLE 2 - incorrect -> correct cases (baseline wrong, quantized correct) (..contd)

---

3. GSM8k

• TABLE 1 - correct -> incorrect cases (baseline correct, quantized wrong)

• TABLE 2 - incorrect -> correct cases (baseline wrong, quantized correct)