MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization

Thank you for your constructive comments! We'll discuss the reference by Alizadeh et. al in the revised paper.

The primary concerns were regarding:

Weaknesses:

• The use of $\ell_\infty$-norm: MagR aims to reduce the range of the pre-trained weights (not the quantization noise), and $\ell_\infty$-norm is naturally the best to serve this purpose. So the nature of MagR is different than $\ell_1$-gradient regularization proposed by Alizadeh et. al. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error. In fact, we can show that the $\ell_2$ quantization error is $O(\delta)$. Our additional experiments also demonstrate that the layer-wise quantization errors indeed are reduced by applying MagR; see the figure in the attached pdf file. Another notable advantage of $\ell_\infty$-norm over the general $\ell_p$-norm is its closed form of proximal operator, which is the core of the algorithm design for MagR.

• Activation quantization: By its nature, MagR is designed to improve the performance of weight quantization without introducing inference overhead. So we evaluated our method for only weight quantization. We intend to investigate activation quantization in our future work.

• Ablation study on $\alpha$: $\alpha$ is the parameter balancing the tradeoff between the output discrepancy and the max magnitude of the weights. We show the ablation study on $\alpha$ for channel-wise quantization on LLaMA2-7B as below. Note that the choice of $\alpha$ does not depend on the bit-width and $\alpha=0.001$ is the best choice for channel-wise quantization.

  Model	$\alpha$	W/A	Wiki (PPL)	C4 (PPL)
  LlaMa2-7B	0.005	4/16	5.84	7.55
  	0.001	4/16	5.70	7.28
  	0.0005	4/16	5.72	7.29
  	0.0001	4/16	5.78	7.35
  	0.00001	4/16	5.81	7.40
  	0.005	3/16	6.64	8.74
  	0.001	3/16	6.41	8.23
  	0.0005	3/16	6.49	8.38
  	0.0001	3/16	6.83	8.79
  	0.00001	3/16	7.08	9.19

Questions:

• Fraction ranks in Table 1: The statistics in Table 1 not only show that all feature matrices are nearly rank-deficient, but also gives us an idea of how low-rank they are and the overall distribution of their fraction ranks across the architectures. Intuitively, MagR works better for low-rank feature matrix because its kernel space would be larger (of a higher dimension), and MagR potentially produces weights with smaller $\ell_\infty$-norm.

• The choice of $\beta$: Thanks for your suggestion. Since our main focus is on MagR preprocessing to reduce the quantization error, we prioritized simplicity and efficiency of the quantization algorithm by fixing the $\beta$, which has proven to be effective. We agree that optimizing the quantization error with respect to $\beta$ could potentially further improve the performance, and we intend to investigate this in the revision.

• Preprocessing runtime: Thanks for your suggestion, we will rename it.

• Inference time: MagR essentially replaces the original pre-trained weights with a new set of weights that have a smaller magnitudes prior to actual quantization, without sacrificing the original accuracy or altering the architecture. Consequently, our MagR+OPTQ achieves exactly the same inference efficiency as OPTQ. This is immediately supported by the widely-used inference kernel from the AutoGPTQ library. In contrast, at inference time, QuIP requires performing a random linear transformation on the activations before multiplying the quantized weights. Since the code for QuIP's inference is not released, we cannot compare them directly. But according to QuIP's own report, QuIP's inference speed is at least 1.5 times slower than OPTQ for the OPT-66B model (81 ms vs 53 ms).

Thank you for the review! We'll add ablation study and fix the typos and incorrect format as suggested, and include the details for deriving Alg. 1.

Other concerns were regarding:

Weaknesses:

• Theoretical analysis: Our new results establish that:

  • The layer-wise $\ell_2$ quantization error is $O(\delta)$, where $\delta$ is the quantization step of the uniform quantizer. This motivates our preprocessing method based on $\ell_\infty$-norm of the weights since minimizing $\ell_\infty$-norm amounts to minimizing $\delta$.
  • We can also prove the convergence rate of the proposed MagR algorithm for the $\ell_\infty$-regularization. Specifically, $f(w^k) - f(w^*) = O(1/k) \to 0$ as $k\to \infty$, where $f(w) = \frac{1}{2}|| Xw - X\hat{w}||^2 + \alpha ||w||_{\infty}$ is the objective function, $w^k$ is the $k$-th iterate generated by MagR, and $w^*$ is the true minimizer.
  • The layer-wise output $\ell_2$-error after MagR preprocessing obeys $|| Xw^* - X\hat{w}|| = O(\sqrt{\alpha})$, where $\alpha>0$ is the penalty parameter.

• Why MagR works:

  • MagR effectively reduces the range of the pre-trained weights by employing $\ell_\infty$-minimization, as illustrated by Figure 1. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error. In fact, given the activations $X\in\mathbb{R}^{m\times n}$, quantizer $\mathcal{Q}$ with quantization step $\delta>0$, pre-trained weights $\hat{w}\in\mathbb{R}^n$, the following analysis shows that the $\ell_2$ quantization error is linear in $\delta$:
    $|| Xw_q - X\hat{w}|| \leq ||X \mathcal{Q}(\hat{w}) - X\hat{w}|| \leq \sigma_{\max}(X) ||\mathcal{Q}(\hat{w}) - \hat{w}|| \leq \frac{\sigma_{\max}(X) \sqrt{n}}{2}\delta.$ Our additional experiments also demonstrate that the layer-wise quantization errors indeed are reduced by applying MagR on randomly sampled layers; see the figure in the attached pdf file.

  • We also note that MagR can preserve all the layers' outputs (before performing quantization), ensuring that the pre-trained model's performance remains unaffected. This is crucial in the PTQ setting, as the goal is to search for the quantized model only within the local neighborhood of the pre-trained model. The following table shows that MagR preprocessing indeed approximately maintains the perplexity (ppl) of the pre-trained model with minor degradation. Since we are minimizing the regularization $\frac{1}{2}|| Xw - X\hat{w}||^2 + \alpha ||w||_{\infty}$ for preprocessing, for the minimizer $w^*$ it holds that $|| Xw^* - X\hat{w}|| = O(\sqrt{\alpha})\to 0$, as $\alpha \to 0$. When choosing a small $\alpha$ (equivalent to imposing large penalty on the fidelity term), $|| Xw^* - X\hat{w}||$ will be small but not 0. This explains why the model performance degrades slightly after MagR (before quantization).

    Model	Method	Wikitext2 (PPL)	C4 (PPL)
    LlaMa2-7B	Original	5.47	6.97
    	After MagR	5.52	7.04
    LlaMa2-13B	Original	4.88	6.46
    	After MagR	4.92	6.52
    LlaMa2-70B	Original	3.31	5.52
    	After MagR	3.35	5.56

Questions:

• Ablation studies: $\alpha$ is the parameter balancing the tradeoff between the output discrepancy and the max magnitude of the weights. We show the ablation study on $\alpha$ for channel-wise quantization on LLaMA2-7B as below. Note that the choice of $\alpha$ does not depend on the bit-width, and $\alpha=0.001$ works well for all channel-wise quantization.

  Model	$\alpha$	W/A	Wiki (PPL)	C4 (PPL)
  LlaMa2-7B	0.005	4/16	5.84	7.55
  	0.001	4/16	5.70	7.28
  	0.0005	4/16	5.72	7.29
  	0.0001	4/16	5.78	7.35
  	0.00001	4/16	5.81	7.40
  	0.005	3/16	6.64	8.74
  	0.001	3/16	6.41	8.23
  	0.0005	3/16	6.49	8.38
  	0.0001	3/16	6.83	8.79
  	0.00001	3/16	7.08	9.19

  Model	$\beta$	W/A	Wiki (PPL)	C4 (PPL)
  LlaMa2-7B	1	3/16	6.43	8.33
  	0.9	3/16	6.41	8.23
  	0.85	3/16	6.48	8.39
  	0.8	3/16	7.12	9.46
  	1	2/16	16.99	24.12
  	0.9	2/16	20.88	31.78
  	0.85	2/16	16.76	24.45
  	0.8	2/16	16.73	23.73

• Why MagR+OPTQ not achieve better results in Table 3: The reasoning tasks in Table 3 require diverse knowledge and skills. Quantization may impede the model's ability to excel across all tasks. Therefore, a quantized model with better perplexity does not necessarily imply higher multi-task accuracy, especially when the perplexity values are close.

• Impact of outliers on PTQ: Outliers have much larger magnitude than other weights, which result in a large quantization step and a large quantization error. MagR addresses this issue by minimizing $\ell_\infty$-norm to reduce the quantization step.

Thank you for the review! The primary concerns were regarding:

Questions:

• Rank deficiency of feature matrix: Section 4.1, specifically Table 1, demonstrates that the feature matrices across all the layers of the LLaMA family models have very small singular values, less than 0.01 times the maximum singular value. This indicates that the feature matrices are approximately rank-deficient, which ensures that we can approximately preserve the original model's performance after applying MagR. It is important to note that exact rank-deficiency is not a requirement for MagR to work. Notably, this approximate rank-deficiency is not unique to the LLaMA models; it also applies to the other models like OPT, as discussed in the QuIP paper. The threshold value of 0.01 is not a calculated parameter nor an algorithmic hyperparameter. Changing this threshold will not alter the feature matrices or affect the quantization performance of the proposed method.

• Efficiency of proximal gradient descent: The preprocessing times on a single A100 GPU are 15 min for Llama2-7B, 30 min for 13B, and 3.5 hr for 70B. We believe that MagR can be readily applied to larger LLMs with 100+B parameters.

• Model performance after MagR: The table at the bottom shows that the perplexity values degrade slightly after MagR. But since the range of the weights are reduced (Figure 1), we are able to use a smaller quantization step for the quantizer, which gives a smaller quantization error (see the table below) and improves the quantization performance.

Limitations:

• Why MagR works:

  • MagR effectively reduces the range of the pre-trained weights by employing $\ell_\infty$-minimization, as illustrated by Figure 1. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error. In fact, given the activations $X\in\mathbb{R}^{m\times n}$, quantizer $\mathcal{Q}$ with quantization step $\delta>0$, pre-trained weights $\hat{w}\in\mathbb{R}^n$, the following analysis shows that the $\ell_2$ quantization error is linear in $\delta$:
    $|| Xw_q - X\hat{w}|| \leq ||X \mathcal{Q}(\hat{w}) - X\hat{w}|| \leq \sigma_{\max}(X) ||\mathcal{Q}(\hat{w}) - \hat{w}|| \leq \frac{\sigma_{\max}(X) \sqrt{n}}{2}\delta.$ Our additional experiments also demonstrate that the layer-wise quantization errors indeed are reduced by applying MagR on randomly sampled layers; see the figure in the attached pdf file or the table below.

    Model (4-bit)	Layer	RMSE With MagR	RMSE Without MagR
    LlaMa2-7B	14	0.1327	0.1575
    	65	0.1421	0.1616
    	121	0.1622	0.1879
    	184	0.2025	0.2319
    	217	0.2198	0.2542
    LlaMa2-13B	14	0.1289	0.1518
    	88	0.1647	0.1892
    	145	0.1836	0.2127
    	215	0.2098	0.2435
    	271	0.2245	0.2618

  • We also note that MagR can preserve all the layers' outputs (before performing quantization), ensuring that the pre-trained model's performance remains unaffected. This is crucial in the PTQ setting, as the goal is to search for the quantized model only within the local neighborhood of the pre-trained model. The following table shows that MagR preprocessing indeed approximately maintains the perplexity (ppl) of the pre-trained model with minor degradation. Since we are minimizing the regularization $\frac{1}{2}|| Xw - X\hat{w}||^2 + \alpha ||w||_{\infty}$ for preprocessing, for the minimizer $w^*$ it holds that $|| Xw^* - X\hat{w}|| = O(\sqrt{\alpha})\to 0$, as $\alpha \to 0$. When choosing a small $\alpha$ (equivalent to imposing large penalty on the fidelity term), $|| Xw^* - X\hat{w}||$ will be small but not 0. This explains why the model performance degrades slightly after MagR (before quantization).

    Model	Method	Wikitext2 (PPL)	C4 (PPL)
    LlaMa2-7B	Original	5.47	6.97
    	After MagR	5.52	7.04
    LlaMa2-13B	Original	4.88	6.46
    	After MagR	4.92	6.52
    LlaMa2-70B	Original	3.31	5.52
    	After MagR	3.35	5.56

Thank you for the review and for pointing out relevant references. We'll add the references [1,2] and discussions.

The primary concerns were regarding:

Weaknesses:

• MagR vs QuIP, DecoupleQ:

  • It is possible to run additional coordinate descent iterations on top of OPTQ to further pursue the ppl, as shown in the updated results for the 2-bit 70B model. We also include the ppl results on 7B/13B models for QuIP. Our method outperforms QuIP in most cases. Regardless, QuIP trades off inference speed for accuracy, whereas MagR does not.

    Model	Method	Wbits	Wiki (PPL)	C4 (PPL)
    LlaMa2-7B	QuIP	4	5.94	8.01
    	MagR+OPTQ	4	5.70	7.28
    	QuIP	3	6.50	8.74
    	MagR+OPTQ	3	6.41	8.23
    	QuIP	2	27.13	31.33
    	MagR+OPTQ	2	16.73	23.73
    LlaMa2-13B	QuIP	4	5.01	6.88
    	MagR+OPTQ	4	4.97	6.63
    	QuIP	3	5.34	7.34
    	MagR+OPTQ	3	5.41	7.19
    	QuIP	2	10.09	13.13
    	MagR+OPTQ	2	11.14	14.45
    LlaMa2-70B	QuIP	2	6.33	8.94
    	MagR+OPTQ	2	5.95	8.53

  • DecoupleQ proposes to use block-wise optimization (similar to Omniquant) on top of GPTQ to refine the solution. Bottomline is that decoupleQ is still a quantization method (like GPTQ/Omniquant), MagR can be used as preprocessing for DecoupleQ to further improve its performance (MagR + DecoupleQ).

• Quantization runtime comparison: We reported the runtime of MagR+OPTQ and MagR+RTN in Table 4. We meant that the entire runtime of MagR+OPTQ is roughly half of Omniquant. The following table shows the quantization runtimes of QuIP and OmniQuant. All the methods were tested on a single NVIDIA A100 GPU.

  Method	LlaMa2-7B	LlaMa2-13B
  QuIP	54 min	68 min
  Omniquant	73 min	2 hr
  OPTQ	22 min	40 min
  MagR+OPTQ	35 min	70 min

• Prior works on weight regularization: Thanks for pointing out the reference. The goal of R2 Loss is similar to MagR,which regularizes a smaller range, but the reference uses a traditional end-to-end training. The training is from scratch and the regularization term is added to the original cross-entropy loss. The targets are CNN models. This method cannot be used for LLMs since the pre-training phase is too expensive to repeat. Moreover, their minimization is carried out by the conventional (sub)gradient method, whereas we investigate efficient algorithm which takes advantage of the proximal operator of $\ell_\infty$-norm.

• Calibration set: The calibration dataset consists of 128 randomly selected 2048-token segments (context length) from WikiText2, which follows the routine of prior PTQ works.

Questions:

• Inference efficiency: MagR essentially replaces the original pre-trained weights with a new set of weights that have a smaller magnitudes prior to actual quantization, without sacrificing the original accuracy or altering the architecture. Consequently, our MagR+OPTQ achieves exactly the same inference efficiency as OPTQ. This is immediately supported by the widely-used inference kernel from the AutoGPTQ library. In contrast, at inference time, QuIP requires performing a random linear transformation on the activations before multiplying the quantized weights. Since the code for QuIP's inference is not released, we cannot compare them directly. But according to QuIP's own report, QuIP's inference speed is at least 1.5 times slower than OPTQ for the OPT-66B model (81 ms vs 53 ms), in addition to the power consumption overhead.

• Is QuIP incompatible with group-wise quantization?: We think that QuIP should be compatible with group-wise quantization. QuIP first applies random linear transformations to preprocess the weights and activations and to smooth out outliers, then uses OPTQ to perform the actual quantization. Like OPTQ, it should be compatible with group-wise quantization. But they never reported such results. The drawback of QuIP is that it requires random linear transformations on the feature matrix (so-called incoherence processing) not only in the preprocessing stage, but also in the inference phase.

• Results in Table 3: All the results in Table 3 are for channel-wise quantization.

• Omniquant/QuIP with MagR: We believe that MagR could enhance the performance of Omniquant or QuIP, but the improvement may not be as significant as with OPTQ. For instance, QuIP's incoherence processing also helps reduce weight magnitude, as reported in the paper. Omniquant uses block-wise minimization and learns the quantization step (weight clipping parameter) via SGD. In contrast, OPTQ is a fast, greedy, gradient-free algorithm. MagR+OPTQ achieves both simplicity and effectiveness, without introducing any inference overhead.

Thank you for your careful and detailed response.

While I find the concept of MagR in regularizing weight values before quantization interesting, the paper seems to lack essential information needed to verify the significance of the proposed work:

1. As you correctly noted, R2 Loss [2] was evaluated on CNNs using a full training approach. Since the loss term designed to regularize the weight distribution can be applied across various model architectures, it is crucial to thoroughly clarify the differences between R2 Loss and MagR, particularly in terms of loss term design or training efficiency. Although the difference between R2 Loss and MagR is briefly discussed in the rebuttal, the explanation provided is not detailed enough to fully clarify the distinction.

2. The proposed MagR+OPTQ does not consistently achieve the best ppl/accuracy results compared to previous works.

3. If the key novelty of MagR lies in pre-processing LLMs before quantization, it should be compatible with various quantization methods to enhance ppl/accuracy. However, the paper only discusses MagR+OPTQ without exploring the results of MagR combined with other quantization methods. Consequently, it is unclear whether MagR should be regarded as a general pre-processing solution or if ‘MagR+OPTQ’ is intended as a new quantization scheme. If MagR is indeed a broadly applicable pre-processing method, it would be highly valuable. If not, I have reservations about the significance of ‘MagR+OPTQ,’ given the limited ppl/accuracy improvement as discussed earlier.

Therefore, I still perceive MagR as a limited contribution and will maintain my stance.

1. ... it is crucial to thoroughly clarify the differences between R2 Loss and MagR, particularly in terms of loss term design or training efficiency ...

As we previously noted, R2 regularization applies $\ell_\infty$ penalty to the traditional network loss during end-to-end model pre-training, optimized using standard SGD, which is practically infeasible for LLMs. In contrast, MagR, as a concurrent approach, operates directly on the pre-trained model in a layer-by-layer manner, utilizing linear least squares loss to preserve each layer's output. MagR employs proximal gradient descent algorithm specifically tailored for this objective, enabling efficient processing of LLMs. This mathematically elegant algorithm represents a main contribution of our work.

2. ... MagR+OPTQ does not consistently achieve the best ppl/accuracy results ...

While we don’t claim that MagR+OPTQ always achieves the highest accuracy, it does perform at or near the top in most cases. Moreover, its accuracy can be further enhanced by applying techniques like additional coordinate descent iterations or learnable weight clipping as suggested by Reviewer 5GSj. Accuracy is just one of many critical performance metrics. More importantly, we introduce a technique that incurs no additional overhead at inference time while achieving both training efficiency and accuracy comparable to the state-of-the-art, which typically sacrifices inference speed. Inference speed is crucial for real-world applications, particularly in resource-limited settings such as edge computing.

3. ... the paper only discusses MagR+OPTQ without exploring the results of MagR combined with other quantization methods ...

We would like to remind the reviewer that we also reported the substantial performance gain of MagR over the nearest round method (MagR+RTN vs RTN) in Table 2.

Here we also demonstrate that MagR can enhance the performance of QuIP:

In summary, MagR preserves the behavior of the pre-trained model, as demonstrated by the new data, while allowing for small quantization steps in the subsequent PTQ process, leading to reduced quantization error. So theoretically, MagR can be combined with other PTQ methods.

Dear authors,

Thank you for your thorough response to our comments and questions. I have one more question.

In the attached figure, you compare MagR's quantization error with OPTQ. Do you have time to add the quantization error with the transformed weights of QuIP? This would be quite useful in understanding whether your optimization is theoretically superior to the random linear transformation of QuIP (beyond QuIP's additional compute overhead).

Also, is the quantization error calculated before or clipping, namely, is $\beta$<1 for these results?

Thanks

Thank you for your questions.

In response to the second question, we evaluated the quantization errors for INT4 quantization, and we did not apply weight clipping at 4-bit (i.e., $\beta=1$). This means that the reduction in quantization error is entirely due to the MagR preprocessing.

Regarding the quantization error with QuIP, it targets a different quantization error in a distinct parameter space compared to our method or OPTQ. Specifically, we solve $\min_{W_q\in\mathbb{Q}} ||X(W_q - W)||^2$, where the focus is directly on minimizing the error in the original weight space. In contrast, QuIP first applies orthogonal transformations $U$ and $V$ to the weights and activations before quantization: $\min_{W_q} ||XV V^{\top} (W_q - W)U||^2 := \min_{\tilde{W_q}\in\mathbb{Q}} ||\tilde{X} (\tilde{W_q} - \tilde{W})||^2$. Here, $\tilde{X} = XV$, $\tilde{W_q} = V^{\top} W_q U$, and $\tilde{W} = V^{\top} W U$ represent the transformed activations and weights. QuIP only quantizes the transformed weights $\tilde{W_q}$, and their corresponding $W_q$ is actually float. In contrast, we directly quantize $W_q$ as is done in standard OPTQ. As a result, the quantization errors between our method and QuIP are not directly comparable.

We hope we have addressed the reviewer’s questions.

We hope our rebuttal have addressed the reviewers' concerns, and we would greatly appreciate it if you could reconsider your rating.