The paper does not provide sufficient proof of the correctness of its algorithm. There is no error analysis for the approximation methods presented. While the authors detail their approach for solving Equations 3 and 4, they do not demonstrate that these solutions are equivalent to Equation 2, which is the main problem of interest.

We thank the reviewer for poiting out this omission. Please find in Appendix C all the proofs of the results presented in our manuscript.

The theoretical analysis provided by the authors is overly simplified. The assumptions of full rank in Propositions 3.3 and 3.4 are questionable. Empirical studies have suggested that activations in large transformer models often exhibit a structure that can be approximated as low rank, making these assumptions potentially unsuitable.

Thank you for your comment. We would like to clarify that although we have presented the 'invertible' case for simplicity, our approach is readily extendable to scenarios where a regularization term is applied to the covariance matrices. This extension is precisely the case we consider in practice, as detailed in the paragraph 'Numerical Stability' (line 303).

The paper does not include an analysis of the time complexity associated with adding low-rank weights. This omission leaves a gap in understanding the computational implications of the proposed method.

We thank the reviewer for this important remark. Currently we are only emulating the implementation of our approach and do not leverage a specific cuda kernel adapted to our computational scheme in order to perform the low-rank correction. However, for the sake of evaluation, we have added an experiment evaluating the speedup obtained with a naive cuda kernel implementing our approach and compare it against the full precision version. This experiment is presented at the begining of the rebuttal.

The contribution and novelty of the paper are somewhat limited. While the authors improve the accuracy of post-training quantization through low-rank correction, additional analyses on aspects such as time complexity, convergence, and error bounds would enhance the paper’s competitiveness and impact.

We concur that these additional theoretical considerations may indeed bolster the approach. Although our proposed algorithm is underpinned by a robust theoretical framework, we contend that the primary value of our work lies in its practical relevance to the community, rather than its theoretical contributions.

Thank you for poiting out the typos. We have corrected them.

The paper contains several propositions but they lack proofs.

Thank you for poiting out this omission. We have added all the proofs in the Appendix C.

The approach of adding a low rank matrix is similar to approaches in these papers that the authors could cite: "LoRA: Low-Rank Adaptation of Large Language Models" and "QLoRA: Efficient Finetuning of Quantized LLMs".

These papers aims to tackle the finetuning stage of LLMs rather than their post-training quantizations, but we agree that they rely on additional low-rank weights. We have followed the reviewer suggestion and cited them in the Introduction.

Furthermore, the paper could consider generative tasks for benchmarks such as gsm8k. Typically, generative tasks are harder for quantized model than multiple choice tasks.

We agree that we did not evaluate on generative tasks. However, we are relying on the lm-eval datasets that are currently the standard evaluation benchmarks used to measure the performances of quantized models. Please refer to this very recent paper (Spotlight NeurIPS 2024) as an example of such an evaluation "QTIP: Quantization with Trellises and Incoherence Processing". Additionally, we have reported the perplexities (PPL) obtained by the various approaches in all our tables for a comprehensive evaluation of their performances.

Dear Reviewer,

Thank you for your comments and review. We believe that we have addressed all your concerns and have implemented the necessary changes in the current version of the manuscript. Additionally, we noticed that you rated our manuscript with a score of 1 for Presentation, which appears to be an outlier compared to the comments of other reviewers. We understand that this decision was based on your initial feedback, but we hope that the issues you raised have now been resolved. If you agree with our assessment, we kindly ask you to reconsider your overall score.

Thank you again for your time and your review.

Dear Reviewer,

We sincerely appreciate your thoughtful review and are delighted to hear that we successfully addressed all your concerns. We are also grateful for the increase of 2 points in the Presentation score.

While the detailed scores have seen significant improvement and all concerns have been resolved, we noticed that the overall score remains unchanged. We kindly request a reconsideration of the overall rating to better reflect the reevaluation of the manuscript.

Thank you once again for your time and effort in reviewing our work.

The paper does not analyze the computational cost associated with the added low-rank correction matrix. While the method effectively reduces quantization error, the impact on inference time and memory usage is not thoroughly explored.

We thank the reviewer for this important remark. Currently we are only emulating the implementation of our approach and do not leverage a specific cuda kernel adapted to our computational scheme in order to perform the low-rank correction. However, for the sake of evaluation, we have added an experiment evaluating the speedup obtained with a naive cuda kernel implementing our approach and compare it against the full precision version. Additionally, we have added the sizes of the different models when the rank is set to 10% of the original sizes. Both experiments are presented at the beginning of the rebuttal in the paragraphs titled 'Experiment: Latency of LRC' and 'Experiment: Memory Footprint of LRC', respectively.

The authors leave the ideal implementation of the low-rank computation for future work [...]. Providing guidance or preliminary implementation details could have made the paper more impactful.

Thank you for this insightful comment. We followed the reviewer's suggestion and included in Appendix B.2. the discussion from our new experiment on latency, as presented at the beginning of the rebuttal. Additionally, we will clarify that the proposed kernel is a naive implementation of LRC that performs operations sequentially. We hypothesize that these operations can be executed in parallel.

Although the paper identifies activation quantization as the primary source of error, it does not propose novel activation quantization schemes [...]. Addressing this limitation within the paper could have further strengthened the contribution.

Thank you for your comment. We would like to highlight that LRC is a novel quantization scheme primarily designed to mitigate quantization errors in activations. Notably, when activations remain unquantized, the introduction of additional low-rank weight matrices exerts minimal impact on the performance (see Table 3), and low-rank correction terms are not needed in this setting. Therefore, although LRC is structured as additional weights, its principal objective is to rectify the errors associated with activation quantization. We acknowledge that LRC is currently presented using RTN for activation quantization (and GPTQ for weight quantization), however the framework proposed allows other quantization techniques to be applied instead. We believe that improving RTN is definitively an opportunity for further enhancement, but we consider it beyond the scope of this work.

Is there a trade-off between the compression ratio and the computational overhead introduced by LRC?

In an idealized scenario, it would be possible to design a CUDA kernel that computes both the full and low-rank terms in parallel, thereby eliminating any computational tradeoff for LRC. However, such a kernel is not currently available, resulting in a tradeoff between accuracy and computational overhead. While we did not investigate this specific tradeoff, Figure 2 (and Figure 4 in Appendix B.3) illustrates a similar tradeoff by measuring accuracy against the chosen rank.

Can the LRC method be extended to other model compression techniques, such as pruning or knowledge distillation? How would the low-rank correction approach interact with these techniques?

That is an excellent point, thank you. For instance, in the context of pruning, if one aims to correct the pruned model by incorporating additional low-rank weights, it would be feasible to substitute Eq. (5) with a layer-wise pruning objective while employing the same computational scheme as the one proposed in this work (LRC). Although this represents an intriguing application of our work, we consider it beyond the scope of this study.

How sensitive is the performance of LRC to the choice of calibration dataset used for computing the activation statistics? Would using a more diverse or domain-specific calibration dataset lead to better results?

Thank you for this suggestion. We have performed an additional experiment showing the effect of the choice of the calibration dataset on the performances of LRC. Please refer to the experiment presented at the beginning of the rebuttal in the paragraph titled "Experiment: Effect of the Calibration Dataset" to see the results obtained. We have also added these results in Appendix B.1.

The authors suggest that improved activation quantization schemes could further enhance the performance of LRC. What specific properties should these improved schemes have, and how might they be developed?

Although LRC does not impose specific requirements on the quantization scheme for activations, we consider this step to be a fundamental limitation of current quantization techniques. Currently LRC can mitigate this gap by incorporating low-rank terms, but this may not suffice to fully recover the original model performance if a sufficiently small rank is chosen. Therefore, to enhance the efficiency of LRC, it might be essential to improve the quantization of activations.

Dear Reviewer,

We sincerely appreciate your valuable comments and questions, which have greatly contributed to improving our manuscript. We believe that we have thoroughly addressed all your concerns and appreciate the opportunity to clarify our work. As the rebuttal period draws to a close, we would like to confirm whether our responses have adequately addressed your concerns.

Thank you again for your time and your review.

It is not clear how the rank of adaptation would influence the efficiency. This would become my main concern for this paper. I strongly recommend the authors to add an experiment to evaluate its enhancement of memory usage and speed.

We thank the reviewer for this important remark. Currently we are only emulating the implementation of our approach and do not leverage a specific cuda kernel adapted to our computational scheme in order to perform the low-rank correction. However, for the sake of evaluation, we have added an experiment evaluating the speedup obtained with a naive cuda kernel implementing our approach and compare it against the full precision version. Additionally, we have added the sizes of the different models when the rank is set to 10% of the original sizes. Both experiments are presented at the beginning of the rebuttal in the paragraphs titled 'Experiment: Latency of LRC' and 'Experiment: Memory Footprint of LRC', respectively.

The presentation of this paper is good, but not excellent enough. The authors should add an introduction of GPTQ method and Cholesky in their appendix (since they are parts of the main algorithm) for presenting to the broader audience.

Thank you for acknowledging the quality of our presentation. We have followed the suggestion of the reviewer and added additional background on Cholesky and GPTQ in Appendix A. For the convenience of the reader, we also report the added paragraphs below.

GPTQ Algorithm. The GPTQ algorithm, introduced by Frantar et al. (2022), is a post-training quantization technique designed to efficiently reduce the precision of weights in large language models (LLMs) while maintaining their performance. To achieve this, the authors propose to approximate a solution of the layer-wise quadratic approximation problem defined as:

$\min_{\widehat{**W**}\in\mathcal{C}(b)\cap \mathbb{R}^{d^{\text{out}}\times d^{\text{in}}}}\mathcal{L}_{\text{q}}(\widehat{**W**}):=\Vert **W****X** - \widehat{**W**} **X**\Vert_2^2\;$

where $**W**$ is the original weight matrix, and $\mathcal{C}(b)$ is the constraint set of matrices admitting a certain bit per weight $b>0$ precision. The main difficulty of solving exactly this optimization problem resides in the constraint set $\mathcal{C}(b)$, making the problem non-convex. To approximate a solution, Frantar et al. (2022) propose to improve the computational scheme of the greedy approach originally proposed by LeCun et al. (1989) for pruning, and then adapted for quantization in (Frantar & Alistarh, 2022), by removing the ordering in the greedy quantization process, and applying the algorithm in parallel over multiple columns.

Cholesky Factorization. Cholesky factorization is a numerical method used to decompose a symmetric positive-definite matrix (PD) into the product of a lower triangular matrix with positive diagonal coefficients and its transpose. This technique is particularly useful in solving systems of linear equations, performing matrix inversion, and computing the determinant of a matrix. More formally given $\Sigma$ a symmetric PD matrix, there exists a unique lower triangular matrix $**L**$ such that

$\Sigma = **L****L**^\top .$

To compute the Cholesky factor $**L**$, one can rely on the Cholesky Algorithm which is a modified version of the Gaussian elimination and requires $\mathcal{O}(n^3)$ FLOPs where $n$ is the size of $\Sigma$.

The choice of dataset in $X$ should be carefully considered, but I don’t see any analysis about how $X$ will affect the performance. I recommend the authors to add a discussion about this and their assumption for rigorousness.

Thank you for this suggestion. We have performed an additional experiment showing the effect of the choice of the calibration dataset on the performances of LRC. Please refer to the experiment presented at the beginning of the rebuttal in the paragraph titled "Experiment: Effect of the Calibration Dataset" to see the results obtained. We have also added these results in Appendix B.1.

Please answer the issues and questions in the Weakness and point out my potential misunderstandings. I am happy to discuss and enhance my rate.

Thank you. We hope we have answered your questions, and we would be delighted to continue the discussion if any points require further explanation.

The experiments are performed on models which do not overlap with the QuaRot paper (their primary baseline) and hence it is difficult to compare directly.

Concerning this point, we thought it would be more interesting to compare our method with Quarot's performance on more recent LLMs. We did replicate the QuaRot method (using code from those authors), but we are delighted to compare our method with Quarot on the LLMs used in the original paper. We have added to the tables 1 and 2 of the manuscript the performances obtained by the different quantization methods on Llama 2 7B and 13B. For the convenience of the reader, we state the results again here:

Llama 2 (7B) without groupsizing

Llama 2 (7B) with groupsizing (128)

Llama 2 (13B) without groupsizing

Llama 2 (13B) with groupsizing (128)

Moreover, their method uses additional bits (aka low rank correction factors) and hence comparison to QuaRot is not a strictly fair comparison, a small increase in model size though it may be.

We agree that LRC incurs additional memory footprint. More precisely, in our experiments, we settled on adding low-rank weight matrices with a rank corresponding to 10% of the original size, therefore we are effectively at 5.6 bits (4 + 0.1 * 16) per weight. We argue that this additional memory usage is worth spending for improved downstream-task performance. Note also that the SVD approach (LQER) requires the exact same bit precision. We will clarify this point in the manuscript.

The authors mention a number of contemporary work (Quip, Quip#, LQER etc) but provide an empirical comparison against a very limited baseline (QuaRot).

We focus mainly on comparing with QuaRot as it was the SoTA approach (at the time of the submission) to quantize LLMs in the weights-and-activations setting. Note also that Quip and Quip# do not handle the case where both weights and activations are quantized. Finally, we do compare with an improved version of LQER (where QuaRot is additionally applied) which we called SVD in tables 1,2 and 3.

Limited Contribution: The paper stitches together many well known building blocks in the PTQ literature to build a sane, effective technique. In my opinion, it is a sound engineering feat, but still has high overlap with the previous work on the topic by Zhang et al (2024) and Ou et al (2024).

We agree that our work is a closely related to the prior works cited, as they all leverage low-rank weights to improve the quantization process. Note however that Ou et al. (2024) is improving on Zhang et al (2024) by proposing to replace the SVD factorization by a projection which takes into account the statistical properties of the output activations, however they completely discard the quantization of activations, which is in our opinion, the main motiviation of why one should add low-rank weights in the first place. In this work, not only we improve the proposed factorization of Ou et al.(2024) (and as a direct consequence the one proposed in Zhang et al (2024)) but most importantly we incorporate the effect of quantizing activations into the quantization process. The latter being the main reason why one would require addditional low-rank weight matrices to correct quantization errors.

This is also a well known technique and has been applied in other works such as [1]

Thank you for highlighting this reference. We have included the following discussion of this work in our manuscript:

In [1], the authors improve the methodology of Ou et al.(2024) by considering a joint formulation of the quantization problem to optimize for both the quantized weights and the low-rank terms. However, the authors only focus on the quantization of the weights, leaving aside the quantization of activations. In this work, we also consider a joint formulation, however our focus is on improving the quantization of activations. We improve on prior research by incorporating both the empirical distribution of activations and the errors induced by activation quantization into our analysis to optimize the low-rank weight matrices.

We would like also to mention that for weight only quantization, several other approaches have already closed the gap with the full precision model (see Table 3), and therefore the introduction of low-rank correction terms might not be needed. Our work shows that low-rank correction terms significantly reduce the accuracy gap with original models when activations are also quantized.

The authors claim that they can close the accuracy gap using rank equivalent to 30%, this seems to be true only for Phi3 which the authors present as an unqualified statement. Moreover, I cannot find the corresponding results in the evaluation section.

Thanks for pointing this. We have conducted additional experiments showing that using a rank of 30% of the original sizes at W4A4 enables to consistently recover the performances of the full precision model. Please find below the comparison between the full precision models and their quantized versions with LRC at W4A4 and a rank set at 30%. More details can also be found in Appendix B.3.

In L431, the authors use the phrase "10% additional ranks" which I believe is not meaningful.

We have corrected this formulation by "by incorporating low-rank weight matrices with ranks set to 10% of the original matrix sizes".

L399-407 sub section "on the effect of rank" does not mention the table/fig where the results can be found.

Thank you. We have corrected this error and referred to Figure 2.

Table 1, for Phi3, LRC(1) outperforms LRC(5), a weird anomaly since one would expect performance to increase monotonically with iters

Thank you for pointing this out. This phenomenon may arise from numerical instabilities associated with matrix decomposition and inversion processes inherent in LRC. To mitigate this undesirable effect, we are conducting experiments to optimize the regularization parameters $\epsilon_x$ and $\epsilon_y$ as specified in line 303. We anticipate presenting conclusive results from these experiments in the coming days.

Best results not provided in bold font

We have corrected this, thank you.

Why is it necessary to store the low rank matrices in full precision? Couldn't they also be quantized?

Certainly, it is indeed feasible to quantize these matrices, which would result in a small reduction in memory footprint. However, it is unlikely that this would lead to significant improvements in throughput. This is because the low-rank matrix multiplications are memory-bound, and in this scenario, the activations X (which are not quantized) will dominate the throughput. To address this, we would need a sequence of operations that compute low-rank matrix multiplications in mixed precision (for the non-quantized low-rank weight matrices and the activations), followed by rescaling for each low-rank matrix. Fusing these operations is challenging with the current PyTorch tools, although a dedicated kernel could potentially achieve good throughput. We leave these details for future work

Dear Reviewer,

We sincerely appreciate your valuable comments and questions, which have greatly contributed to improving our manuscript. We are also grateful for the increase in your score.

My intent is not to dispute whether memory is worth spending but that the increase in memory should be explicitly stated in a table column (similar to average bit precision or bits per weight in Ou et al) so that the reader can make an informed choice.

Thank you for clarifying this point. We definitively agree that the reader should be informed about the memory footprint required by LRC, and we have added a column in Table 3 of the manuscript that reports the model size obtained by the different quantization methods. These results are also detailed at the beginning of the rebuttal in the paragraph titled "Experiment: Memory Footprint of LRC".

Why does LRC outperform FP16 on so many cases?

Thank you for your insightful remark. The evaluation of quantized models' accuracy on this type of benchmarks can be subject to noise, as quantized models may correctly answer some queries that full precision models fail to address. This phenomenon has been investigated in a recent study [1], where the authors suggest that distance metrics such as KL divergence should be used instead. In our work, we report both Perplexity (PPL), which is closely related to the KL divergence, and the accuracies obtained on various benchmarks to mitigate the noise.

We hope that these clarifications adequately address your final concerns.

Thanks again for reading our rebuttal, and for your detailed review.

[1] Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee. "Accuracy is Not All You Need" (2024)

Below, we present the sizes of various models (in GB) at W4. It is noteworthy that low-rank methods incur approximately 13% additional weights relative to the original models when compared to QuaRot. We have also added a column in Table 3 of the manuscript to report these memory footprints.

Several reviewers inquired about the overhead associated with incorporating low-rank passes in FP16 with the int4 matrix multiplication. We did not address this in the paper for two main reasons: (a) numerous factors can influence timing, such as model size, batch size, hardware, and implementation specifics; and (b) developing an optimized kernel necessitates CUDA development, which is beyond our immediate expertise and outside the scope of this paper.

Nonetheless, we provide here two artefacts that we hope the reviewers find useful: a discussion of the computational implications of the ranks, and a pytorch experiment with latency measurements. These artefacts can also be found in Appendix B.2.

Quantizing models is appealing because of both memory and latency improvements. In our experiments, we settled on 10% additional ranks because this seems like a fair trade-off in terms of memory: effectively we are at 5.6 bits (4 + 0.1 * 16). We argue that this additional footprint is worth spending for improved downstream-task performance.

The additional FLOPS required are just 13% of the original model. Roughly speaking, int4 matmuls are twice as fast as fp16 on cuda devices, making a 'ballpark' estimate of our throughput 63% of FP16. But FLOPS are misleading: throughput on large models is often bottlenecked by data movement and deployment of models is often limited by footprint. As LLMs are deployed on specialist devices like Apple's silicon, or Qualcom/AMD accelerators on PC, the 'shape' of the hardware will lead to different latency results. In many cases, a mixed precision computation may be runnable using complementary parts of the hardware (e.g. int4 on an accelerator and fp16 on cpu).

We set up this simple timing experiment on an Nvidia A100 device to time the cost of a forward pass. We use a batch size of 32, sequence length of 2048, and matrix sizes from the Llama model series. We used Cutlass to implement a basic int4 kernel. We timed the cost of quantizing the activations, computing the int4 kernel, computing the low-rank matmul in fp16, and adding the results. Our pytorch module looks like this:

baseline_mod = torch.nn.Linear(feature_dim_in, feature_dim_out, bias=False).cuda().to(torch.float16)
class Int4Lowrank(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.quant = Quantizer(input_clip_ratio=1.0)
        self.U = torch.nn.Linear(feature_dim_in, ranks, bias=False).to(torch.float16)
        self.V = torch.nn.Linear(ranks, feature_dim_out, bias=False).to(torch.float16)
        self.lin_4bit = Linear4bit.from_float(baseline_mod, weight_scales=s_w)
    @torch.compile()
    def forward(self, x):
        return self.lin_4bit(self.quant(x)) + self.V(self.U(x))

Here are the timings of this simple layer, with warmup, repeated 100x. Matrix sizes taken from the Llama family.

We see that adding low-rank weight matrices does increase the latency of these operations as expected, though we still retain speedup relative to full FP16. In each row of each table, we have highlighted the choice of ranks that is above (next power 2) the 10% factor we used in the main experiments in the paper.

We have included numbers from very small ranks to emphasize a limitation of this experiment: even with a very small number of ranks added (128) there is latency loss. This implies that data movement is important, and that a fused kernel could improve latency.

This experiment is also limited in that it does not account from groupsizing, which would make the addition of low-rank matrices more appealing in terms of latency since int4 operations would themselves be reduced in speed.

We conducted a new experiment to investigate the impact of the calibration dataset selection on the performance of LRC. Our observations indicate that the choice of the calibration dataset does not significantly affect the performance of the quantized models on downstream tasks. Below, we present a comparison of LRC performance with a rank set to 10% of the original size on Phi-3 at W4A4. These results have also been added in Appendix B.1.

Results with groupsizing (128)

Results without groupsizing