Thank you for the valuable feedback. We address below your questions and concerns.

---

Q1. The objective is derived from changes in the end-to-end loss, which is similar to previous work & Concern on the simplicity of the approximation method.

We believe the simplicity of our approach should not be mistaken for a lack of novelty. On the contrary, we view the simplicity of our approach as a strength rather than a limitation, as it achieves strong and robust performance gains through a simple and efficient method (see Figure 2 and Tables 3-5 of the paper).

That said, we agree that exploring alternative grouping strategies, such as incorporating sophisticated clustering algorithms for more accurate Hessian approximation, is a promising direction for future research.

---

Q2. Evaluations on zero-shot and few-shot downstream tasks

Please refer to our response to Q1 from Reviewer WqtU, where we evaluate our method alongside baselines on eight zero-shot downstream tasks and the MMLU benchmark under a 5-shot setting. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with especially notable improvements under extreme quantization scenarios.

---

Q3. Comparison with uniform scalar quantization methods

We note that comparisons with uniform scalar quantization methods are included in our experiments. Specifically, Table 3 includes the baselines of weight-only uniform scalar quantization methods such as GPTQ and QuIP, and Table 5 includes comparisons for weight-and-activation uniform scalar quantization methods like QuaRot and SpinQuant. Our method consistently outperforms these baseline approaches in both weight-only and weight-and-activation settings, and shows effectiveness even when used with a uniform quantization scheme as well (SpinQuant + GuidedQuant (Ours) in Table 5).

---

Q4. Evaluations on different models

Please refer to our response to Q3 from Reviewer NTd9, where we conduct more experiments on Llama-3-8B and Llama-3-70B under a weight-only scalar quantization setting. The results show that our method consistently outperforms the baseline methods in these settings as well.

---

Q5. Derivation for closed-form solution in Section 4.2

The closed-form solution $\mathbf{c}^\ast$ in Section 4.2 (Eq. 10) is the standard least squares solution $\mathbf{c = A^\dagger y}$ for the problem $\min_{\mathbf{c}} \\| \mathbf{y - A c} \\|^2$, where $\mathbf{y = X w}$ and $\mathbf{A =X P}$. We assume that $\bf{A^\top A}$ is invertible, hence $\mathbf{A^\dagger = (A^\top A)^{-1} A^\top}$. This is a common assumption in quantization. In practice, $\bf{A^\top A}$ is not always invertible. To address this, we add $\bf \lambda I$ to $\bf{A^\top A}$, with a small $\lambda = 10^{-7}$, as commonly done in prior work.

---

Q6. Clarification on the perplexity evaluation setups and ensuring a fair comparison

To ensure a fair comparison, we use the perplexity values reported in previous papers only when the evaluation setup strictly matches ours, i.e., same calibration and evaluation dataset, context length, and a calibration data size that is equal to or larger than ours (giving the baselines an advantage). When any of these criteria were not met, we reproduced the baseline methods under the exact same setup as our experiments. We will clarify this in the revision.

---

Q7. Clarification on measuring GPU cost in Table 7 and 8

In Tables 7 and 8, we report the end-to-end runtime for each quantization method, along with the number and type of GPUs used (indicated in parentheses). For example, "1×R6A" refers to the runtime measured using a single RTX 6000 Ada GPU, while "4×R6A" indicates that the method was run using four RTX 6000 Ada GPUs.

---

Q8. Marginal improvements in quantization results with higher bits

The performance gains from our method are especially prominent in more challenging quantization scenarios. In particular, our approach demonstrates clearer advantages in lower-bit settings or when applied to models that are inherently harder to quantize, such as Llama-3 models. In these settings, our method yields more substantial improvements. We will include these discussions and corresponding results in the revision.

---

Q9. Non-uniform scalar quantization format for the GPTQ method

Non-uniform scalar quantization maps weights to non-uniform grids. GPTQ is designed to minimize the layer-wise output reconstruction error for a fixed quantization grid, thus it can be extended to non-uniform grids by also optimizing the choice of the grid. The GPTVQ 1D baseline does this; it applies the GPTQ algorithm to optimize assignments for a fixed quantization grid while optimizing the non-uniform codebook using gradient descent. We included GPTVQ 1D as a baseline in Table 3 in the paper and showed that both LNQ and LNQ with GuidedQuant consistently outperform it across all evaluated models and bit-widths.

Thank you for your positive review. We address below your questions.

---

Q1. Results on newer models

Please refer to our response to Q3 from Reviewer NTd9, where we conduct more experiments on Llama-3-8B and Llama-3-70B under a weight-only scalar quantization setting. The results show that our method consistently outperforms the baseline methods in these settings as well.

---

Q2. Evaluations on zero-shot and few-shot downstream tasks

Please refer to our response to Q1 from Reviewer WqtU, where we evaluate our method alongside baselines on eight zero-shot downstream tasks and the MMLU benchmark under a 5-shot setting. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with especially notable improvements under extreme quantization scenarios.

---

Q3. Clarification on the gradient / Hessian cache size

We note that GuidedQuant only requires the Hessian averaged within groups for each layer. Since this averaged Hessian can be computed using the group-wise averaged gradient values, there is no need to cache the full gradient with respect to the output tensor. Instead, we only store one averaged scalar gradient value per group, per layer.

For example, in the case of the Llama-2-7B model with $g=4$ number of groups, each layer only needs to cache 4 scalar values per layer for averaged gradients. The total gradient cache size is therefore computed as:

• $1024$ (batch size) $\times$ $4096$ (sequence length) $\times$ $4$ (number of groups) $\times$ $32$ (number of layers) $\times$ $7$ (number of modules per layers) $\times$ $2$ bytes (stored in 16-bits) = $7.5 \text{GB}$.

This averaging based approximation also applies to the Hessian computation, substantially reducing the storage cost compared to storing the full Hessian without approximation. This is how the cache sizes reported in Table 8 in the paper are derived. We will clarify this in the revision.

Thank you for your positive review. We address below your questions.

---

Q1. Lack of empirical evidence showing improved modeling after grouping approximation (Section 3.3).

We highlight that the GuidedQuant objective, after applying the grouping approximation, consistently demonstrates a clear empirical advantage over both the layer-wise objective and the diagonal Hessian approximation (i.e., the weighted $k$-means objective). This is evidenced by the performance gains shown in Figure 2, Tables 3, 4 and 5 of the paper.

To provide further qualitative evidence, we visualize a submatrix of the Fisher information matrix corresponding to the first two output channels in the linear layers of the first Transformer block of Llama-2-7B in Figures A1 and A2. Since each output channel contains $d_{\mathrm{in}}$ weights, the visualized matrix has dimensions of $2 d_{\mathrm{in}} \times 2 d_{\mathrm{in}}$ (here, $d_{\mathrm{in}}=4096$ for all layers except mlp.up_proj for which $d_{\mathrm{in}}=11008$). The visualizations reveal that the original Fisher information matrix exhibits strong off-diagonal values and a prominent block-diagonal structure with blocks of size $d_{\mathrm{in}} \times d_{\mathrm{in}}$, corresponding to interactions within each of the two output channels. GuidedQuant approximately preserves the off-diagonal terms within each output channel, thereby capturing significantly more structural detail than the diagonal approximation (SqueezeLLM), which ignores the cross-weight interactions. We believe this better structural modeling helps explain the performance gains achieved by our method. We will include these results and discussions in the revision.

• Figure A1: https://drive.google.com/file/d/1ilhJarLN1u1nNBzT2ywDCxg8IRz-T2oz/view?usp=sharing
• Figure A2: https://drive.google.com/file/d/19LJkHuKebYyOvPsqsjJQG_5PkJAWr7Bz/view?usp=sharing

---

Q2. Comparison with mixed-precision variant (dense-and-sparse) of SqueezeLLM.

Please refer to our response to Q2 from Reviewer WqtU, where we present a comparison between our method and SqueezeLLM under a mixed-precision (dense-and-sparse) setup. The results show that our method consistently outperforms the SqueezeLLM in this setting as well.

---

Q3. Results on newer models other than Llama-2 (Reviewers NTd9, i4yb, UEhr)

We have conducted additional experiments on newer models, including Llama-3-8B and Llama-3-70B, comparing our method with SqueezeLLM under a weight-only scalar quantization setting. We present the results in Table A4. LNQ with GuidedQuant consistently outperforms the baselines, demonstrating the robustness and effectiveness of our approach. We plan to conduct more comprehensive experiments on these newer models and will include the results in the revision.

• Table A4: https://drive.google.com/file/d/11bgdJ5eOO5s3LjjcZgVPyAzPsXDr6JJZ/view?usp=sharing

---

Q4. The gradients are computed on calibration data, which is unaddressed in Appendix A when connecting the objective to the Fisher matrix.

In Proposition 3.1, we define the Fisher information matrix $\mathbf{F}$ as the empirical Fisher information matrix computed over the calibration data, $\mathbf{F} = \frac{1}{n} \sum_{i=1}^n \nabla \ell_i (\mathbf{w}) \nabla \ell_i (\mathbf{w})^\top$. Accordingly, the proof assumes that gradients are computed using the calibration data and is connecting the objective to this empirical Fisher information matrix. In the revision, we will explicitly refer to $\mathbf{F}$ as the empirical Fisher information matrix on the calibration data and clarify that it is an approximation of the true Fisher information matrix.

---

Q5. Minimal gains with a bigger number of groups, and lack of explanation on the choice of hyperparameter $g$ (e.g., $g=4$ in 7B models).

We believe that achieving strong performance even with a small number of groups is a strength of our method, making it particularly effective and robust in resource-constrained settings. Regarding the hyperparameter choice, we selected the number of groups $g$ to be as large as possible within the limits of our computational and memory constraints. We will clarify this in the revision.

---

Q6. Additional comments and suggestions.

Thank you for the suggestions. We will revise the writing to improve overall clarity, including correcting typos, reducing redundant explanations, and refining figure captions for clarity.

Thank you for your positive review. We address below your questions.

---

Q1. Evaluations on zero-shot and few-shot downstream benchmarks (Reviewers WqtU, i4yb, UEhr).

We provide evaluations of our methods (LNQ and LNQ + GuidedQuant) alongside baselines (SqueezeLLM and GPTVQ 1D) under the weight-only scalar quantization settings, using Llama-2-7B and Llama-2-13B models, in Table A1. The evaluation includes eight zero-shot tasks: BoolQ [1], PIQA [2], SIQA [3], HellaSwag [4], WinoGrande [5], ARC-easy [6], ARC-challenge [6], and OBQA [7]. For a few-shot benchmark, we include results on the MMLU [8] benchmark in a 5-shot setting.

Table A1 reports both accuracy and standard error for all methods. We highlight the best-performing results, as well as those whose accuracy falls within the top score $\pm$ standard error, under the same bit width constraint. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with notable improvements in extreme quantization scenarios, such as 2-bit quantization. While we currently report results on a subset of models and settings due to time constraints, we will perform a more comprehensive set of experiments and include the full results in the revision.

• Table A1: https://drive.google.com/file/d/1XnAs0V8CwTKpoul5I3cIVecE5JL8cgto/view?usp=sharing

---

Q2. Comparison with Dense-and-Sparse variant of SqueezeLLM (Reviewers WqtU, NTd9).

To ensure a fair comparison, all methods in the paper were evaluated under uniform precision. The dense-and-sparse variant of SqueezeLLM, which preserves a small fraction of weights in 16-bit precision to maintain accuracy, is orthogonal to our method and can be combined with it. Accordingly, we report results for SqueezeLLM, LNQ, and LNQ + GuidedQuant methods, with the dense-and-sparse approach applied to all of them, in Table A2.

Following the original SqueezeLLM paper, we retain 0.45% of the weights in 16-bit and evaluate with 2-, 3-, and 4-bit quantization on the Llama-2-7B model. The results show that LNQ with GuidedQuant consistently outperforms the baselines in the dense-and-sparse setting as well, demonstrating the superiority and robustness of our method. We will include these results in the revision.

• Table A2: https://drive.google.com/file/d/1dORQnlmUi1FPQmuSR8n1OebNQ5h0M8Gi/view?usp=sharing

---

Q3. In weight-and-activation quantization (Table 5 in the paper), the GuidedQuant does not show obvious advantage over the baseline.

This is primarily because we present weight-and-activation results using relatively high bit-width for weight quantization (4-bit), where the benefits of our method are less evident. Our method demonstrates more significant improvements in more aggressive quantization settings. To illustrate this, we conducted additional experiments with lower bit-widths for weights, specifically 2-bit and 3-bit, while keeping activations and KV caches at 4-bit precision (denoted as W2A4KV4 and W3A4KV4, respectively), on Llama-2-7B model. The results, shown in Table A3 below, demonstrate that GuidedQuant significantly outperforms baseline methods in these more extreme scenarios, highlighting the strength of our approach under stricter bit-width constraints. We will include these new results and the corresponding discussion in the revision.

• Table A3: https://drive.google.com/file/d/1PaDjWnzsWEYVd5Lwdd_Xoath72Fwfghr/view?usp=sharing

---

References

[1] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. 2019.

[2] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. 2020.

[3] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. 2019.

[4] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? 2019.

[5] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. 2021.

[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.

[7] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.

[8] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. and Steinhardt, J., Measuring massive multitask language understanding. 2020.