DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization

We are grateful for your review and positive feedback. We will address each of the weaknesses and questions raised (in a combined way) in detail below. We will be happy to answer any further questions.

Strengths2: But I'm not sure if the gradient of qr decomposition is stable or not.

Thank you for the comment. Regarding the gradient stability of QR decomposition, we provided empirical analysis in our reply to SzJU Question4. Under various initializations and hyperparameters, the autograd-based QR-Ortho consistently converged and showed strong robustness to learning rate. Hence, it is stable in practice.

Weakness1: The main weakness is that the improvement is not significant. The performance is very marginal compared to SpinQuant. The improvement on compression time and memory is large. But I doubt whether compression time is an important objective in quantization, compared with time and memory during inference, or performance.

Thanks for the feedback. First, we would like to highlight the practical importance of compression time in post-training quantization (PTQ), especially as model sizes and deployment scales continue to grow. Our primary contribution is improving the efficiency of rotation matrix optimization, a point that has also been positively noted by reviewers sxB6 and SzJU.

For example, SpinQuant requires over 40 GPU hours (8×A800) to optimize LLaMA 2-70B. As models continue to scale (e.g., LLaMA 3.1 405B), and development cycles accelerate, optimization cost becomes an increasingly important factor in the applicability of PTQ techniques.

This trend is also observable in practice: although some existing methods can achieve strong accuracy, their relatively high computational cost can limit practical adoption. In contrast, methods with lower overhead (e.g., AutoGPTQ) have seen broader use in the community, due to their balance between speed and effectiveness. Similarly, while SpinQuant provides strong accuracy, its higher resource cost may affect its ease of use in large-scale or time-sensitive settings.

Another point we would like to emphasize is that, DartQuant supports further accuracy improvements by increasing calibration data size, especially when extreme efficiency is not the primary concern. For instance, using 512 calibration samples on 7B/8B models, the optimization time remains moderate (about 24.97 minutes), approaching the runtime of SpinQuant while offering higher accuracy. The experimental results are as follows:

From the results, it can be observed that our method achieves a significant accuracy improvement over SpinQuant when the optimization time is aligned. In summary, we believe that DartQuant is more practical for deployment with a small calibration set due to its high optimization efficiency. On the other hand, in scenarios where optimization time is not a major concern, our method can achieve outstanding accuracy. If you have any further questions regarding this, please do not hesitate to let us know.

Weakness2: In four rotation matrix R1 ... R4, only R1 and R2 are learned by DartQuant, while R3 and R4 use Hadamard rotations like previous work, which limits the value of this method. By the way, the Hadamard matrix of R3 and R4 is random, right?

In current experiments, we apply DartQuant to R1 and R2, and use random Hadamard for R3 and R4 to minimize inference latency, following SpinQuant and OSTQuant.

However, DartQuant is also compatible with R3/R4 optimization. Below is the accuracy improvement when optimizing R4 with DartQuant.

Question1: What is the metrics of outlier in Figure 3? It should be included in the article.

Outliers in Fig.3 follow PrefixQuant[1]: values >3×median (before rotation). We will clarify this definition in the appendix.

[1] Chen, Mengzhao, et al. "Prefixquant: Static quantization beats dynamic through prefixed outliers in llms." (2024).

Question2: Which baseline is used in Table 1? It should be stated in the article.

The baseline in Table 1 is consistent with Table 2 and detailed in Appendix C. We will clarify this in Table 1 to avoid confusion.

Question3: Line 283 states "In the main results, we apply GPTQ to rotate the weights.", but GPTQ doesn't have rotate. I guess it's "reconstruct"?

You are absolutely correct that Line 283 contains a misstatement. GPTQ does not involve weight rotation; rather, it reconstructs weights via quantization-aware optimization. We will revise the wording to accurately reflect that GPTQ applies reconstruction, not rotation.

Question4: Why could DartQuant avoid overfitting? It still use calibration data for both rotate and reconstruct (if above is true).

DartQuant uses calibration data, but unlike end-to-end methods, it optimizes a proxy target: distribution alignment via Whip loss, not final quantization error. This indirect optimization reduces overfitting to calibration data. As shown in Table 5, performance remains stable across calibration sizes, with no “diagonal effect.” Thus, the distribution-alignment design helps DartQuant avoid overfitting seen in SpinQuant.

Qusetion5: Line 25 is wrong. lora doesn't improve inference efficiency.

We acknowledge the inaccuracy in Line 25: while LoRA is effective in reducing fine-tuning cost and memory footprint, it does not bring improvements to inference efficiency. We will amend this reference to prevent any misunderstanding.

We appreciate your attention to detail and your valuable suggestions, which help us improve the clarity and accuracy of our work.

We sincerely thank the reviewer for the encouraging comments, and we are glad to hear that you found our idea interesting. Below we provide detailed responses to each of your comments and concerns.

Question1: I found the idea interesting. The issue with the current scheme is that the improvement is a bit marginal even when we compare against QuaRot which doesn't do anything (no training). However, this is nothing with the paper, but more with the PTQ schemes. It seems that we hit the wall, considering the limits of the 4-bit compression. That's why we cannot expect large improvements in all these kinds of work.

We appreciate your valuable observations. While QuaRot indeed serves as a strong baseline for 4-bit PTQ quantization, we believe the improvements introduced by rotation matrix optimization are still meaningful—particularly for more recent and challenging models such as LLaMA 3 8B/70B, 3.2B, or 1B, and in even lower-bit quantization settings.

For instance, in the 4-4-4 setting (Table 2), DartQuant improves the performance of LLaMA 3 8B from QuaRot’s 12.29/57.32 (PPL/Zero-Shot) to 10.78/62.38, and LLaMA 3 70B from 11.38/61.50 to 8.13/69.05. These are non-trivial improvements.

We would also like to emphasize that, in the original paper, we prioritized extreme optimization efficiency by using a very small number of calibration samples. However, if one trades some speed for better accuracy, DartQuant benefits significantly from larger calibration sets (512 samples same as [1], SpinQuant use 800 samples). Detailed results are as follows:

As shown, under larger calibration sets, DartQuant consistently outperforms QuaRot on both models. In this light, we respectfully suggest that: "Yes, it is possible to expect substantial improvements from our method."

[1] KurTail: Kurtosis-based LLM Quantization, SLLM workshop ICLR 2025

Question2: Considering the above fact, I think the improvement would be even closer (compare to something like QuaRot) in more recent 4-bit formats. For this, can you please provide an experiment where you compare DartQuant with QuaRot (I do not suggest SpinQuant as I know that will take 4-5 hours to run) on NvFP4? I would improve my score if we get non-trivial improvement over that format.

To further illustrate DartQuant’s advantages in more difficult scenarios, we conducted additional experiments on LLaMA 3.2B and 1B under 4-bit and 3-bit quantization. In line with your suggestion, we also included evaluations under the NvFP4 format.

Specifically, we implemented NvFP4 quantization using the FP4 codebase from the microsoft/microxcaling repository. Due to the use of complex secondary quantization in NvFP4, the inference speed is substantially reduced. Given limited time, we report results only on the two smaller models mentioned above. The quantitative results are presented below.

The results show that for these more quantization-sensitive models, there remains considerable room for improvement over QuaRot. In the 3-bit setting, the benefits of rotation matrix optimization become even more pronounced. This supports our hypothesis that the more aggressive the compression, the more valuable such optimization becomes.

Regarding NvFP4, although the Whip loss was originally designed for uniform quantization and may not be optimal for non-uniform formats like FP4, we find that the distribution tightening it induces still benefits the overall quantization. As a result, DartQuant + NvFP4 yields measurable gains over QuaRot + NvFP4.

Looking ahead, we plan to explore alternative loss functions that are more tailored to the characteristics of FP4 and other non-uniform quantization formats, to further improve performance in such settings.

In summary, we believe that for more difficult quantization scenarios—especially low-bit formats and challenging models—exploring rotation matrix optimization remains a promising and meaningful direction.

Dear reviewer gzqM,

We would like to thank you for your thoughtful comments and suggestions for our paper. We hope our response has addressed your questions. If you have any further concerns, we would be happy to clarify them. If our response has resolved your concerns, we would greatly appreciate your positive feedback and a possible improvement in your score.

Thanks!

Dear Reviewer gzqM,

Thank you very much for your response! We would like to address your further concerns promptly and thoroughly one by one.

I agree with the author that these are non-trivial improvements. However, the gap with the original model remains significant—for example, in the 4-4-4 case on LLaMA3-8B, there's a drop of around 2 perplexity points, which is substantial. In configurations where DartQuant performs closer to the baseline—such as 4-4-16 on LLaMA3-8B—it achieves 0.1 PPL points better than QuaRot and only 0.01 worse than SpinQuant.

First, we appreciate your recognition of the performance improvements achieved under the 4-4-4 setting.

In response to your doubts about the marginal gains of our method over QuaRot at 4-4-16 on LLaMA3-8B, we wish to clarify that DartQuant reduces PPL relative to QuaRot by 1.16 (from 11.74 to 10.58). I guess you are talking about the results of the 4-8-16 setting. As a result, to further demonstrate the superiority of DartQuant over QuaRot, we additionally provide a comparison using 512 calibration samples (for details about the alignment of calibration set sizes, please refer to the clarification in point 2). The comparative results are shown in the table below:

As indicated, DartQuant reduces PPL loss by more than 50% compared to QuaRot. We would like to emphasize once again that, when a certain amount of optimization time is acceptable, DartQuant also delivers significant performance gains over QuaRot under the 4-8-16 setting.

Thank you so much for providing more results. However, how many data points did you use for QuaRot in the above table? It would be great if we can see the results on the same number of calibration samples for DartQuant, QuaRot, and SpinQuant.

There are two places where calibration samples are used. One for rotation matrix optimization, and one for the GPTQ weight quantization process.

For rotation matrix optimization: (1) QuaRot employs only random Hadamard matrices for rotation quantization, without optimizing the Hadamard matrices; thus, no data points involved, and the results presented in our table are directly from the original QuaRot experiments. (2) SpinQuant utilizes 800 data points to optimize the rotation matrices end-to-end. We use 128/512 samples for DartQuant, which is less than SpinQuant. Despite the smaller calibration set, as has shown in the rebuttal, DartQuant still significantly outperforms SpinQuant. In summary, the table provided in our rebuttal reflects a fair comparison. Furthermore, to strictly align with SpinQuant, we will report the results of DartQuant with 800 samples very soon within the discussion period.

In addition, if your concerns pertain to the calibration set size used in the GPTQ quantization process, we clarify that, in all experiments (both in the original paper and the rebuttal), we used 128 calibration sets of length 2048 sampled from WikiText2, ensuring complete fairness in this regard as well.

Further explainations about the value of DartQuant

It seems that our biggest disagreement lies in whether DartQuant has its significant contribution considering the accuracy drop for the 4-4-4 case. From your viewpoint, although DartQuant has a non-trivial improvement over QuaRot, there is still a drop of around 2 perplexity on models harder to quantize (like LLaMA3-8B), which makes DartQuant useless in practice. While we value your perspective, we respectfully hold a different opinion.

First, for some circumstances where accuracy is not very sensitive, the 4-4-4 deployment is still a good choice considering the efficiency and accuracy trade-off.

Second, although DartQuant still have more than 2 perplexity drop by now, we believe it already achieves a significant improvement over previous methods. This is significant, because as an researcher in this field, what we are doing is to push the quantization limit step by step, and we believe DartQuant is an meaningful step towards more efficient and accurate PTQ method for LLMs. Just like QuaRot and SpinQuant, which have not solved the W4A4KV4 quantization problem, but they have inspired the following research to make a further improvement. And we believe our findings in DartQuant paper could also inspire further research in this field. Thus, we sincerely request you to evaluate our contributions to this field.

We sincerely thank the reviewer for the highly positive feedback and your thoughtful questions. We deeply appreciate your recognition of our work. We provide our point-by-point responses to the raised questions below.

Weakness: If will be helpful if the author show some inference latency/throughput/memory results even though it may be the same as other ratation-based methods like SpinQuant.

As suggested, we evaluated the inference performance of DartQuant (W4A4KV4) using LLaMA 2-7B on single A800 GPU under various batch sizes and input lengths using randomly sampled prompts and compared it with FP16 baseline. Specifically, we report prefill time, decoding speed (fixed 50-token generation), and peak memory usage (torch.cuda.max_memory_allocated()). Results are provided below:

Question1: Could the author clarify the quantization format for activations? I assume the weight matrix is quantized using groupwise scales and integer quantization. What about activations?

Thank you for raising this. As stated in Line 258, all activations are quantized using per-token asymmetric integer quantization.

Question2: The Whip loss is impressive. Does the Whip loss assume the activation distribution is symmetric over zero? What if the activation mean is not zero?

This is an excellent question that highlights a potential limitation of the current Whip design. Indeed, Whip implicitly assumes that the activation distribution is approximately zero-mean. While this assumption often holds in practice, it is not strictly guaranteed.

When the activation mean deviates significantly from zero, the effectiveness of Whip may be reduced. In such cases, applying zero-mean normalization to the activations before computing the loss could yield more stable optimization. We will add a discussion of this assumption and its implications in the revised version. We sincerely thank the reviewer for this insightful observation.

Question3: In Algorithm 1, what does the function "Row_sampling(X)" mean?

"Row_sampling(X)" refers to randomly sampling a subset of activation tokens from the full matrix for gradient computation in each optimization step. This strategy improves efficiency by reducing computation without sacrificing performance.

To avoid potential confusion caused by the name, we will rename this function to "Token_sampling(X)" in the revised version for better clarity.

Question4: Could the author give more information about the objective in Algorithm 1 like variance across initialized with different random seeds and learning rates? Is the problem easy to solve for gradient descent?

Thank you for this thoughtful question. As the optimization involves random sampling, the process is inherently non-deterministic and not exactly reproducible across different random seeds.

Nonetheless, we conducted multiple independent runs under varying random conditions (implicitly covering different seeds and sampling paths). As shown below, the final loss consistently converges to similar values across runs, and with comparable iteration counts. This empirical evidence suggests that the optimization problem in Algorithm 1 is well-conditioned and tractable via gradient descent, without suffering from instability or high variance.

We thank the reviewer for their valuable comments and suggestions. We have thoroughly reviewed each point and provide our detailed responses below.

Weakness1&Qusetion3 More model & task

In Appendix H, we experimentally demonstrate the effectiveness of our method on two MoE models (Mistral-8x7B and Deepseek-moe-16B). Following your suggestion, we also conducted comparative experiments on Qwen, Phi, and Mistral models across 5 zero-shot tasks, as well as additional evaluations on math reasoning datasets GSM8K and MathQA.

• Our method significantly outperforms QuaRot in perplexity, zero-shot accuracy, and math reasoning on all three models.

• Notably, on the math reasoning task GSM8K, our method achieves a remarkable improvement over QuaRot (e.g., a 16.52% increase on Qwen 2.5 3B), indicating promising advantages on tasks sensitive to quantization complexity.

Due to cost, we only ran LongBench on LLaMA 3 8B, where our method nearly halves the performance degradation compared to QuaRot. Different subsets of Longbench use different metrics (including F1 score, Rouge, etc.). For simplify, we used the average of these scores to evaluate the overall performance. The comparison of scores for each subset can also be seen in the following table.

Weakness2&Qusetion2 Mixprecision comparison

In Appendix F, we provide detailed comparisons with two classical mixed-precision quantization methods (QUIK and Atom). In line with your advice, we also compare with [3] and [4]. Using the publicly available code from [3,4], after aligning quantization settings, we performed the comparisons. Note that [3] does not support Mistral, and [4] doesn't support Qwen, so these results are omitted. The results are as follows:

From the result, DartQuant performs comparably to [3] and better than [4]. Notice that, [3] achieves higher accuracy by combining rotation with high-bit outlier protection, while DartQuant improves low-bit quantization precision through optimizing the rotation matrix. The two methods are algorithmically orthogonal and can be combined for further improvements.

Weakness3 vs [5]

We have reported Whip vs. kurtosis results under DartQuant settings in Appendix I, demonstrating Whip’s superiority. To further validate this, we compared Whip to the method in [5]. As [5]’s code is unavailable and the authors have not yet responded to our inquiries, we re-implemented their approach strictly following their published settings, using 512 calibration samples (128 samples in paper for efficiency). The following results (with [5]’s paper results as reference) show that, with equal calibration samples, our Whip loss achieves significantly better performance on two different models. This aligns with Figures 6 (lines 395-362) and Appendix I, confirming that Whip pushes activations closer to uniformity, outperforming other losses.

Weakness4 Why whip

The Whip loss formulation follows the transformation in Eq. 3. As shown in Fig. 5, Eq. 3 expands the near-zero region of the original distribution to a larger range. The Whip gradient near zero is much larger than that at the tails (due to the exponential function), which is key to Whip’s efficiency, as it rapidly pushes small values up in the rotated data. Under norm-invariant constraints, outliers are quickly suppressed.

We also use Llama 3 8B tested two functions with similar shapes but gentler gradients than Whip:

• $L1= \sum ReLU(1-|x_i|/\alpha)$, where $\alpha$ is set to three times the median (outlier threshold)

• $L2= \sum 1/(|x_i|+1)$

With identical hyperparameters, these alternative losses show inferior performance compared to Whip, demonstrating Whip’s advantage.

Weakness5 Speedup detail

We clarify that QR-Orth achieves a 1.4× per-iteration speedup over Cayley (Table 4), a 41× “total speedup” to converge to the same loss (line 375), and a 47× speedup in rotation matrix training compared to SpinQuant (line 332).

Weakness6&Qusetion1 QR-Orth vs Cayley

Section 5 of [1] points out that the Cayley transform’s iterative form is a contraction mapping, with Theorem 1 proving convergence only if step size $\alpha < 2 / \|W\|$. Otherwise, updates may diverge or leave the manifold, limiting Cayley SGD’s step size and slowing convergence.

QR-Orth, however, projects updated parameters back onto the manifold without this constraint, allowing larger update steps and faster convergence.

In the same hyperparameter settings, we measured the average parameter update magnitude over 10 iterations: QR-Orth’s updates are 1.2× larger than Cayley’s, consistent with the reasoning above.

Notably, the Cayley implementation follows SpinQuant’s original hyperparameter configuration in paper, which has been tuned and validated for this task. To rule out suboptimal hyperparameter effects, we tested multiple learning rates. Across all tested settings, QR-Orth achieved consistently lower losses than Cayley, confirming our claims.

Finally, we sincerely thank the reviewer for pointing out the lack of citation. We acknowledge that QR-based orthogonalization is a well-established tool in manifold optimization [2], and we do not claim novelty in the algorithm itself. Our contribution lies in being the first to adapt QR-Orth for rotation-matrix training in LLM post-training quantization. We will revise the final version to clarify this point and avoid any overstatement.

Weakness7 Memory usage

As Fig. 4 illustrates, DartQuant’s process has two stages. First, the LLM runs once on the calibration set to collect activations stored locally. Then rotation matrices are trained offline on this stored data, so no large model loading is needed during rotation optimization. For large models such as LLaMA 70B, we adopt a layer-wise loading strategy allowing inference to be performed on a single GPU. We further clarify these details in Appendix K to avoid misunderstandings.

Finally, we plan to release our code to facilitate reproducibility within the community.

Weakness8 Visualization

Thank you for your suggestion. We will add histograms of post-rotation activation distributions in the revision to better demonstrate DartQuant’s effectiveness.

Limitations

In addition to being tailored for uniform quantization formats, we acknowledge a potential limitation regarding the distributional assumption in the Whip loss. Specifically, Whip assumes that activations are approximately zero-mean, which generally holds true for most transformer layers. However, in rare cases where the activation mean significantly deviates from zero, the effectiveness of Whip may degrade. As a remedy, we suggest applying mean-centering or zero-mean normalization prior to optimization, and we will explicitly include this consideration in the revised version. We also plan to explore adaptive variants of Whip that are robust to non-zero-mean activations in future work.