QuEST: Stable Training of LLMs with 1-Bit Weights and Activations

Why is the method named QuEST?

We apologize for the omission, QuEST stands for Quantized, Efficient, and Stable Training.

Can you explain the overlapping W3A3 and W4A4 curves in Figure 1?

The decrease in training precision corresponds to fixed parameter count models shifting to the left. The performance degradation due to increased compression increases the loss and shifts the points upward. Between W3A3 and W4A4, those effects cancel out, landing the points on seemingly “the same” loss-to-model-size curve. In Section 4.4, we presented the “precision efficiency (eff(P))” metric to quantify this trade-off and accurately determine the optimal precision to be W4A4.

What challenges do you anticipate in scaling to larger models (7B+)?

We did not observe any training instabilities or unexpected performance degradation when scaling the model size. As such, the main challenge we anticipate is the need to commit hundreds of thousands of GPU-hours to test this novel approach for a production-scale pre-training run.

Comparison with recent works like ARB-LLM, QuaRot, and Kumar et al. (2024).

We did not include a comparison to PTQ methods (QuaRot or ARB-LLM) since they are not competitive with QAT: many PTQ methods work in one-shot, whereas QAT methods perform training or re-training by definition.

To fully address this concern, below we present a comparison of C4 validation PPL of the 800M models trained by us using BF16, QuEST INT4, and QuEST INT8. We then show numbers obtained relative to QuaRot PTQ (as suggested by the reviewer), and round-to-nearest quantization (RTN):

These results clearly show that PTQ isn’t competitive with QAT. Specifically, notice that our W4A4 model is more accurate than the QuaRot model in W8A8. To your question, ARB-LLM focuses on the simpler problem of weight-only quantization which is not the main focus of our work, whereas Feng et al. (2024) focuses on PTQ. As such, we largely omitted these comparisons from the paper.

We will provide additional background citations and clarification on this topic in the next version of the paper.

Please see the answer to Reviewer HH36 for the relationship with Kumar et al. (2024).

Experiments only cover small models (30M-800M parameters).

Notably, the largest models we trained (800M) closely match the size of the Meta Llama-3.2-1B model (up to a smaller embedding layer). This puts us at the lower end of current standards in terms of model sizes. Importantly, our study considers slightly larger models than the concurrent work of Kumar et al. (2024).

Runs on the 800M model require around ~1600 A100 GPU hours, with the total experimental cost of our submission being of around 12000 GPU hours. This makes it hard to scale further in an academic setup. Nevertheless, we have confirmed out results at 1.2B-parameter scale in the answer to Reviewer HH36, and are working towards a 3B-parameter run on 300B tokens (~40,000 A100 GPU hours).

We hope the reviewer can appreciate that this is very computationally-intensive and does not fit within the scope of the rebuttal.

Testing is limited to C4 dataset only, making all conclusions restrictive, e.g., lacking reasoning tasks like GSM8k or AIME performance.

Please note that our testing is not limited to C4: Appendix A.3 included the 0-shot evaluations of some of the models. To better address this issue, we present additional 0-shot evaluations (HellaSwag, ARC-Challenge, ARC-Easy, PiQA, Winogrande) of a broader set of models we trained:

https://github.com/QuEST2025/speedup/blob/main/zero-shots.md

These results are again consistent with the C4 evaluations. As for the reasoning tasks like GSM8k or AIME, they are outside the reach of 1B-parameter models without an explicit instruction fine-tuning phase.

The success of the Hadamard transform in PTQ makes HT's effectiveness in QAT unsurprising.

Regarding the Hadamard Transform (HT): We emphasize that the context in which we analyze HT is different from all the mentioned PTQ methods. Specifically, PTQ methods utilize the Hadamard transform to mitigate outliers in model weights and activations, and obtain better quantization grid fit. In addition to this, we show novel effects of HT on:

1. Improving gradient estimator alignment, as discussed in Section 3.2 (line 205)
2. Circumventing the “dead weight problem” in gradient masking, as discussed in Section 3.3 (line 215) and Appendix A.1, by making sure that all weights get gradient.

These effects are unique to QAT; to the best of our knowledge, we are the first to explore them.

Detailed discussion on how its (QuEST’s) advantages manifest.

The fact that QuEST enables stable training across scales has the following implications:

1. If using QuEST, INT4 is the “optimal” bit-width for weights and activations in terms of inference effectiveness, that is, the accuracy that can be obtained at a given model size. This is a strict improvement relative to STE, which was found in Kumar et al. (2024) and in our experiments to only provide competitive low-bit training at 7-8 bit weights and activations.

2. As the reviewer remarked, this finding holds and transfers across all model scales: thus, future large-scale pre-training runs could leverage this technique to produce highly-accurate models with low precision.

3. Our approach also enables a direct comparison between the “effectiveness” of different precisions P, namely the efficiency factor eff(P). Thus, based on runs on small models, the user can determine the “optimal” precision for a given model architecture and hardware target (which may influence the set of precisions supported).

To validate our findings at a larger scale, we trained a 1.2B-parameter model over 40B tokens (2’000 A100 GPU hours), which was the largest size we could train on our academic cluster. The accuracy findings, including C4 loss and 0-shot evaluations, confirm that our findings indeed scale to this larger model size: https://github.com/QuEST2025/speedup/blob/main/1200M.md

We are currently working towards a 3B-parameter run on 300B tokens (~40’000 GPU hours), but we hope the reviewer can appreciate that this is very computationally-intensive and does not fit within the scope of the rebuttal.

Detailed explanations of [GPU Kernel] optimizations.

We believe there may be a slight confusion here. As stated in the paper, we utilize the highly optimized CUTLASS operations for the “raw” matrix multiplications in both 16-bit (BF16) and 4-bit (INT4) precisions. Thus, these basic operations are heavily optimized by NVIDIA, the makers of the library and the hardware. Our main focus is to reduce the inference overheads over the QuEST format: that is, quantization/dequantization, clipping and Hadamard multiplication. This is overviewed in Section 5, and we would be happy to describe it further in the discussion.

The paper only compares QuEST with BF16, which is insufficient to comprehensively evaluate the effectiveness of the method.

Our speedup ratio experiments compare: the BF16 baseline, our approach (QuEST), and an idealized 4bit version which does not require the Hadamard multiplication (No HT).

We use BF16 data type in the baseline because

1. Our experiments use smaller-scale Llama-3 models. The full-precision data type for Llama-3 models is BF16.

2. BF16 and FP16 are common weight types in the popular open-source models (e.g., Llama, Qwen, etc.). BF16 and FP16 are equally supported by modern GPUs (e.g., Ampere and Hopper architectures). They have the same computational performance. The results for BF16 also apply to FP16.

Our BF16 baseline is the official NVIDIA libraries (i.e., cuBLAS/cuDNN) that implement the GEMM routine used under-the-hood by e.g. PyTorch. These codes have been heavily optimized by NVIDIA to achieve near-optimal performance on their hardware.

The results show that:

1. Our approach leads to significant inference speedups relative to the extremely competitive full-precision baseline, which is standard for all open models, especially on large matrices. Note that this occurs at the same or better accuracy (as per our results).

2. The overheads of our format, including activation clipping and Hadamard transforms, are small (less than 10% on average, and 30% in the worst case).

3. Our results are not surprising, since they track well with the expected speedups from low-precision GEMM for the corresponding matrix sizes.

We would be happy to run additional comparisons that the reviewer would find relevant in this context.

Tests should be conducted with different sequence lengths and batch sizes.

We agree; to address this, we conducted more tests on different sequence lengths and batch sizes. You can find the speedup results using this link https://github.com/QuEST2025/speedup/blob/main/tables.pdf

In addition, we conducted further experiments on the scalability of our CUDA kernels in isolation, by increasing the arithmetic intensity of the GEMM problems. The following table shows the speedup achieved with a fixed N = 4096 while varying the M (batch times sequence length) and K (hidden) dimensions.

We hope this clarifies your concerns, and are happy to continue the discussion if the reviewer finds it necessary.

Thank you for the detailed review! We address all your questions below.

I'm not convinced that the 'trust factor' approach to masking out gradients is well motivated.

Thank you for raising this. First, trust factor masking is motivated theoretically by directly targeting the source of error in the QAT iteration, relative to standard SGD.

Specifically, the “ideal” SGD iteration is $x_{t+1} = x_t - \nabla L( x_t)$. Instead, in QAT we execute $x_{t+1} = x_t - \nabla L( Q(x_t) ),$ where $Q(x_t)$ is quantization. The “error” between these iterations is precisely the $\| \nabla L( Q(x_t) ) - \nabla L( x_t ) \|$ term we seek to minimize.

The theory of inconsistent SGD iterations shows that this “error” directly impacts SGD convergence: see the work of Nadiradze et al. (https://arxiv.org/pdf/2001.05918), who bounded this error by applying smoothness, and Lin et al. (https://arxiv.org/pdf/2006.07253) who investigate the same for sparse projections. In this context, our work investigates a fast heuristic for minimizing this error in the context of quantization.

Second, our practical results confirm that trust factors are key to good practical convergence. Besides Figure 2, we also illustrated this in Appendix Figure 10, which showed that properly tuned trust factors lead to much better loss than both clipping (s = 1) and STE (large s).

There does not seem to be sufficient evaluation to establish QuEST as a new SOTA for QAT.

Note that, generally, QAT methods usually fall into two categories:

1. General schemes that can be applied to any bitwidth (such as STE, PACT, LSQ, and QuEST).
2. Schemes specialized to some bit-widths, e.g. binarization, such as AdaBin.

For general schemes, LSQ was SOTA: e.g., recent work from IBM & MIT (https://arxiv.org/abs/2404.03605) on outlier suppression still uses a variant of LSQ, whereas the recent work of Kumar et al. only uses STE. Since our work is focused on general scaling laws, we did not consider specific binarization schemes, as they do not directly “port” across bit-widths.

To fully address your concern, we have ported AdaBin to our setting and compared it with QuEST at 30-100M scale. Remarkably, we found that QuEST consistently outperforms AdaBin in the W1A1 setting (for which AdaBin is specifically designed). Moreover, the stripped-down QuEST without the Hadamard recovers the performance of AdaBin:

https://github.com/QuEST2025/speedup/blob/main/AdaBin.md

We hope this clarifies our positioning: we believe QuEST is indeed the new SOTA for general QAT.

The claim of improved efficiency seems somewhat orthogonal. The goal of our kernels is showing that QuEST models can execute fast; this is not obvious since they require a Hadamard multiplication and dynamic clipping of the activations on the forward pass.

It was unclear to me how well optimized the baseline in Figure 6 is.

The baseline in Figure 6 is near-optimal for BF16; please see more details in our reply to Reviewer HH36.

Editorial comments and comments about supplementary material.

Thank you for the detailed examination and useful editorial notes! We will address all these in the next revision, and add a more detailed README for the CUDA code structure.

The use of Hadamard Transform seems similar to PTQ (e.g., QuIP#).

Please see “Regarding the Hadamard Transform (HT)” part of the reply to Reviewer Xwq9.

It seems incorrect to refer to "Pareto-optimal frontier" as a metric.

Pareto-optimality is defined in the introduction (page 1, col.2, l.49-50), and follows Frantar et al., ICLR24. We say that an approach X (e.g., QuEST INT4) is Pareto-superior to Y (e.g., STE INT8) if X provides better accuracy at the same model size, or, symmetrically, smaller size at the same accuracy. Figure 1 shows that QuEST INT2 is Pareto-superior to BF16 pretraining, but inferior to QuEST INT4. Thus, QuEST brings the “optimal” precision in terms of accuracy-vs-size to INT4, since no other method dominates QuEST INT4 across sizes.

Kumar et al. (2024) don’t use the same terminology, but their metrics are similar. E.g., in Section 4.3.2 of their paper, they solve for P*, the precision that yields minimal loss at some model size, which is the “Pareto-optimal” precision.

We used a similar setting to Kumar et al. (2024) for the scaling law study, but improved significantly on their findings in terms of optimal precision via a new method: QuEST brings down the “optimal” training precision to 4bit, down from the 7-8bit precision found to be “optimal” for STE by Kumar et al. (2024).

...the sentence on lines 140-142 is true regardless of the smoothness of the loss function

Indeed, we could obtain a bound of $T^2 |S_{small}|$ just by summing over indices $k$ in $S_{small}$. We used smoothness since if $\gamma < 1$ we would get a better bound $\gamma^2 T^2 |S_{small}|$. We thank the reviewer for this note and will simplify the derivation to not use smoothness.