Quartet: Native FP4 Training Can Be Optimal for Large Language Models

We thank the reviewer for providing an extensive evaluation of our work and provide the requested clarifications:

Can you clarify how you arrived at the scaling law in section 4.1?

Conceptually, we proposed this form to unify the laws proposed for forward-only compression by Kumar et al. [4], and the theoretical analysis of training with noisy gradients of e.g. QSGD [2], which suggest a separation of the impact of low precision on the forward and backward passes, in the parameter and data terms, respectively. Practically, the ablation for the general form is presented in Appendix A.2 in the supplementary material. In short, the A,B,E,$\alpha$,$\beta$,$\gamma$ form had by far the lowest fit error and was originally tested (by Busbridge et al.[1]) for models up to 7B parameters. This verified that it’s accurate and robust. The inclusion of eff$_N$ models fundamental reduction in capability, modeled similarly in numerous works cited around line 193 of our submission. On the contrary, eff$_D$ implies that the quantized gradient estimation allows for (albeit slower) convergence to the same loss, which aligns with the theoretical analysis of noisy gradient training [2], and the closing of the performance gap on Figure 1 (a).

Can you clarify section 4.2, especially from lines 233 to the effective loss formula?

Assuming baseline FP8 training and inference, parameter count $N_{max}$ corresponds 1 to 1 to a certain inference cost (latency). Additionally, under fixed model size, training compute can be expressed in training corpus size $D_{max}$ (compute ~ $N_{max} \cdot D_{max}$) or data saturation ratio $D_{max}/N_{max}$. Combined, these two quantities constitute a bias in the model cost functional space. Assuming certain inference and training speedups (spfw, sptr) for a certain training scheme and model sizes, various schemes can be mapped into that space for iso-compute comparisons. The effective loss formula presents cost-normalized parameter count and corpus size. Figure 1 (b, c) presents the regions in that space where certain precisions yield lower loss.

Why was LSQ omitted in Table 2?

LSQ was shown by Panferov et al.[3] to consistently yield higher loss and ~20% lower eff$_N$ than QuEST. We chose the more modern and better performing method as the candidate. Additionally, LSQ can’t be directly applied to backward pass quantization as it has learnable parameters of its own. We will clarify this.

The NVIDIA 5090 GPU is a slightly different compute capability to B200 and GB200 cards. Are there any differences here in terms of the FP4 capabilities?

Yes. According to the Blackwell data sheet [4], we have that:

• CC=120 (RTX 5090) compared to CC=100 (B200) has a higher relative speedup of FP4 compared to FP8 (4x in theory vs 2x in theory).
• CC=120 (RTX 5090) compared to CC=100 (B200) doesn’t offer a stochastic rounding instruction.

The first fact means that FP4 would have a potentially lower performance “ceiling” on the B200 relative to the 5090. These aspects were not well-documented at the time of submission. We’ll make sure to include them in the discussion. We are currently working on a B200 version of our kernels.

As the paper is focused on using a specific accelerator architecture (NVIDIA Blackwell) why not focus on NVFP4?

We chose MXFP4 specifically to not focus exclusively on Blackwell: MXFP4, being the standard OCP Microscaling Format, will see wider adoption among AI accelerator hardware. For instance, it will be supported on AMD hardware as well, whereas NVFP4 will (most likely) not be supported.

At the same time, our kernel design does apply to NVFP as well. Specifically, we have already extended our forward kernels to NVFP4 (see response to reviewer dMMD) and plan to do so for the backward pass as well.

In line 387, how are the baselines implemented? For example, are they both using stochastic rounding? Is this actually using a framework like Megatron or Deepspeed?

The baselines are implemented as either custom GEMM CUDA kernels for FP8 or basic torch.nn.functional.linear for BF16. For hardware support reasons discussed above, RTN instructions were used instead of stochastic rounding. The speedups are reported for quantized forward and backward passes over linear layers that would constitute models of certain size, effectively omitting non-linear operations and optimization steps from the comparison. They are expected to have negligible impact for larger models.

Were any loss spikes present during training?

We observed no spikes in either loss or gradient norm. Figure 3 (c) shows a sample of the training dynamics.

Figure 1: Isn’t the scaling law fit dependent on the training method used for those precisions? Why isn’t this specified?

We acknowledge this question and will clarify. The parameter and data efficiencies eff$_N$ and eff$_D$ indeed depend not only on the precision but generally the quantization scheme (method+precision). That’s why it can be used to both compare different methods and different precisions for the same method. We’ll adjust the wording to make it clearer.

[1] https://arxiv.org/abs/2502.08606 [2] https://arxiv.org/abs/1610.02132 [3] https://arxiv.org/abs/2502.05003 [4] https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf

We are grateful to the reviewer for their work. To address their concerns, we highlight that we performed quantization of linear operation in transformers, retaining attention operations in high (BF16) precision. More specifically, we isolated the quantization logic inside the linear layers, retaining all the remaining operations in the architecture in their original form.

This was motivated since:

• Linear layers’ are the most computationally expensive part of LLM training and inference for short-to-medium context training and inference.
• Work on low-precision attention and efficient long-context modeling is orthogonal to linear layer speedups. The training sequence length of 512 tokens that we utilized and the subsequent omission of the attention operation from individual latency benchmarks reflects this choice.

We acknowledge this limitation of our work, and we will include it in the discussion, but we emphasize that we consider the effect of attention quantization and its impact on long-context training an orthogonal research direction. (In fact, since submission time, orthogonal work on FP4 attention was posted independently [5].)

Regarding architectural verification, we note that our setup closely follows experimental design of prior work on precision scaling laws, notably Kumar et al. [1], Panferov et al. [2], Tseng et al. [3] and Busbridge et al. [4]. All these references focus on dense (non-MoE) Llama-like transformers.

We emphasize that the proposed principles for scaling-law-based and iso-compute comparisons can be directly applied to other model architectures, such as MoE, but that would require designing, verifying and fitting of novel scaling law forms. As such, we leave this for future work.

[1] https://arxiv.org/abs/2411.04330 [2] https://arxiv.org/abs/2502.05003 [3] https://arxiv.org/abs/2502.20586 [4] https://arxiv.org/abs/2502.08606 [5] https://arxiv.org/abs/2505.11594

We would like to clarify the use of FP8 as a baseline in this paper:

• From the accuracy perspective, we reported the BF16 results as an upper bound on FP8 accuracy whenever we needed to compare against FP8. This is a more-than-fair comparison, since FP8 methods do not outperform BF16 precision training, while some methods were shown to match it. We thus believe that this is a fair comparison against the best accuracy results FP8 training could possibly offer.

• From the speedup perspective, our intention was to compare against an ideal FP8 baseline method that only performed FP8 cast + GEMM, which would ignore any overheads due to error mitigation and would represent the fastest these FP8 kernels can be.

However, after the submission deadline, we have realized that our evaluation induced a small but unnecessary performance overhead in the FP8 benchmarking relative to the ideal version described above. After repeating all the runs for the FP8 baseline, the impact of removing this overhead is the following:

• On the smaller (800M) model, the ideal FP8 version reaches 20% higher inference throughput than in our submission, and 25% higher training throughput. For instance, our inference speedup relative to FP8 for this model size is reduced 1.9x to 1.6x.
• On the larger (52B) model, the ideal FP8 version reaches the same inference throughput as in our submission, and <= 10% higher training throughput. Our end-to-end speedup stays roughly the same at this model size.

We apologize for this error, but stress that the performance difference is small enough that it does not significantly affect our FP4 optimality claims: in particular, the optimal training precision for the Llama and Qwen models would still be FP4, whereas the Gemma3 model remains borderline.

To sum up, our FP8 baseline aimed to upper bound both the accuracy and the fastest possible performance of FP8 training, representing the performance of an ideal FP8 training method.

We thank the reviewer for their perspective, apologize for the error, and promise to add a more thorough explanation of our comparison setup and its implications.

Do you have data points for Table 3…

All the data points and the code to produce all the figures, including Figure 1 (a) that shows the data you requested, are present in ./notebooks/plots.ipynb in the supplementary materials.

We thank the reviewer for constructive feedback and address their questions in their original order.

Only C4 perplexity is reported…

We note that we determine optimality based on pair-wise evaluation loss comparisons. That means that as long as downstream (e.g. zero-shot) performance is consistent with loss, the comparisons remain valid. To demonstrate this consistency, we present 0-shot evaluations of the trained model in the table below:

One can see that the zero-shot results show that the models provide non-random performance, and are consistent with the evaluation loss across the training methods.

Regarding the PMA proxy limits…

The effect can be observed from two sides:

1. The statistical effect of magnitude alignment.
2. The fact that RTN is better for shorter training and SR - for longer.

The first effect is expected to persist regardless of model size as it is a statistical property of quantized matrix multiplication during the backpropagation. The latter can be observed in the training runs of Tseng et al. [1] (see their Figure 5): one can observe that RTN converges faster in the beginning, but is overtaken by SR during the later stages of the training. As such, we claim that PMA effects will persist at that scale.

How would Equation (1) break if quantization error interacts non-linearly with model width or depth?

Equation (1) models the loss for a particular training setup, determined, among everything else, by model hyper-parameter (depth, width, LR) scaling with parameter-count. In our experiments, we scaled model depth and width proportionally, leading to depth ~ width ~ N^(⅓). One can notice, that if quantization had separate multiplicative effects on depth and width, it could still be represented as a multiplicative factor on N. To separate these effects, one would need to construct a scaling law that would decouple depth from width and accurately model the effect of quantization as a function of these two parameters. That could possibly lead to an alternative depth-to-width strategy in the presence of quantization. That would constitute an interesting future research direction.

The implementation of Quartet is specialized for the MXFP4 format.

We chose the MXFP4 format as it is the standard OCP Microscaling Format, with broader possible adoption, since it may also be supported by AMD devices. However, our kernel design is compatible with NVFP as well, and should perform similarly in terms of performance, and slightly better in terms of accuracy (since the scale quantization is more flexible in NVFP). We are working on an NVFP version of Quartet.

Although NVFP4 could provide slightly higher accuracy due to smaller group sizes and finer scales, we expect it to perform similarly to MXFP4 due to similar distributional properties of the underlying FP4 format that were shown to be connected to efficiencies in Table 2. Additionally, limited range of the FP8 shared exponents of NVFP4 could create problems for the backward pass with small gradient norm, requiring additional scaling techniques to prevent underflows.

Table 3 shows that the LSS-INT4 baseline becomes unstable…

In our experiments, this baseline doesn’t simply diverge, but rather produces NaN iterations out of nowhere and without prior loss spikes. Its performance being close to Quartet while stable and the use of an unbiased backward pass indicate good alignment. We theorize that the lack of Hadamard preprocessing on the gradient outliers in the backward pass is what’s leading to overflows for LSS-INT4. Determining the exact cause would require extensive debugging of their implementation, which is unfortunately beyond the scope of our work.

Great speedup results in (Figure 3)...

From our benchmarks, quantization-related operations constitute 20-40% of latency depending on the batch size and layer shape. We provide the breakdown for certain matrix shapes in “A.3 Performance breakdown” in Appendix.pdf in supplementary materials. Additionally, see response to reviewer PgVh for slightly adjusted performance measurements.

… providing a higher-level Triton implementation …

We do provide a full reference Triton pseudo-quantization implementation in ./notebooks/benchmark_mxfp4.ipynb in supplementary materials. In fact, we used it for some of the training runs because they were performed on Hopper-generation machines without hardware FP4 support.

[1] https://arxiv.org/abs/2502.20586