FP4 All the Way: Fully Quantized Training of Large Language Models

$**Q1a**:$ Although the 7B training shown performs at par with BF16 baseline, it's unclear if FP4 FQT scales to larger models (30B+) or more diverse architectures beyond the 200B token setting.

$**A1a**:$ We continued the training presented in Fig. 6 to 1T tokens, followed by 40B tokens of QAF. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16), showing the ability of the proposed method to work in a larger scale dataset. Then we compare the BF16 baseline with FP4 model on downstream tasks (similar to Table 3), including new tasks as required by the reviewer - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS):

We agree with the reviewer that it will be interesting to see the proposed method on larger models (30B+), however it requires more compute resources than we currently have: just our experiments on a 7B model for 1T tokens took around 30 days on 256 devices.

$**Q1b**:$ Additionally, as recent work [1] has also proposed an end-to-end FP4 training framework, the authors are encouraged to clarify the distinctions and provide insights into how their approach compares or complements this effort.

$**A1b**:$ Please note we already cited this work as [18], and its distinctions from us are explained in lines 103-109 in the "related work" section and Table 2. Specifically, our work is the first work that shows full fp4 training, i.e, quantization to fp4 of the 3 general-matrix-multiplications (GEMMs) (forward, backward, and update - Equations 1,2,3). In contrast, [18] quantized only the forward GEMM, i.e can accelerate only $\sim 33$% of the GEMMs, which dramatically reduces the potential acceleration. Moreover, they trained only up to 100B tokens, which is an under-trained regime; we extend it to a 10x larger dataset. Please see Table 2 for a full comparison with [18].

$**Q2**:$ Due to the use of shared scaling across blocks, stochastic rounding, and dynamic precision switching, this could likely incurs non-trivial computational cost. Providing a quantitative analysis of these costs would offer clearer insight into the trade-offs between efficiency and precision stability.

$**A2**:$ Shared scaling across blocks is supported natively in hardware in Blackwell architecture without any acceleration cost, i.e it allows a 2x acceleration in comparison to FP8 and 4x in comparison with BF16. We expect additional AI accelerators also to support it natively without additional cost.

Stochastic rounding is also supported natively in the latest Nvidia's CUBLASS (v8.8) or in Gaudi2, without any significant overhead.

Notice we do not use dynamic precision switching. We only increase once the precision of the backward (Eq.2) and update (Eq.3) GEMMs when starting the QAF process. In order to calculate the total cost, we can calculate the average bits operations during all process (pretraining + QAF), as we detail next. Recall we extended the pretraining phase to 1T tokens, followed by a 40B-token QAF phase. During pretraining all three GEMMs operate in FP4, while during QAF only one of the three GEMMs uses FP4 (with the remaining two in BF16). The resulting average-bits calculation is:$[(1000\times3\times4) + (40\times1\times4) + (40\times2\times16)] / [1040\times3] = 4.3$ bits. We will include this result in the camera-ready version.

$**Q3**:$ The authors are also encouraged to include comparisons to all-nearest or all-stochastic rounding configurations. This would be beneficial in understanding the relative comparison of different schemes for stable training.

$**A3**:$ In Fig. 3 we show the comparison to all-nearest (marked as "full RtN"). Moreover, we conducted an additional experiment similar to Fig. 3, where we independently applied RtN to one of the six elements while the remaining elements used SR. The resulting losses are as follows: Full SR: 3.022; Weights Forward RtN: 3.002 ; Activations Forward RtN: 3 ; Weights Backward RtN: 3.021 ; Activations Backward RtN: 3.032 ; Gradient Update RtN: 3.041 ; Gradient Backward RtN: 3.039.

This experiment highlights the effectiveness of the proposed regime, as described in Equations 4, 5, and 6, where SR is applied only to neural gradients during the update and backward GEMMs, and to activations during the update GEMM. We will include this experiment in the camera-ready version.

$**Q4**:$ How does the FP4-trained model compare to FP16 baseline on downstream tasks (e.g., QA, summarization, code generation)? Does full quantization affect generalization across domains or modalities? The authors are encouraged to include a few difficult baselines to demonstrate the at-par performance wrt BF16 baseline.

$**A4**:$ Please see the response A1a, where we show the evaluation on additional tasks for summarization (xsum), QA (trivial qa) and code generation (mbpp) as required by the reviewer.

$**Q5**:$ How sensitive is the training stability to the $\sqrt{3}$ gradient-to-noise threshold? Also, can the authors elaborate on how and when precision switching is triggered in practice?

$**A5**:$ The $\sqrt{3}$ gradient-to-noise threshold serves as a useful indicator for assessing the effectiveness of quantized training and identifying when it is no longer beneficial to continue training in low precision. Figure 5 illustrates a scenario where this threshold is crossed, suggesting that full low-precision training should be stopped. In contrast, for the experiment shown in Figure 6, the gradient-to-noise ratio remains above the threshold. This was a significant positive result that justified our decision to allocate the resources to continue training in 4-bit all the way to 1T tokens. It also demonstrates that with the optimal format (NVFP4) and our split rounding strategy, large-scale FP4 training can be successfully completed without even encountering the stagnation point that our theory predicts.

$**Q6**:$ How does the method behave under varying batch sizes? Smaller batches often lead to noisier gradients, does this make FP4 more prone to training collapse? Also, how sensitive is the FP4 training pipeline to initialization schemes?

$**A6**:$ Following the reviewer’s request, we run 3 runs with different micro-batch sizes (1K,10K,20K) in a 125M model trained over 30B tokens using the proposed scheme on Gaudi2 devices, we get similar training losses: 3.027, 3.025, 3.03 We believe that the use of per-block quantization (in contrast to per-tensor quantization) reduces the effect of batch size, mentioned by the reviewer. Regarding different initializations, please refer to response A7 below, where we tested multiple seeds (including different initializations) and did not observe any convergence issues.

$**Q7**:$ Given that the authors use of stochastic rounding for backward pass, how reproducible are the training runs across seeds or hardware environments? Are there significant variance in convergence / final performance for different random seeds?

$**A7**:$ Following the reviewer’s request, we conducted 5 runs with different seeds on a 125M model trained over 30B tokens using the proposed scheme on Gaudi2 devices. The resulting losses are: 3.03, 3.027, 3.025, 3.027,3.029 yield a loss standard deviation of 0.001, which is not significant. Moreover, we ran the same experiment on an A100 GPU and obtained an equivalent loss of 3.024. As the reviewer can see, we did not observe any convergence issues across different seeds or hardware platforms. We plan to include these ablation studies in the camera-ready version.

$**Q8**:$ As a suggestion, the authors are encouraged to further consider robustness related limitations, especially the potential vulnerability of their approach to noisy gradients or its applicability in vision and multimodal settings, where different modalities may exhibit different gradient distributions or magnitudes that could affect the stability of FP4 training.

$**A8**:$ Thank you for this suggestion. We will add it to the limitations section, explicitly emphasizing that our experiments focus on language tasks, while vision tasks are left for future work.

$**General comment:**$ In response to the reviewer's questions, we would like to clarify a potential misunderstanding regarding the different training phases: (1) Long Pretraining: All three general matrix multiplications (GEMMs) described in Equations 1, 2, and 3 are quantized to FP4. (2) Short QAF: The forward GEMM (Eq.1) is quantized to FP4, while the backward (Eq.2 ) and update (Eq.3) steps are performed in BF16.

This means the final model contain FP4 weights and can run inference directly in the FP4 datatype without requiring any additional processes (such as PTQ). We plan to improve and clarify this explanation in the camera-ready version.

$**Q1**:$ "Lacking an isolated round-to-nearest impact per matrix, similar to Fig. 3..."

$**A1**:$ In response to the reviewer’s request, we conducted an additional experiment similar to Fig. 3, where we independently applied RtN to one of the six elements while the remaining elements used SR. The resulting training losses are as follows: Full SR: 3.022; Weights Fwd RtN: 3.002 ; Activations Fwd RtN: 3 ; Weights Bwd RtN: 3.021 ; Activations Bwd RtN: 3.032 ; Gradient Update RtN: 3.041 ; Gradient Backward RtN: 3.039.

This experiment highlights the effectiveness of the proposed regime, as described in Equations 4, 5, and 6, where SR is applied only to neural gradients during the update and backward GEMMs, and to activations during the update GEMM. We will include this experiment in the camera-ready version

$**Q2**:$ "While the theoretical derivation in 4.1 is intuitive ..."

$**A2**:$ We appreciate the reviewer’s comment and agree with their observation. Our analysis identifies a threshold for training ineffectiveness (i.e., when progress stalls) rather than proving instability. This occurs when the gradient signal becomes too weak to overcome the quantization noise, causing the optimization process to plateau. Because our goal is to define this practical point of failure, the derivation in Section 4.1 is intended as a local metric to guide training decisions, not as a global proof of convergence. Consequently, we do not claim to bound the absolute change in loss or present a full convergence guarantee. As shown in Figure 5, this interpretation is supported empirically: when the metric falls below the proposed threshold, continuing training in low precision is no longer effective, and an intervention such as increasing the numerical precision is required to resume progress.

$**Q3**:$ "The 220 B-token..."

$**A3**:$ We have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline with FP4 model on downstream tasks (similar to Table 3), including new tasks - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS):

As the reviewer can see, the proposed method also generalizes for larger datasets. We will upload these results in the camera-ready version.

$**Q4**:$ "No measured total BOPS, throughput, or average bit-width across FQT + QAF..."

$**A4**:$ As stated in A3, we extend the pretraining phase to 1T tokens, followed by a 40B-token QAF phase. As noted in the general comments, during pretraining all three GEMMs operate in FP4, while during QAF only one of the three GEMMs uses FP4 (with the remaining two in BF16). The resulting average bit calculation is:$[(1000\times3\times4) + (40\times1\times4) + (40\times2\times16)] / [1040\times3] = 4.3$ bits. We will include this result in the camera-ready version. Note the 5.2 bits result mentioned by the reviewer assumes fully 16bit QAF, which is wrong (see general comment above).

$**Q5**:$ "Experiments rely on Gaudi2 simulation..."

$**A5**:$ We agree with the reviewer that this is one of the limitations of our paper, as wrote in the limitation section. It is worth noting that even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators. As stated in the limitations section, based on prior time-to-train improvements from BF16 to FP8 (35-40 % time-to-train acceleration) and the additional 2× matrix multiplication speedup from FP8 to FP4, we provide a rough estimate of an 85% reduction in training time.

$**Q6**:$ "Stochastic-rounding implementation is unspecified..."

$**A6**:$ The paper includes in the abstract anonymous full code reference to reproduce the experiments, specifically notice "experimental/fp4.py" line 6 which includes the stochastic rounding implementation. In general, the implementation includes taking the full precision value, adding random uniform noise with support equal to the bin size, then rounding to nearest quantization value: $Q(x) = \mathrm{round} (x+ U)$, where $x$ is full precision value, $U\sim [-0.5\delta,0.5\delta]$ is random uniform noise and $\delta$ is the local spacing between two representable FP4 values around x.

$**Q7**:$ "Related-work survey omits two sub-FP8 methods..."

$**A7**:$ Thank you for these references, we will add them in the camera ready version. Notice that, neither QuEST nor AdaBin represents fully quantized training. In these works, quantization is applied only to the forward general matrix multiplication (Equation 1), while the backward (Equation 2) and update (Equation 3) steps are kept in high precision. This significantly reduces the potential acceleration compared to our approach. As explicitly stated in QuEST [1], page 4: "although the backward computation is performed w.r.t. the quantized weights and activations, the multiplications and gradient operands are performed in standard 16-bit precision.".

$**Q8**:$ "Training hyperparameters are incomplete..."

$**A8**:$ We maintain all hyperparameters consistent with Llama2 paper, which includes the use AdamW optimizer with $\beta_1=0.9$, $\beta_2=0.95$. We use cosine learning rate schedule, with 2000 steps of warmup, peak learning rate of $3\times10^{-4}$ and decay to 0.1 of the peak lr. We use global batch-size of 4M tokens. All these details are available in the supplied code and will be added in the camera-ready version.

$**Q9**:$ "All results are based on a single seed..."

$**A9**:$ Table 3 includes running 0-shot downstream tasks on the final FP4 model and the BF16 baseline, which is a deterministic process (No stochastic process during inference). A possible way to get confidence bars is run all training again with different seed, but it is not feasible (full 1T tokens training take 30 days). In order to get an approximate confidence bar, we run the inference process presented in table 3 with 3 close checkpoints and calculate the std of these runs:

$**Q10**:$ "No empirical study on how long QAF must run..."

$**A10**:$ Great question! We conducted an ablation study to determine how long QAF must run, i.e., how many tokens are required to reach the same loss as the BF16 baseline. Starting from three different checkpoints, we obtained the following results:

• From 200B tokens – QAF requires 20B tokens (10%).
• From 500B tokens – QAF requires 28B tokens (5.6%).
• From 1T tokens – QAF requires 40B tokens (4%).

As the reviewer can observe, for larger and real-world datasets, the QAF ratio decreases. Additionally, we experimented with different peak learning rates for QAF and concluded that it should be similar to the training learning rate—using a larger LR causes divergence, while a smaller LR prevents QAF from converging to the baseline. We plan to include this ablation study in the camera-ready version.

$**Q11**:$ "No exploration of combining the method with PTQ for additional compression..."

$**A11**:$ SmoothQuant and GPTQ are post-training quantization (PTQ) methods designed to take models trained in high precision and quantize them to lower precision (typically INT4) after training, while attempting to minimize quantization error for inference. As noted in the "general comment" above, our proposed pretraining + QAF approach produces a model directly in 4-bit precision (FP4), with no additional steps required to enable 4-bit inference. Therefore, we believe PTQ methods are not necessary in our proposed framework. We decided to focus on FP4 and not in INT4 since it is supported natively in modern hardware, such as Blackwell.

$**Q12**:$ "Minor text issue: .."

A12: Thank you, we will fix them in the camera-ready version

$**Q1**:$ "As mentioned by the authors, the lack of validation in FP4 hardware is a key limitation of this paper."

$**A1**:$ We agree with the reviewer that this is one of the limitations of our paper. It is worth noting that, even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators.

$**Q2**:$ "More evaluation results such as GPQA, IFEveal ( Open LLM Leaderboard 2 ), EvalPlus will add more insight on how well the quantized model is trained."

$**A2**:$ We have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline with FP4 model on downstream tasks (similar to Table 3), including new tasks as required by the reviewer - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS):

$**Q3**:$ "Section 4.1 can include quantized SGD convergence analysis to make sure and understand the conditions of quantized SGD convergence. Is it possible to derive some mathematical guarantee that quantized SGD converges given that higher precision SGD converges? Example: refer to Section 4 of [2]"

$**A3**:$ Thank you for this interesting reference. The global convergence guarantee in [2] and our local analysis in Section 4.1 work together. The theory in [2] predicts that training will eventually stall in a noise-dependent neighborhood, while our diagnostic tool helps identify in real-time when that stalling has begun.

$**Q4**:$ "Did authors use any sort of Maximal Update Parametrization (muP)? What was the effect of quantization on that? [1]"

$**A4**:$ We thank the reviewer for this valuable suggestion. In our work, the primary goal was to demonstrate that full FP4 training can achieve convergence with results on par with the BF16 baseline, using the same hyperparameters. This means we did not explore techniques such as muP, but we consider this an interesting direction for future work. Notice, as mentioned in A2, the main experiment (1T tokens) requires approximately 30 days on 256 devices.

$**Q1**:$ "It would be great to evaluate your methods across different model sizes or architectures"

$**A1**:$ In the camera-ready version, we will include additional experiments involving smaller models with 125M and 350M parameters, achieved final on-par training loss with the BF16 baseline (125M: 2.922(BF16) Vs 2.918 (FP4). 350M: 2.67 (BF16) Vs 2.673 (FP4)). Furthermore, we have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline with FP4 model on downstream tasks (similar to Table 3), including new tasks - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS):

The Scaling to larger model sizes is not feasible with our current resources. Our experiments are based on the LLaMA 2 architecture, which is widely adopted in most LLMs.

$**Q2**:$ "A key limitation is that the proposed FP4 training is only simulated on Gaudi2 hardware, which lacks native FP4 support. As such, the actual speed and memory benefits remain speculative. It would be great if you could provide a solid prediction for the cost reduction."

$**A2**:$ We agree with the reviewer that this is one of the limitations of our paper (as we wrote in our limitation section). It is worth noting that even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators. As stated in the limitations section, based on prior time-to-train improvements from BF16 to FP8 (35-40 % time-to-train acceleration) and the additional 2× matrix multiplication speedup from FP8 to FP4, we provide a rough estimate of an 85% reduction in training time.

$**Q3**:$ "The paper lacks qualitative or in-depth comparison with other recent FP4 training approaches. The baseline is only BF16."

$**A3**:$ Our paper is the first and only work to demonstrate full FP4 training, where all three general matrix multiplications (GEMMs) described in Equations 1, 2, and 3 are quantized. Therefore, there is no prior work that allows for a fair comparison. The most related studies, summarized in Table 2, apply only partial quantization (offering less potential acceleration) and are conducted on smaller datasets.

$**Q4**:$ "Details of the training configuration—such as optimizer type, learning rate schedule, batch size, and initialization—are only briefly mentioned and could be more thoroughly documented for full reproducibility."

$**A4**:$ We maintain all hyperparameters consistent with Llama2 paper, which includes the use AdamW optimizer with $\beta_1=0.9$, $\beta_2=0.95$. We use cosine learning rate schedule, with 2000 steps of warmup, peak learning rate of $3\times10^{-4}$ and decay to 0.1 of the peak lr. We use global batch-size of 4M tokens. All these details are available in the supplied code and will be added in the camera-ready version.

$**Q5**:$ "Could the authors clarify the scope and applicability of the proposed FP4 training method? Specifically, are there particular model types, sizes, or tasks for which this approach is especially suitable or advantageous? Or do the authors believe it can generalize effectively to any LLM training scenario?"

$**A5**:$ The primary applicability of the proposed FP4 training is to accelerate the pretraining phase, which is the most computationally expensive step and becomes increasingly demanding with larger datasets and models. Our work focuses on the LLaMA architecture, which is one of the most widely used architectures in modern LLMs, and we believe our approach can be effectively extended to most LLM models. While vision transformers were not the focus of this work, their similar architectural structure suggests that our method could also provide acceleration in that domain.