Towards Fully FP8 GEMM LLM Training at Scale

We thank the reviewer for their feedback and hope the concerns will be fully addressed with the following response.

The study lacks necessary ablation experiments | The modification of transformers

We would like to point the reviewer to the Section 3 (after Table 1) as well as Section 4, where we present extensive ablations and a step-by-step transition from OP architecture, that faces early divergence, to FOG-Max variant. We further validate with kurtosis observations (Figure 3) to isolate the components causing instability or outlier amplification. Including Section 5, we have ablated around 8 architectures, covered 3 model sizes (0.4B, 1.5B, 8B) and different data regimes (50BTs, 125BTs, 420BTs). We would also like to quote the reviewer 5MCk: "The paper conducts extensive experiments, including comparative experiments on models of different types and scales" and "The authors provide a wide range of comprehensive evaluation metrics".

To further address the reviewer's concern, we conducted additional experiments, this time starting from Llama architecture and reaching a new variant we call FOG-SwiGLU. In this study, QK norm is introduced after Post-Layernorm, considerably delaying divergence. The delayed divergence confirms the benefits from its entropy regularization effect, avoiding peaky softmax outputs. Finally, Freezing QK ensures convergence while matching the Llama3 BF16 baseline. The following table presents further details about the study:

*Frozen QK-norm gains
**Note that the FOG-SwiGLU ablation scales up model size and data budget compared to the other 390M architectures.

Reviewer also mentioned that “ there is a lack of more possibilities, such as activations outside of GeLU “. We remind that FOG-Max, a key variant among the initially proposed FOG architectures, uses xIELU activation function which belongs to the family of quadratic and learnable activation functions. The above-mentioned additional study, further validates the stability of a 4th variant that simply keeps SwiGLU gated activation function. This demonstrates unprecendented generalization, unlike prior work which is tied to a scaled version of gated activations

Lacked a key explanation for eliminating outliers

The core motivation of eliminating outliers stems from the fact that, because of the limited dynamic range of FP8 formats, the presence of large outlier features increases the risk of under- or over-flowing values. This is motivated in lines 26-32, 89-97, 158-170 as well as the references cited therein [1,2,3,4]. Moreover, our experiments support that mitigating large outliers is critical to enable FP8 training.

• In Figure 3, large activation outliers were present on during cooldown, resulting on divergence of an otherwise stable FP8DPA training run.
• Figure 5 demonstrates large activation outliers prior to the divergence of an FP8DPA run.

Speaking about outliers, this brings the topic of outlier amplification profoundly studied on Llama by the prior work [4], introducing the smooth SwiGLU variant. It shows FP8 outliers are amplified by SwiGLU that can act quadratically late in training (cf. Lines 183-191). One of our work's novelties is allowing FP8DPA training with not only scaled gated activations but a wide range of non-linearities stemming sub-linear in behaviour (GeLU), Gated (SwiGLU), and especially quadratic activation functions (xIELU). The fact that such a quadratic activation function works smoothly with FOG architecture, as evidenced in the Table addressing the previous point in this rebuttal, although it is expected to amplify outliers, is an important signal in favour of FOG's robustness. We aim to make this point, currently raised in Line 183, clearer in the next revision.

[1]: Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, in Advances in Neural Information Processing Systems.
[2]: Mingjie Sun, Xinlei Chen, J Zico Kolter, Zhuang Liu, 2024, Massive Activations in Large Language Models, in First Conference on Language Modeling.
[3]: Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann, 2024, Understanding and Minimising Outlier Features in Neural Network Training, in The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[4]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations.

Additional arguments and experiments

We understand the reviewer seeks more detailed deductive analysis of outlier minimization. Therefore, we provide two additional arguments with theoretical and empirical analysis of negative consequences when such features are present.

Argument 1: Outliers fundamentally lead to less-precise quantization

Let's start with a useful definition. $\tau$-outlier: Given $\mathbf{x} \in \mathbb{R}^d$ and $\sigma = \mathrm{rms}(\mathbf{x})$, element $x$ of $\mathbf{x}$ is a $\tau$-outlier if $|x| \geq \tau \sigma$.

As $\tau$ increases, $x$ becomes a larger outlier ($\sigma$ represents the natural magnitude of $\mathbf{x}$). In practice, $\tau \gg 1$.

Before FP8 quantization, each tensor is scaled with $s(\mathbf{x}) := \mathrm{MaxFP8Value}/\mathrm{absmax}(\mathbf{x})$ to better utilize limited FP8 dynamic range.

Theorem 1

Let $\mathbf{x} \in \mathbb{R}^d$ have $\tau$-outlier $x_j$, and $\mathbf{x}'$ identical to $\mathbf{x}$ except $x_j'$ is $\tau'$-outlier with $\tau' > \tau$.

Then for any subset $T \subseteq \\{1\ldots d\\} \setminus \{j\}$, vector $\mathbf{x}_T'$ will be quantized less accurately than $\mathbf{x}_T$.

In other words, larger outlier values lead to less precise FP8-quantized results.

Proof

Let $r = \mathrm{FP8MaxValue}$, $s = r/\mathrm{absmax}(\mathbf{x})$, and $s' = s(\mathbf{x}')$. Since $\mathrm{absmax}(\mathbf{x}) \geq |x_j| \geq \tau \sigma$, then $s \leq \frac{r}{\tau \sigma}$ (similarly, $s' \leq \frac{r}{\tau' \sigma}$). Let $m = \mathrm{absmax}(\mathbf{x}_T)$ and $m' = \mathrm{absmax}(\mathbf{x}_T')$. Elements of $\mathbf{x}_T$ lie in $[-m, m]$. After scaling, range becomes $[-rm/(\tau \sigma), rm/(\tau \sigma)]$ in $s \mathbf{x}_T$, and $[-rm/(\tau' \sigma), rm/(\tau' \sigma)]$ in $s' \mathbf{x}_T'$. Since $\tau' > \tau$, $[-rm/(\tau' \sigma), rm/(\tau' \sigma)] \subset [-rm/(\tau \sigma), rm/(\tau \sigma)]$, so $\mathbf{x}_T'$ has smaller range. This narrowed range contains fewer n-bit representable numbers, proving the theorem.

The proof applies to any subset $T$, including the set of "typical" values (e.g., 90%-quantile). Theorem 1 guarantees large outliers worsen quantization on 90% of tensor elements.

Empirical confirmation
We measured activation values of Llama and FOG-max 1.5B during mid-training on a micro batch of data, (precisely the second FFN layer's input, before quantization). Observation: Llama presents a 688-outlier while FOG-max shows only a 183-outlier. Using 90%-quantile, we get that 90% of Llama's activation coefficients scale to $[-0.289, 0.289]$ range, while FOG-max allows a much broader range of $[-2.984, 2.984]$.

Argument 2: Outlier Cross-Term and Saturation

Standard architectures contain outliers in weights and activations carrying critical information [1, 2]. During FP8 GEMMs, the informative products outlier x outlier terms typically exceed FP8's range, causing information loss.

Proof: This stems from FP8's limited dynamic range (E4M3: max normal magnitude ±448). Theoretically, multiplications like 447x440; 5x200; 20x25 would yield infinity. Even if the accumulator is of higher precision, such products will likely cause clamping to the same maximum value=448 (saturation) later during casting, erasing magnitude differences. In per-tensor scaling, outliers cause the scaling factor to be small ($s=448/amax$). As un‑scaling in subsequent operations scaling uses the inverse (large) factor, this leads to more outliers emerging downstream: multiplication by $1/s$ then inflates all saturated entries to match the magnitude of the original outliers. This propagation of large values can create new outliers downstream and, combined with the loss of magnitude information, can disrupt training dynamics and in severe cases cause divergence.

Empirically, the FFN final output from previous argument shows: 90% of FOG-max elements scale to $[-11.75, 11.75]$—much larger and better FP8-representable than Llama's $[-0.617, 0.617]$ range.

Finally we hope the reviewer is now convinced that this work's motivation is technically solid and kindly ask for re-evaluation given these theoretical and empirical explanations. We're happy to continue discussion if new concerns arise.

[1] Bondarenko et al., "Outlier Dimensions that Disrupt Transformers are Driven by Frequency," EMNLP, 2021.
[2] Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," arXiv:2208.07339, 2022.

We thank the reviewer for the submitted review and weaknesses highlighted. Please find next a detailed answer for each point raised next.

The benefit of fp8 attention

The main advantage of FP8DPA training in our work is the significant throughput boost compared to existing methods, as shown in Table 4. Moreover, we kindly refer the reviewer to the answers of the next points in the current rebuttal, where we include new experiments comparing other FP8 recipes, and efficiency on longer sequence lengths. We believe these additions further strengthen the advantages of FP8DPA training.

Sequence length beyond 4k

We thank reviewer for this excellent point. First, we note that the sequence length of 4096 adopted in our experiments lies within the standard configurations of current small and large scale language model training setups such as DeepSeek-v3, OLMo2, Qwen2.5, and SmolLM3. This pre-training stage usually takes the largest amount of time, mainly because of the attention's quadratic complexity. For instance, only ~100B out of 11.3T of SmolLM3, and 0.8T out of the nearly 16T of Llama3.1 405B tokens were consumed in the long-context extension training phase. Hence, it matters the most to speed up this compute-consuming moderate-context phase of training.

Nonetheless, long context training phase can still be relatively expensive. Therefore, by safely performing attention computation in FP8, our novel approach benefits the most from context extension. Indeed, attention tends to dominate the computational cost of the forward-backward pass as sequence length increases, which leads to even larger efficiency gains when using FP8DPA compared to previous FP8 approaches. We demonstrate this trend at 8B model scale in the following throughput (tokens/second/gpu) table:

A batch size of 1k was used for all experiments, providing even better boosts than the ones reported in the paper (+40.2% --> +42.6%). Clearly, the gap between the FP8-only approach (Smooth-SwiGLU) and our FP8DPA setting becomes larger as context increases. We note that training with even longer contexts usually relies on strategies like ring attention, which bring back the sequence length processed per GPU to values closer to 16K. We thus limit ourselves to ablations of up to 16K.

To address the concern about FOG's robustness in such long context scenarios, we conducted two additional experiments (FOG-Max FP8DPA; Llama BF16) at 390M scale for 50BTs, using a sequence length of 16384. Results can be summarized as follows: A similarly smooth loss progression with an equivalent final performance, around ~2.28 for FP8DPA FOG and 2.22 for BF16 Llama. Such a small loss gap may not translate to downstream performance, especially considering the possibility of running FOG long-context training for +38.8% more tokens at the same FLOP budget.

More rigorous benchmark is needed

We understand the reviewer's concern about the "easy" benchmarks used in the evaluation. We first draw the attention of the reviewer to Table 4 of the appendix, where a wider set of benchmarks is detailed. While most of these are known to be "early signal" tasks, that is exactly the reason why we chose them:

• The scale of 390M models trained on 50BTs and 1.5B models on 125BTs is not large enough to expect for advanced capabilities to emerge. Therefore, evaluating on "harder" tasks would result in close-to-random scores.
• The selected tasks evaluate a wide arrange of skills (e.g. hellaswag for common-sense inference, OpenBookQA for assessing understanding of different subjects, etc).
• A quick analysis of the scores shows that these benchmarks are far from being saturated by the trained models while exceeding the random performances. Thus, the comparison is fair for the scale.
• Prior work [1] report performance on a similar suite of benchmarks, supporting their relevance in this context.

Lack of clarity in Table 4 | Granular FP8 recipes overhead

The throughput reported in Table 4 for Llama+SmoothSwiGLU indeed utilizes BF16 computation in the attention, as enabling FP8 attention results in an early divergence shown in Figure 6. As explained in the Introduction and furthered detailed in Table 3 of the appendix, we associate the label "FP8" to the setting where attention is in BF16 and label "FP8DPA" the settings were FP8 attention computation is used. Next, we include simple FP8 training with OP architecture and granular FP8 approaches as these were not shown to diverge and remain viable FP8 candidates. We compared the throughput of a set of 8B models with the same batch size (1k) across 8 nodes in the following table.

* NVIDIA's Transformer Engine implementation.

Our approach still yields the highest efficiency. Responding to the concern that "fp8 attn does not bring much actual benefit (efficiency) compared to already working fp8 recipes", we argue, as motivated in the lines 33-35 of the paper and further evidenced by the table above, that using granular scaling factors such as NVIDIA blockwise recipe, provides sub-optimal throughput gains. Such FP8 kernels are already heavily optimized but the overhead is inevitable from such a scaling strategy.

Moreover, while the extra ~4.4% throughput boost compared to the fastest competing FP8 alternative (Llama+SmoothSwiGLU) could be seen as small, the proposed FOG architectures still have several advantages compared to it. Our FOG architecture family admits a several QK regularization functions, and flexible choice of activation functions, generalizing even to MoEs as demonstrated next. Moreover, the benefits of FP8DPA training become more significant as the sequence length grows, as shown previously, making our recipe more valuable during long-context training.

In addition of GeLU and xIELU activations validated in the paper, we additionally experimented with a 1.5B FOG-max FP8DPA variant with SwiGLU activation, which we train for 125B tokens. This new experiment resulted in a successful convergence, similar to the BF16 baseline. Moreover, the choice of GeLU and xIELU activations in the proposed variants was deliberate and further validates that FOG can generalize to other activations. xiELU is a quadratic function and thus amplifies input outliers. Therefore, succesful FP8DPA training highlights FOG's robustness to outliers, and draws new limits of previous work [1], as discussed in lines 183-191. Moreover, the SmoothSwiGLU trick is not enough for stable FP8DPA training. We believe that allowing the use of quadratic activation functions with FP8DPA training is an important novelty of our work that we should make clearer in the next revision.

Limited generality with regard to other architectures

The architecture suite proposed encompass various architectural choices with different normalization and activation function choices, as furthered evidenced in the previous point. While future transformer variants are impossible to predict fully, we provide state-of-the-art results for contemporary architectures and allow for flexibility within our architecture. Moreover, a key contribution of our work highlights the importance of kurtosis and outlier minimization for FP8 training monitoring, bringing a valuable tool to the scientific community to diagnose FP8 acceleration when such future transformer variants are proposed.

While we don't provide extensive MoE results, we expect FOG to generalize well. Aside from the routing mechanism, MoEs follow similar forward passes as dense models. Our key changes, post-norm and QK regularization, don't interfere with MoE-specific components, and new FOG-SwiGLU stable variant further supports compatibility of our appraach. Still, we adapted FOG-flash-390M to an MoE with 2 active experts among 8 (resulting in a 1.8B model). We trained it for 25BTs, following the experimental setup in our paper, with both FP8DPA and BF16 precision. The resulting 1.8B-8E FOG model converged to a loss of 2.477 with BF16, and 2.483 with FP8DPA. Which suggests good generalization of FOG for MoEs under FP8DPA regime.

Finally, we performed throughput measurements on a 41.5B-8E model. As with dense models, our architecture provides the largest boost compared with other approaches.

FP8 optimizer states

In addition to the points discussed in the Limitations section, we ran a new FP8DPA experiment using FOG-flash at 390M scale with FP8 optimizer moments, following the setup in our paper. The model converged successfully to a loss of 2.645, nearly identical to 2.649 obtained with higher-precision moments.

Higher precision for the final projection layer

As mentioned in the Limitations section, we decided to explore this further in a future work. We believe the final projection layer is the most sensitive part of the architecture.

[1]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations.

We thank the reviewer for their time and thoughtful feedback. We are glad they found many aspects of the work promising. Below, we address the questions and concerns raised.

Definition of xIELU

Although we provide the reference work [1] that introduces xIELU in line 186 of the submitted manuscript, we agree that this non-linearity has been proposed relatively recently. And adding its definition to the manuscript would give readers a clearer picture of FOG-MAX variant. We thank the reviewer for pointing this out and will provide a clear definition in the appendix upon acceptance. As we mention xIELU, let's note that this choice was deliberate. It's an inherently quadratic function and thus is an outlier-amplifier. Therefore, choosing an architecture with xIELU while demonstrating that our method still remains very stable highlights FOG's significant robustness to outliers, pointing to the limits of the excellent former work [2], as discussed in paragraph: L183-L191. We believe that the FOG's robustness when using quadratic activation functions is an important strength of our work that we should make clearer in the next revision.

Loss Drop of Llama3 in Figure 7

The loss drop pointed out in Figure 7 is expected, and matches the LR cooldown during the last phase of the widely adopted WSD LR schedule. This phenomenon is positive and allows matching cosine LR schedule's final performance [3]. Note that the drop is consistently observed for all architectures reaching the final LR cooldown phase, such as around 40BT in Figure 2; around 40BT for the 390M model and 100BT for the 1.5B model in Figure 6. For the long FOG-Max-1.5B run trained on over 420BTs, LR cooldown hasn't been applied yet as the only goal was showcasing stability. To address the reviewer's concern, we've pursued this run by conducting an extra 25BTs of LR cooldown, which resulted in an expected loss drop reaching a final value of 2.27, improving upon Llama-1.5B final loss in Figure 7.

Further Explanation of the FP16 baseline + Lack of ablation

We first clarify that BF16 is the standard half-precision format used for training [4]. Although model weights can be stored in FP16, GEMMs are done in BF16 for more dynamic range. In the following we assume the reviewer refers to this widely adopted setting, that we call BF16 training.

"I would expect the authors to provide more comparisons between the two architectures in the experiments": We kindly refer the reviewer to Table 4 of the supplemental materials, where we report a wide variety of downstream performance results following extensive experiments to compare FOG variants against the Llama BF16 baseline. We note that for each task and each FOG variant, downstream performances are provided for both BF16 training and FP8DPA training. We also point to Figure 4 where upstream performance (loss) aligns with task performances, and to Table 4 where we report throughput comparisons.

"I still do not understand why it can match the original FP16 LLaMA3 with pre-norm and SwiGLU": Table 4 shows that all BF16 FOG-variants exhibit similar performances to original BF16 Llama3. Indeed, FOG is designed to preserve both stability and performance when reducing GEMMs precision to FP8, even when FP8 computation of the dot product attention (FP8DPA) is carried out. This is highlighted by the scores reached with FP8DPA training as well as the equivalent final losses. We explain this "equivalent performance with both BF16 and FP8DPA" by the robustness of the architecture as well as the choices of expressive activation functions and beneficial QK-regularization. Further, we note some consistent trends such as FOG-max slightly outperforming other architectures, including baseline. In [1], it has been shown that xIELU outperforms SwiGLU thanks to its linearly increasing gradient for positive inputs and its trainable gradient for negative inputs. QK-RMSNorm and QK-Tanh play a role of entropy regularizers, preventing spiky attention softmax outputs, crucially contributing to FOG's performance and stability. In the following table where we introduce two additional experiments, we re-confirm the stabilizing effect of QK regularization under FP8DPA regime, by the delayed divergence. The starting point of the architecture ablated below is now Llama and QK is introduced after Post-LN.

*Frozen QK-norm gains.
**Note that the FOG-SwiGLU variant test scales up model size and data budget compared to the other 390M architectures.

To conclude, our work shows that FP8 precision is sufficient to reach the long standing half-precision performances without trade-offs, while providing flexible architectural choices (QK-regularizers, sub-linear/quadratic/gated activations) with a training acceleration that scales better with long context than all previous approaches, thanks to the FP8 attention computation.

[1]: Allen Hao Huang, Imanol Schlag, 2025, Deriving Activation Functions Using Integration.
[2]: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, 2025, Scaling FP8 training to trillion-token LLMs, in The Thirteenth International Conference on Learning Representations.
[3]: Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Benallal, Leandro Von Werra, Martin Jaggi, 2024, Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, in The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[4]: Dhiraj Kalamkar, et al., 2019, A Study of BFLOAT16 for Deep Learning Training.

Thank you very much for the rebuttal!

First, based on your responses to Q1 and Q4, I would appreciate it if we could both clarify the definition of xIELU and include the SwiGLU variants in the main experiments, as it can also work under the FP8DPA regime. I understand the xIELU may be more powerful than SwiGLU. However, it is not widely adopted today, so it remains to be seen whether it can scale to larger models. Therefore, I think further results with SwiGLU + FP8DPA with other designs such as QK Norm (becoming common these days) will enhance the quality of this work. People can try your method with fewer component replacements.

Second, based on your explanation, I understand the meaning of Figure 7. However, it is hard for the audience to distinguish between these two training curves (one with annealing and one without). Therefore, I recommend that the authors use the same training configuration in the Figure for the next version.

Moreover, I was wondering whether the authors could provide some comparison or discussion with DeepGEMM, proposed by Deepseek, and MXFP8 on NVIDIA GPUs. I was wondering whether these designs can lead to more stable FP8 training and enable FP8DPA.

Additionally, for the authors' response to other reviewers, I suggest focusing on additional experiments that optimize FP8 states. I was wondering whether the authors could provide more details on this part, including both the precision for momentum and variants, and the througputs after applying FP8 optimization states.

Thank you very much!

We thank the reviewer for their time. However, we believe several of the concerns raised do not accurately reflect the contributions of our work. We address these points below with clarifications and supporting evidence.

Short text scenarios

We thank the reviewer for pointing out to larger context length scenarios. First, let's note that the sequence length of 4096 adopted in our experiments lies within the standard configurations of current small and large scale language model pre-training setups, such as DeepSeek-v3 [1], OLMo2 7B,13B,32B [2], Qwen2.5 0.5B-72B [3], and SmolLM3 3B [4]. This moderate context-length pre-training stage usually takes the largest amount of time, mainly because of the prohibitive (quadratic) cost of attention. For instance, Qwen3 models were trained with sequence length 4k for more than 35T tokens before switching to longer values for a few hundred billion tokens. It took only 100B tokens out of 11.3T tokens for SmolLM3 to perform the final long context expansion phase. Finally Llama3.1-405B trained on long sequences for only 0.8T tokens out of nearly 16T tokens in total. Hence, speeding up the main moderate-context phase of pretraining has the biggest impact on overall training efficiency.

Nonetheless, long context training after the main phase can still be relatively expensive because iterations become considerably slower. Our method benefits from longer contexts, because we perform attention computations in FP8. Indeed, attention tends to dominate the computational cost of the forward-backward pass as sequence length increases, which intuitively yields even larger gaps compared to previous FP8 approaches.

We demonstrate this trend at 8B model scale by benchmarking and report throughput values (in tokens/second/gpu) results in the following table:

A batch size of 1024 was used for all experiments, providing even better efficiency gains than the ones reported in the paper (+40.2% --> +42.6%). Actually, when GBS increases, computation time takes over communication. Clearly, the gap between the FP8-only approach (Smooth-SwiGLU) and our FP8DPA setting becomes larger as context increases. To be more specific, we do not claim higher accelerations with higher context lengths because such settings generally require more model sharding, resulting in lower boosts. But we do show intuitively and empirically that the gap with existing FP8 methods increases with the context length: which is a strength that we'll make sure to highlight in the next revision.

Finally, we performed an additional experiment and ablated Llama BF16 and FOG-max FP8DPA 390M-sized models, for 50B tokens with a much longer sequence length of 16k, using the same configuration as specified in our paper at this scale (Table 1 of the Appendix). The results were consistent with those reported at 4K sequence length: stable loss progression and equivalent final performance. Therefore there are no "serious value problems" with longer sequence length training, but instead the opposite of even stronger efficiency gains.

The volume of data validated in the paper is insufficient

We would like to present the following 3 arguments to answer this point:

• Practical implementations of state-of-the-art models such as DeepSeek-v3-671B scale data to a point that corresponds to roughly 1.2 x the compute-optimal data budget taken from the chinchilla rule of thumb [5]. The factor is around 11x for Llama-3.1-70B and even less for Llama-3.1-405B. Therefore, we still do believe that scaling training of a 1.5B model to 420 billion tokens, which is 14x the compute-optimal, can be considered as a long data regime at this scale. Especially, given the consistency in performance and stability we've demonstrated when scaling model size.
• One of this paper's novelties, is the use of kurtosis to monitor and potentially predict future divergences. In Figure 4, we have provided low kurtosis guarantees for much longer than the above mentioned practical chinchilla factors, as well as a sublinear progression further strengthening the stable data scaling argument.
• Judging from most of the practical implementations, scaling the training of a 1B model to 27T tokens is almost never done. Further, distillation seems to be the go-to recipe in general to obtain cheap but strong small language models (cf. Gemma3, Falcon3).

The paper appears structurally incomplete, and crucially, key appendixes are missing

Thank you for your comment. We would have appreciated more specificity and clarity regarding the "structural incompleteness". There is a misunderstanding, as our appendix was already provided in the standard supplementary material submission as instructed by NeurIPS, and was always available. Regarding the reviewer claims about "missing appendices":

• "Section 5.2: Claims weight decay analysis exists 'later in the Appendix'".
  • We kindly redirect the reviewer to lines 41-46 and validation experiments represented by Table 5 of the supplemental materials uploaded in addition of the main manuscript, where such analysis can be found.
• "Section 5: Promises hyperparameter details 'available in the Appendix'".
  • These details are described in lines 1-6 and clearly represented in Table 1 of the Appendix, present in the supplemental materials uploaded in addition of the main manuscript.

Table 4 Precision Labeling Inconsistency

We appreciate the reviewer for pointing out the "BF6" typo in Table 4. We'll make sure to correct it in the next revision. However, although we understand the reviewer might have been influenced by the misbelief of "missing appendix", we'd still like to raise our concerns about overemphasising over a clear typo. We'd like to point the reviewer to the comment made by reviewer 5MCk: "This paper is well written, logically clear, and fully expresses the motivation and methods".

We thank reviewer r2ET for the feedback. As we addressed all of the concerns, we hope it is now clearer that the paper is technically sound and that no parts are missing. We kindly ask the reviewer to adjust the scores accordingly.

[1]: DeepSeek-AI, et al., 2025, DeepSeek-V3 Technical Report.
[2]: Team OLMo, et al., 2025, 2 OLMo 2 Furious.
[3]: Qwen, et al., 2025, Qwen2.5 Technical Report.
[4]: Elie Bakouch, et al., 2025, SmolLM3: smol, multilingual, long-context reasoner.
[5]: Jordan Hoffmann, et al., 2022, Training Compute-Optimal Large Language Models, in Proceedings of the 36th International Conference on Neural Information Processing Systems.