Scaling Laws for Floating–Point Quantization Training

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns, and we would be grateful if you could consider increasing the overall recommendation of our work.

Q1: Extension to larger models.

A1: Thanks for the suggestion. We agree with the reviewer that experiments on larger models could further verify the effectiveness of our scaling law. We attempt to answer from the following aspects:

(1) The scaling law exploration of LLMs is essential but extremely expensive in both GPUs and running time. To thoroughly explore the relationships between the loss and different floating-point quantization training factors (e.g., N, D, E, M, B), we have trained 366 models with different settings (model sizes from 41M to 679M) to draw our scaling laws, and successfully validate them on 1.2B models with 8 different settings in Fig. 4. The adopted model sizes are comparable with those in other scaling law works (e.g., Scaling law for precision [1], one of the most related work, adopts model sizes up to 1.7B parameter for validation).

(2) We conduct several additional experiments with model sizes larger than 1.2B. Precisely, we successfully predict the loss of 7B and 70B LLMs (with different settings) based on our scaling law. The detailed model setting, actual loss and predicted loss are shown as follows:

(The results of 1.2B models have already been included in Fig. 4 of our paper)

The actual losses of 7B/70B models are very close to the theoretical value calculated by our scaling law. Therefore, we could confidently claim that our scaling law could extend beyond larger model sizes. These additional results will also be given in the revision.

Q2: Do the scaling laws reduce to existing integer training scaling laws when e = 0?

A2: We thank the reviewer for this insight. Mathematically, our scaling laws approximately reduce to integer quantization formulations when e=0. However, there exist fundamental differences in hardware implementation between integer (fixed-point) and floating-point arithmetic: integer operations rely on dedicated fixed-point units and two’s complement representation, while floating-point architectures require separate processing of sign, exponent, and mantissa bits through distinct computational mechanisms. Such hardware-level disparities suggest that theoretical simplifications may not directly translate to empirical training scenarios. Our current work focuses on floating-point quantization scaling laws ($e \neq 0$), while the formal alignment of the special e=0 integer case with existing theories will be explored as a future direction. This requires joint optimization analysis incorporating hardware instruction sets and numerical representation properties for rigorous empirical validation. We will add this discussion in our revision.

Q3: Would it be more reasonable to fit a curve to SNR or datatype distortion instead of E and M separately? How much of this is dependent on activation and weight distributions? Would some PTQ and QAT papers with random orthogonal transformations change the scaling law?

A3: (1) This is a valuable suggestion. As shown in Eq. 37 of [1], using the Signal-to-Quantization-Noise Ratio (SQNR):

\frac{ \mathbb{E}[\mathbf{W}^2]\mathbb{E}[\mathbf{X}^2] }{ \mathbb{E}[(\mathbf{W}\mathbf{X} - \mathcal{Q}(\mathbf{W})\mathcal{Q}(\mathbf{X}))^2] } \right)$$ for GEMM operations could unify the effects of exponent (E) and mantissa (M) precision while accounting for input distribution modifications like random orthogonal transformations. However, SQNR calculation inherently depends on tensor distributions, which may limit its practical applicability. Therefore, we directly select the raw exponent and mantissa bits rather than the SNR as essential factors in our scaling law for more precise prediction ability. (2) Notably, our theory explicitly models dataset size (D) as a variable, and since weight/activation distributions evolve during training, we believe that our formulation exhibits partial robustness to such distributional shifts. The interaction between PTQ/QAT methods and scaling laws remains an open question requiring systematic analysis of how orthogonal transformations alter quantization noise dynamics. We will add our discussions in revision. ## Q4: Format and references. A4: Thanks for your suggestion. We will fix them in revision. ## References: [1] Kuzmin, Andrey et al. FP8 Quantization: The Power of the Exponent.

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation of our work.

Q1: Novelty. Several works have already emphasized the importance of exponent bits.

A1: (1) The central contribution of this work is to build a scaling law for floating-point quantization training, which quantitatively models the relationships between the loss L and several essential factors, e.g., data size D, model size N, exponent E, mantissa M, and block size of scaling factors B, by providing a joint scaling law formation in Eq. 1. We are the first to give this scaling law. We highlight that the discussions on the importance of exponent bit and mantissa bit are just one part of our multiple contributions, and the novelty of our work still holds.

(2) Although there are some efforts that have explored the importance of exponent bit while FP quantization (e.g., FP8 Quantization: The Power of the Exponent), our efforts and discussions on exponent and mantissa within the scaling law are still novel and valuable: (a) Different from existing works, our work firstly provides a scaling law that could quantitatively predict the model loss via D, N, E, M, B, rather than simply focusing on the importance of E and M. Our work enables accurate loss prediction of LMs in practice, verifying the importance from a quantitative aspect. (b) Besides the quantitative marginal effects of exponent and mantissa, we further explore the quantitative joint effects of D, N, E, M, B on model loss.

(3) Besides the novel findings on E and M, we also discover several valuable observations and insights with extensive experiments that could facilitate future low-precision LLM training in the last paragraph of Sec. 1.

We will add this discussion and related works in our revision.

Q2: Evaluation setting. The authors don’t provide experiments with benchmark datasets.

A2: (1) Thanks for the suggestion. This work concentrates on providing a scaling law for FP quantization training. Following previous works on scaling laws [1,2] (including [1] for INT quantization), we conduct extensive experiments with different precision settings (366 models) to build the basic scaling law formation of various D/N/E/M/B, and validate its effectiveness on larger models. In Fig. 4, we have clearly shown the good fitting results of our scaling law. The validation points (1.2B models, noted as yellow stars) in the bottom left corner fit well for our scaling law, implying that the scaling law could extend to larger models (the fitting results of 7B/70B models are also satisfactory given in rebuttal). We DO NOT draw conclusions based solely on predicted loss value, but rely on the fitting results of predicted losses and actual losses on real data.

(2) The goal of scaling law is to predict the loss based on essential factors. Therefore, it is natural and best to evaluate the correctness based on the differences between actual losses and predicted losses given by our scaling law. [1] is the most related work that also adopts the same evaluation metric fitting results for validation and does not evaluate on downstream benchmarks. We also clarify that the loss is a practical and accurate indicator that provides an overall evaluation of LMs' capabilities widely used in the real world, which is strongly correlated to the overall performance on downstream tasks. We will add this discussion in our revision.

Q3: How are scaling law equation constants determined?

A3: We have introduced in Sec. 3 and 4.1 that we first conduct experiments to reveal the marginal effects, and then design the joint scaling law. The formations are designed based on empirical insights and the distributions of losses in 366 experiments. Next, we fit and validate the formation via LM settings (D/N/E/M/B) and the corresponding losses to obtain the constants in our scaling laws. These constants are learned from practical losses, not theoretical derivation, similar to other works [1,2].

Q4: Whether optimized bit settings have a similar pattern in Fig. 6?

A4: As stated in Sec. 4.3, Fig. 6 displays the implication that there is an optimal data size under a certain FP quantization setting theoretically based on our scaling law. The validation of our scaling law is in Sec. 4.1 and Fig. 4 (as in A2). Fig. 6 aims to show the intuitive phenomenon of theoretical “optimal data size” in different settings. Hence, using the optimized bit settings in Fig. 5 also has a similar pattern due to the formation. A comprehensive derivation of $D_{crit}$ is in Appendix D.

Q5: Extension to larger models.

A5: Due to space limits, please refer to Reviewer UF7C's A1 for larger LMs' results (7B/70B).

References

[1] Kumar T et al. Scaling laws for precision.

[2] Hoffmann J et al. Training compute-optimal large language models.

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns, and we would be grateful if you could consider increasing the overall recommendation of our work.

Q1: Experiments capped at relatively modest model scales (max ~1B). Do you think the proposed scaling law can extend beyond model sizes > 1.2 B?

A1: Thanks for the suggestion. We agree with the reviewer that experiments on larger models could further verify the effectiveness of our scaling law. We attempt to answer from the following aspects:

(1) The scaling law exploration of LLMs is essential but extremely expensive in both GPUs and running time. To thoroughly explore the relationships between the loss and different floating-point quantization training factors (e.g., N, D, E, M, B), we have trained 366 models with different settings (model sizes from 41M to 679M) to draw our scaling laws, and successfully validate them on 1.2B models with 8 different settings in Fig. 4. The adopted model sizes are comparable with those in other scaling law works (e.g., Scaling law for precision [1], one of the most related work, adopts model sizes up to 1.7B parameter for validation).

(2) We conduct several additional experiments with model sizes larger than 1.2B. Precisely, we successfully predict the loss of 7B and 70B LLMs (with different settings) based on our scaling law. The detailed model setting, actual loss and predicted loss are shown as follows:

(The results of 1.2B models have already been included in Fig. 4 of our paper)

The actual losses of 7B/70B models are very close to the theoretical value calculated by our scaling law. Therefore, we could confidently claim that our scaling law could extend beyond larger model sizes. These additional results will also be given in the revision.

Q2: Hardware validation is missing, which could impact the practical usability of proposed methods.

A2: Thanks for your suggestion. We agree that these related works can provide valuable insights of floating-point quantization in practice. We will add these noted related works in our revision with detailed discussions.

Q3: Can the observed relationships between exponent and mantissa bit precision in your paper be conceptually related or compared with the notion of an 'effective parameter count' introduced by [1]? Would integrating such a concept clarify or enhance your theoretical interpretation?

A3: Thanks for the suggestion. As discussed in Appendix M of [1], the effective parameter count is formulated as a counterpart to the parameter count N in the Chinchilla scaling law. In Appendix C, Eq. 37 of our paper, we derive an analogous concept where our equivalent N demonstrates not only a positive correlation with precision metrics: $P^{\delta + \nu}$ or $(E + 0.5)^\delta(M + 0.5)^\nu$, aligning with their framework, but also a negative correlation with dataset size D. This reveals that increasing training data volume effectively reduces the equivalent parameter count, implying that larger datasets amplify the impact of numerical precision on model expressivity.

References:

[1] Kumar T et al. Scaling laws for precision.