Scaling Laws for Precision

Thank you for the review! You’re right that power laws are common in scaling laws. For this reason, we compare directly to the power law functional form you point out in Appendix C, finding it has a weaker fit to the data than the exponential form we use in the main text. Note that an exponential model might also be reasonable because of the information-theoretic observation that $k$ bits can encode $2^k$ states, so that one may hypothesize that the first few bits contribute a large amount to effective parameter count, with quickly diminishing returns after that. But your point that this is an important baseline comparison is a good one.

We’ve also made a number of significant improvements to the manuscript and uploaded a new version with new Appendix Sections in blue:

• We’ve added preliminary experiments studying quantized training with FP instead of INT, involving over 100 new pretraining runs. Going beyond our functional form, we examine how loss scales with FP bits under non-standard mantissa vs exponent bit allocations. See Appendix N.
• We’ve added a new inference cost model studying the optimal choice of overtraining vs post-train quantization for workloads where inference is the dominant cost. This complements our training cost analysis to now provide insights for practitioners who care mostly about inference. It can be found in Appendix E.3.
• We’ve added experiments in Appendix I testing the effect of per-tensor vs per-channel quantization on scaling behavior with weight vs activation precision, finding that while granularity can explain part of the difference between weights and activations, activations are indeed inherently more sensitive to quantization at the same quantization granularity.
• We’ve added minor improvements throughout the manuscript: reformatting some plots, changing our validation shard, rewriting some paragraphs for clarity, and including more implementation details.
• We’ve added ablations for post-train-quantization: we reproduce our degradation results using AWQ, a modern post-train quantization method, as well as when we train with a linear learning rate schedule instead of cosine. The behavior persists under both ablations. See new Appendix F and G.

We think these additions improve the manuscript substantially. If there’s any experiments that would further improve your assessment of our work, feel free to let us know. Many thanks for the positive review.

• Do the results generalize across hardware setups?

Our results are hardware agnostic. GPUs today broadly do not support matrix multiplication or computation in most of the precisions we study, so all our experiments simulate quantization instead of using literal low-precision types. So the statement that “The experiments are conducted on specific hardware setups that support low-precision computations, such as GPUs optimized for integer-type operations” is not quite true as stated, since we never actually use any low-precision capabilities of the hardware we train on.

This is deliberate and standard in much work in quantization (eg. see [Wang et al, 2024]),and means that the trends we identify will generalize across hardware setups and remain relevant in the future. We do not assume anything about the hardware except for the fact that integer quantization (our focus) is implemented mathematically by some combination of rescaling and rounding, a standard convention [Jacob et al, 2018]. This desire to be hardware agnostic is also an important reason we don’t study or make any claims about performance metrics like latency or throughput, since these vary setup to setup. Even our optimality results just assume a particular functional form for loss and cost, and solve mathematically for the optima, without any hardware assumptions. We’ve added discussion in Appendix D.3 clarifying this point.

• Model size.

First and foremost, we want to be epistemically humble: certainly, the point of scaling laws is in some sense they are scale-invariant, but in practice many people believe that “more is different” so that models at trillion-parameter scale sometimes behave qualitatively differently than those at billion parameter scale. While the agreement between experiments at ~2B param scale and our fits at ~200m param scale suggest our scaling fits may indeed generalize, we welcome and look forward to follow-up work at larger model scale.

A more general comment is that the contribution of the Chinchilla paper [Hoffmann et al, 2022], for instance, was not the literal prediction that the optimal amount of tokens is $D* = 20N$. Instead of using this verbatim, in practice training a real model for deployment involves fitting on one's own architecture/data, see eg. [Dubey et al, 2024]. Instead, one might argue that the contribution of the Chinchilla is the meta-observation that loss as a function of $N, D$ is predictable and the downstream insight that the resulting functional form implies that data and parameters should be scaled in equal proportion. In our setting, whether the compute-optimal training precision is literally 7-8 bits or some slightly different number based on a different architecture/data setup is rather beside the meta-point. The observation that loss as a function of precision can be predicted at all – and that this can have surprising implications of the kind we study – is an important meta-contribution, and this observation we believe will hold even at trillion parameter scale.

We believe our extensive set of new experiments and ablations in direct response to your questions make the manuscript significantly stronger. We’d appreciate it if you consider raising your score, or point us to further desiderata. Thanks again for the detailed and thoughtful review!

--

References.

Ma, Shuming, et al. "The era of 1-bit llms: All large language models are in 1.58 bits." arXiv preprint arXiv:2402.17764 (2024).

Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

Jacob, Benoit, et al. "Quantization and training of neural networks for efficient integer-arithmetic-only inference." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018

Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022).

Wang, Hongyu, et al. "Bitnet: Scaling 1-bit transformers for large language models." arXiv preprint arXiv:2310.11453 (2023).

Thank you for the detailed feedback and close reading! We’ve added a lot more experiments and ablations to the manuscript directly in response to your excellent questions, which we believe strengthen the manuscript substantially. You can find them in new Appendix sections, which are written in blue in the updated manuscript, and we point to them in our response.

• Floating-point precision.

We launched and analyzed over 100 new pretraining runs specifically to address this question (new Appendix N), in which we vary weight training precision now in floating point instead of integer type. In short, we find that our existing functional form works well, as evidenced by having a high validation $R^2$ value (predictive power). One important conceptual difference between floating point and integer type is that FP bits are subdivided into exponent and mantissa bits, so these new pretraining runs assume reasonable and standard bit allocations between exponent vs mantissa as we scale precision. Making similar predictions for nonstandard allocation choices may require adapting our functional form by subdividing precision terms in our scaling law into exponent and mantissa. As a first step towards this, we sweep exponent and mantissa bits on their own and show they do not have the same scaling, since they play different roles in the floating-point representation scheme. But this is a technicality, with any standard bit allocation scheme we find our functional form works out-of-the-box for the preliminary floating-point sweeps we have added, suggesting that our functional forms or variants of them may work across types.

• Extremely low (binary/ternary) precision.

Two things to say here.

• Recent popular work on binary/ternary quantizes only weights. For instance, [Ma et al, 2024] leave activations in 8-bit. This is consistent with our findings that activations/KV cache are more sensitive than weights at extreme precision. The difficulty with extreme quantization lies in the activations/KV cache rather than the weights, as predicted by our Fig3 (left).
• Even just training weights in binary/ternary requires many changes to the standard architecture/training recipe. Training a vanilla (causal Transformer++) language model in the manner we do with binary/ternary weights, for instance, is unstable: the loss will diverge. For this reason, [Ma et al, 2024] make substantial modifications to their training setup, for instance using a custom learning rate and weight decay schedule. For us, changing architecture or training recipe across precisions would make it an unfair comparison across precisions since these changes made for stable low-precision training often do not work well compared to a standard training recipe when things are at high precision.

So the short answer to your question is that there is no obvious principled way to compare to such extreme precision weights in the setup we’re interested in studying since it requires nonstandard architectural/training changes that would not be used for usual training at non-extreme precisions. We’ve added discussion on this point in Appendix B, since readers may also have this question.

• Weight vs activation sensitivity – per-tensor vs per-channel.

We’ve added experiments in Appendix I to test this: we sweep weight precisions and activation precisions during training using both in per-tensor and per-channel at a fixed model size and data budget. We see that the degradation when going from per-channel to per-tensor is larger for activations than weights, and that activations per-channel are comparable to weights per tensor. This suggests that the per-tensor vs per-channel implementation detail accounts for some, but not all, of activations’ increased sensitivity in our main text as compared to weights. This is consistent with much existing work in quantization that finds activations to be harder to quantize than weights themselves.

• Inference-time cost analysis.

This is a great idea. We’ve added an inference-time cost analysis in Appendix E.3. The key tradeoff is that to get cheap inference at a fixed desired loss, you can 1) overtrain small models, 2) train a larger model and post-train quantize, 3) do some combination. Which is optimal if a practitioner's main concern is inference costs? We set up a cost model and solve it numerically, finding that in short, models that are extremely overtrained should be made slightly larger than quantized more aggressively at inference-time if inference is the primary cost concern compared to an overtrained model served in 16-bit. Interestingly, the optimal data budget still increases as inference-time compute budget does, even though D doesn’t appear in the cost function. Further, the analysis reveals the insight that post-train precision also increases (logarithmically) with inference-time compute, for reasons we explore in the Appendix. Thanks for this suggestion!

Thank you for the thoughtful and kind review!

• We ablate the dataset to use C4 (albeit at small scale) in Appendix A, finding similar trends.
• The models we train are indeed standard causal Transformer-based language models. We’ve now mentioned their architecture in Appendix A, and exploring whether these trends are universal across classes of models (Transformers vs SSMs, etc) is an exciting line of future work!

Re: the counterintuitive findings. For post-train quantization, we include a discussion on what could be mechanistically causing it in Appendix H – our working intuition is that as models are trained on more data, they store more information in their weights in a more compressed manner, so that noise of a fixed variance to perturb those weights will have a relatively more damaging effect. And about quantized training, our intuition is that we can think of quantization as applying noise of a variance that decreases quickly with precision on the forward pass. So the precision-parameter tradeoff at the center of our quantized training section is really just a matter of asking: do we expect a 1b model with noise variance 1 on its forward pass to do better than a 200m model with no noise? It does not seem obvious beforehand what the answer to such a question is, which is why a scaling law is needed. This assumes that quantization effects on the forward pass can reasonably be thought of intuitively as noise, which seems to be justified by the fact that our functional forms fit the data well. Thanks for the good questions!

We’ve also added a number of things to the manuscript that we think substantially strengthen it, and uploaded a new version with new Appendix sections in blue:

• We’ve added preliminary experiments studying quantized training with FP instead of INT, involving over 100 new pretraining runs. Going beyond our functional form, we examine how loss scales with FP bits under non-standard mantissa vs exponent bit allocations. See Appendix N.
• We’ve added a new inference cost model studying the optimal choice of overtraining vs post-train quantization for workloads where inference is the dominant cost. This complements our training cost analysis to now provide insights for practitioners who care mostly about inference. It can be found in Appendix E.3.
• We’ve added experiments in Appendix I testing the effect of per-tensor vs per-channel quantization on scaling behavior with weight vs activation precision, finding that while granularity can explain part of the difference between weights and activations, activations are indeed inherently more sensitive to quantization at the same quantization granularity.
• We’ve added minor improvements throughout the manuscript: reformatting some plots, changing our validation shard, rewriting some paragraphs for clarity, and including more implementation details.
• We’ve added ablations for post-train-quantization: we reproduce our degradation results using AWQ, a modern post-train quantization method, as well as when we train with a linear learning rate schedule instead of cosine. The behavior persists under both ablations. See new Appendix F and G.

We believe these additions make the manuscript much more complete and comprehensive. If there’s any experiment that would further improve your assessment of our work, feel free to let us know. Many thanks for the review!

Thank you for the review! The typo is fixed.

We’ve made a number of significant improvements to the manuscript and uploaded a new version with new Appendix Sections in blue to easily find them:

• We’ve added preliminary experiments studying quantized training with FP instead of INT, involving over 100 new pretraining runs. Going beyond our functional form, we examine how loss scales with FP bits under non-standard mantissa vs exponent bit allocations. See Appendix N.
• We’ve added a new inference cost model studying the optimal choice of overtraining vs post-train quantization for workloads where inference is the dominant cost. This complements our training cost analysis to now provide insights for practitioners who care mostly about inference. It can be found in Appendix E.3.
• We’ve added experiments in Appendix I testing the effect of per-tensor vs per-channel quantization on scaling behavior with weight vs activation precision, finding that while granularity can explain part of the difference between weights and activations, activations are indeed inherently more sensitive to quantization at the same quantization granularity.
• We’ve added minor improvements throughout the manuscript: reformatting some plots, changing our validation shard, rewriting some paragraphs for clarity, and including more implementation details.
• We’ve added ablations for post-train-quantization: we reproduce our degradation results using AWQ, a modern post-train quantization method, as well as when we train with a linear learning rate schedule instead of cosine. The behavior persists under both ablations. See new Appendix F and G.

We believe these additions substantially strengthen the manuscript; if there’s any experiment that you think would further improve your assessment of our work, please let us know. Many thanks for the positive and thoughtful review.