Unlocking the Theory Behind Scaling 1-Bit Neural Networks

1. Given that the paper provides numerical results, I found it problematic that the main result of the paper (the functional form of the convergence of the training loss of 1-bit models) is not verified with numerical results.

2. The finding of a "scaling law" in Fig.1 with four data points, no comparison with a theory curve, and significant fluctuations is, in my opinion, not a valid verification of Proposition 4.3 at all.

3. The figures are not properly described/referenced in the text.

4. The introduction of $\kappa$ in the model seems a bit arbitrary, especially considering that it is necessary to "plug an appropriate value of $\kappa$" (Line 391) into the model to ensure learning.

5. The abstract does not adequately explain what is being done in the paper. Instead of citing previous results in the abstract, I would like to see the model and a more precise definition of the results there.

• This paper studies the scaling law of 1-bit neural networks. However, the analysis is performed on simple toy models. The original scaling law as proposed by Kaplan [1] is conducted on Transformers with millions/billions of parameters with huge amounts of pretraining compute. The goal of scaling law should be to study how to reliably predict the performance of pretraining with larger compute. However, the setting of this paper does not fit in the category of “scaling law”, given the small scale experiments conducted on toy models.
• The experiments do not support the claim that there is a scaling law for 1-bit quantized neural networks. The authors do not demonstrate the exact scaling law formula, e.g. what are the values of hyperparameters in proposition 4.3 when fitting this curve to the experiments in Figure 1. Obtaining the coefficient in the scaling law is of practical importance, which can be used to predict performance of training larger 1-bit quantized models.
• What is the justification for using NTK for studying this problem? The introduction of section 3.3 seems rather abrupt, which lacks the discussion for motivation.

• Lemma 4.1: the results in the lemma depend on $\lambda$, which I assume is $\lambda_{\min}(H^*)$ (this should be properly stated). However, if this is the case $\lambda$ should depend explicitly on $\kappa^2$, and the dependence on $\kappa^2$ should be tracked throughout the results. It is better to define $\lambda$ as the minimal eigenvalue of the non-scaled Gram matrix, and have the explicit dependence on $\kappa^2$ in the results.
• It is worth noting that the scaling laws derived in proposition 4.3 are very different form than the Kaplan scaling laws (e.g., $N$ and $D$ in the Kaplan scaling laws are in the denominator with fractional power, and are not exponentiated).
• The authors claim that the function implemented by the quantized network approaches the function computed by the full-precision network as the network size grows, but I do not see how this is reflected in the results. Specifically, it is worthwhile to write down an explicit bound on the difference between the two networks that goes to zero with $m$, when keeping other parameters (e.g. dataset size and network scale) constant. I currently did not see such a bound, but maybe I am missing something.
• Related to the above, the main results that relate the quantized and non-quantized networks (Theorems 5.1 and 5.2) rely on the scaling parameter $\kappa$ being very small. However, clearly taking $\kappa$ to zero makes both functions go to zero, and thus trivially the difference between the two functions becomes small. But this is also true for any (bounded) function multiplied by $\kappa$, so it seems that these results trivially hold. Do the results simply imply that all functions go to zero with $\kappa$ and thus become close, or can these result hold when both functions are bounded away from zero (e.g. by a constant)?
• The functions studied in 6.1 are relatively low-dimensional, which may mean that they can be approximated easily in the kernel regime by any kernel. Did you try experimenting with functions of higher dimensions?

Additionally, there are some unclear sentences/phrases in the paper, e.g.:

• Lines 392-395, these two sentences are not clear.
• Line 347
• Line 260-261
• Line 422: "we aimed to learn rigorous functions" - what does "rigorous function" mean here?

1. The main weakness of the paper is the lack of realistic experiments and the positioning of the paper. The authors use 1-bit LLMs to motivate the paper in the abstract and introduction, but their theory and experiments are for quantized MLPs at a very small scale. The authors should rewrite the abstract and introduction to clarify what their focus is on rather than overemphasizing low-bitwidth LLMs. For example, authors should explicitly state their focus is on quantized MLPs for learning functions, and clarify how their results may or may not extend to large-scale LLMs.

2. In section 3.2, the authors mention $f$ is a two layer attention model; however, the equations suggest it’s a two-layer MLP. Symbols like $m$ are not defined clearly. This section needs better clarity.

3. The choice of experiments (learning rigorous functions) seems arbitrary to me, given that the motivation of the paper is 1-bit LLMs. Can the authors explain this disparity?

4. Authors should look at related papers that try to empirically derive the scaling law of low-bit (binary and ternary) quantization-aware training of LLMs like [1], which might make the arguments in their paper stronger. They also provide the model checkpoints and losses at various scales for 1-bit and 1.58-bit LLMs. I think if the authors want to position their paper as validating the scaling of low-bit LLMs using their theory, they can use the existing checkpoints from this work.

[1] A Kausal et al. Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale