Scaling Laws for Sparsely-Connected Foundation Models

Thank you for the review! We respond to your comments and questions in detail below.

1, Experiments are only tested on two examples. Though due to limited resources, more examples may be challenging, two examples may not be very convincing.

Our main experiments feature 112 training runs for ViT models and 48 training runs for T5 models, plus several runs for n:m sparsity, pruning pretrained models, ablation studies and two larger extrapolation experiments. Many of these take days to execute on multiple accelerators (one extrapolation experiment for example takes 2 days on 128 devices).

Additionally, towards addressing this concern, we have conducted additional experiments for autoregressive decoder-only models, showing that our scaling laws also hold in this context. We present those new results in Appendix G of the most recent revision or, for convenience, at this anonymized link: https://github.com/iclr24-1372/iclr24-1372/blob/master/decoder-only.pdf.

Finally, we would like to note that most scaling works focus only on a single domain and model family, while we study both vision and text models (and now also encoder-decoder and decoder-only variants). This highlights the generality of our sparsity scaling laws.

2, The scaling law has too many degrees of freedom (7 free parameters) that could be hard to generalize to new dataset without sufficient information.

Standard dense scaling laws usually have at least 5 parameters for two inputs N (model size) and D (training data). Hence, we think allowing 2 more parameters to model the impact of an additional third input S (sparsity) is a reasonable increase. The 7 free parameters are fitted over 112 ViT and 48 T5 training runs, respectively, hence the #samples to #parameters ratio is ~16 and ~7, which limits overfitting capability. At the same time, we still observe good fits and extrapolation performance.

We note that the coefficients of a scaling law are usually fit on a per model family and dataset/task basis, only the overall functional form of the law is supposed to generalize across settings, which we demonstrate by studying both vision and language models.

1, What are the 7 free parameters fitted and how are these parameters changes for different dataset and different hyperparameters S, N, D?

First, we would like to note that $S$/$N$/$D$ are inputs to our scaling law and not hyper-parameters, the same coefficient values ($a_S$, $b_S$, …) are used to model validation losses for all $S$/$N$/$D$ configurations. Different model family and dataset/task combinations can have coefficients that are quite different; we list the particular values we obtained for our experiments in Appendix C. However, the overall functional form of the scaling law remains consistent across domains (vision & text) and architectures (encoder-only, encoder-decoder).

2, Procedure to identify in 2.2 Optimal Sparsity is confusing. Could you explain this procedure with more details?

The high level idea is as follows:

1. We first reparametrize our $L(S, N, D)$ scaling law in terms of $L(S, N, C)$ by replacing $D$ with the corresponding amount of training data that, for a model with parameter count $N$ and sparsity $S$, leads to overall compute $C$.
2. Next, we would like to minimize $L(S, N, C)$ with respect to $S$, in order to determine the best sparsity value. This is done in standard fashion by setting the derivative of $L$ with respect to $S$ to 0.
3. Our expression given for the contour lines is found by rearranging terms on both sides. Similarly, $S_\text{opt}$ is determined by solving for $S$ with standard algebraic manipulations. We provide those algebraic details in Appendix D.

Thank you for the review! We respond to your comments and questions in detail below.

My main concern lies in the contribution of section 2 on fair evaluation, which argued that a sparse network should be compared with a dense model with the same number of parameters and compute. A very similar argument is made in [1] (See “training budget” in section 3), which argued that the compute budget for comparing sparse and dense models should be the same. It would be good to clarify the differences to [1] in the paper.

The main purpose of this paragraph was to highlight that this resource normalization is critical when dealing with foundation models that exhibit strong scaling behavior. This is not as important on standard smaller-scale pruning benchmarks (e.g., CIFAR/ImageNet or BERT/GLUE) and is thus often neglected. Nevertheless, as pointed out by the reviewer, training budget normalization has indeed been considered by some prior works as well, in different contexts. We now explicitly state this in Section 2 of the newest revision, referencing Liu et al. (2019) [1], as well as Jin et al. (2022).

Apart from the fact that those works also highlight similar considerations regarding fair evaluation of sparse models, they are quite different in scope. Specifically, neither of them develops scaling laws or studies foundation models.

The evaluation on the encoder-decoder architecture T5 seems limited. The paper do not evaluate the more popular autoregressive decoder-only architecture but the original scaling laws is built on such autoregressive language models.

We initially selected the T5 encoder-decoder architecture to demonstrate that our sparsity scaling law is also applicable to slightly more complex Transformer variants. However, we agree that a standard autoregressive decoder-only setup would be a useful addition and have conducted an additional scaling sweep for autoregressive decoder-only Transformers.

Specifically, we train several autoregressive decoder-only Transformers for next-token-prediction on the C4 dataset, using similar size/sparsity/data settings as in our T5 experiments. The results are presented in Appendix G of the most recent revision or, for convenience, at this anonymized link: https://github.com/iclr24-1372/iclr24-1372/blob/master/decoder-only.pdf.

Overall, we observe all the key aspects of our scaling law also in this setting (sparsity acts as a constant multiplier on size scaling, is mostly independent of the amount of training data, and yields diminishing returns at higher values), which consequently leads to a good coefficient fit as well. This suggests that standard autoregressive decoder-only models exhibit the same general sparsity scaling behavior as ViTs and encoder-decoder models, further confirming our results.

I am curious as to whether lottery ticket hypothesis would hold as to foundation models, as it is the most impactful pruning paper in recent years. Since this work is closely related, i am wondering if the authors have explored the LTH setting for sparse training. If not, it would be good to add a discussion on LTH [2].

We did not explore lottery tickets specifically; this is primarily because first training a dense model only to select a mask and then rewind any optimization progress (by resetting the weights to their initial values) is not a particularly efficient pruning strategy. This is especially the case when fairly accounting for the overall training costs, which was a key goal of our study. The inefficiency of lottery tickets has been shown in [1], which finds that rewinding the weights performs worse than learning rate restarts. Additionally, [2, 3] provide explanations on why lottery tickets fail at scale.

• [1] https://arxiv.org/abs/2003.02389
• [2] https://arxiv.org/abs/2010.03533
• [3] https://arxiv.org/abs/2210.03044

2. Can the proposed scaling laws be useful in the extreme sparsity regime, e.g. 95% or even 99%? Or is it applicable only for the sparsity level within a range?

We did initially also perform a few experiments for 93.75% sparsity (16x compression); however, as also suggested by, e.g. Figure 2, improvements were diminishing and producing those models quite expensive (we have to train 16x larger models to eventually reach a certain target nonzero count). Thus, we decided to focus our computational budget on the most interesting and practical range. In principle, our scaling laws should also extend to higher sparsities (though, one should be careful with extreme extrapolation for any kind of scaling law due to potential problems like training instabilities for very unusual configurations).

The rebuttal have addressed my concerns. I appreciate the authors for adding the experiments on decoder-only Transformers. Thus I maintain my positive assessment of this paper (confidence is increased to 5).

I noticed that the term "computational budget" is mentioned a lot. I would suggest give some details on this, e.g. hardware and training time. In this way, the reader may have a concrete idea on the large-scale nature of the conducted experiments.

Thank you for the review! We respond to your comments and questions in detail below.

Overall, I think this is a good paper; my only concern lies in the scale of the data and modeling. Compared to Chinchilla's law, the author's data and model sizes are significantly smaller.

We would like to note that the Chinchilla paper is a highly expensive large-scale study, which was conducted only after the great potential of scaling had been very clearly demonstrated by models like GPT3 or Gopher. The seminal scaling work by Kaplan et al. (2020) was based on smaller model sizes, which are close to ours.

We would also like to highlight a few aspects that make our study quite computationally demanding, already at the model sizes we consider (we discuss this further in Appendix A):

• Producing a model with sparsity $S$ and $N$ nonzeros parameters requires actually training and pruning an $N / (1 - S)$ times larger model (e.g., 8x larger for 87.5% sparsity).
• We have to sweep not only over model size N and amount of training data D, but also sparsity S, multiplying the number of runs required to collect data for fitting scaling laws by the number of sparsity levels.
• As shown by our work, sparse models become optimal in longer training regimes, hence we want to include several runs in our fitting data which train significantly longer than Chinchilla optimal (for instance, our ViT runs include settings with 1.8B images and correspondingly ~350 billion tokens cost).
• We study both vision and language models, further increasing the number of experiments.

Nevertheless, we agree that further investigations at even larger scales would be valuable and hope to look into this for future work. However, we believe that the current set of experiments already validate our proposed scaling laws.

Also, since the authors mentioned the optimal sparsity setting as an empirical design or product of this claim, could you elaborate on what specific improvements in downstream task or operational efficiency the sparsity of our actual model would bring us under the authors' theoretical guarantee?

The main goal of our work is to understand the scaling behavior of sparsity in the context of foundation models, in general, irrespective of hardware or runtime specific artifacts. This should help ongoing and future efforts on practical sparsity acceleration by answering questions such as: (1) which sparsity levels to expect while recovering accuracy, (2) how long sparse models must be trained to recover accuracy, or (3) how large improvements over dense models can be expected.

In the newest revision, we have added Appendix F, discussing related work on practical sparsity acceleration including: acceleration algorithms for CPU inference (Elsen et al., 2020; Kurtz et al., 2020), n:m sparsity (Pool & Yu, 2021) supported on current NVIDIA GPUs (which we explicitly study in Section 3 as well), and custom hardware specifically designed for sparsity (Emani et al., 2023).

Thank you for the review! We respond to your comments and questions in detail below.

This paper posits that sparsity can reduce training costs, thus presenting research on optimal sparsity. However, in reality, the training costs of current sparse training methods are often the same as those without sparsity. Under these circumstances, is it meaningful to study optimal sparsity?

There are already some promising results for achieving practical training speedups through unstructured sparsity, both via hardware accelerators (Emani et al. (2023), Figure 7) as well as sparsity-aware backpropagation algorithms on commodity hardware (Nikdan et al. (2023)). Nevertheless, we agree that this is still an ongoing area of research, which is exactly why we considered two different cost models in our optimal sparsity analysis:

• Dense: no acceleration from sparsity during training whatsoever.
• Sparse: sparsity is fully exploited as soon as it appears.

In both cases, sparse models become optimal (in terms of size / inference cost) as the training budget is increased; however, this happens more quickly if sparsity can already be exploited during training. These results can also be adapted to different cost models.

In the most recent revision of our paper, we have added additional discussion about related works for utilizing sparsity in practice, for inference speedups, training acceleration and memory reduction (see Appendix F). We believe that our work will be particularly valuable for future work in this area, by providing guidance on optimal sparsity levels, possible accuracy gains and scaling behavior with respect to model sizes and training data.

• Nikdan et al., SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks, 2023.
• Emani et al., A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators, 2023.