Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Thank you for your review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Weakness #1

First, there seems to be a fundamental misunderstanding in your review. Activation sparsity is completely different from pruning, which you indicate by "removing sparse parts of the network to speed up". Pruning realizes sparsity by removing certain parts of the model and this process is independent of the inputs. Its input-independent property can easily cause performance degradation. By contrast, activation sparsity, which we mainly research, is dynamically determined by the inputs and thus potentially compromises less model capacity and downstream task performance. More specifically, activation sparsity does not remove any part from the model. Instead, the sparsity is realized by recognizing the unimportant parameters for each token and then dynamically deactivating them.

Next, how the sparsity affects performance is well studied in this paper. In Table 1, we already provide accuracies on C.R. and R.C. benchmarks given by the dense model and PPL-p% at different p% levels (also, with different sparsity levels).

To further consolidate the advantage of our results, in Figure 14 of the updated manuscript, we give the Pareto curves of task performance v.s. sparsity. These curves show that PPL-p% obtains a better trade-off between task performance and sparsity compared to baseline metrics. Specifically, both baseline metrics start to show significant performance degradation (i.e., decrease by more than 5%) at a sparsity point consistently lower than PPL-p%. Meanwhile, this also demonstrates that high sparsity can co-exist with a low PPL increase ratio and minor performance degradation, especially in ReLU-activated models. Note that these evaluation accuracies and sparsity levels are measured on models after the decay stage, which includes SFT data.

Finally, the works we cited can sufficiently demonstrate the advantages of employing activation sparsity, such as inference acceleration and interpretability. PowerInfer-2 [3], for example, can achieve up to 29.2x speedup and about 40% reduction in memory usage by utilizing activation sparsity. Note that these are all done on smartphones.

Weakness #2

First, the parameters of our models are not empirically determined. We adopt MiniCPM [1] as our experimental architecture and employ muP [2] to specify hyper-parameters for training stability. Such model architecture and parameter specification are both demonstrated to be reliable and well recognized, with considerable citations and application.

Next, all the works on scaling properties are empirical, usually including extensive experiments and curve fitting. Among these works, the most famous one is done by OpenAI [4]. Though also consisting of empirical experiments and curve fitting, this work reveals the quantitative relationship between the performance of AI models and the amount of training data as well as the number of parameters. Its precious experience lays the solid foundation of LLM and leads human beings to the revolution of AI.

A helpful work may not necessarily have all its conclusions proved in a mathematical and rigorous way. In the field of AI, reliable conclusions can be obtained from extensive experiments. As for our work, we have experiments on five scales ranging from 0.1B to 1.2B, where the largest model has 12 times the number of non-embedding parameters as the smallest one. Besides, models are fully trained with hundreds of billions of data. Such extensiveness and expensiveness can already provide some generalizability for our findings.

Finally, even from the statistical perspective, fitting four parameters with dozens of samples per curve is a quite reliable practice.

Weakness #3

The findings of our paper are far beyond "ReLU is sparser". The "dying ReLU" may be a reasonable explanation for the higher sparsity of ReLU-activated models, but it cannot explain another two important findings related to ReLU: (1) ReLU-activated models are well comparable in performance with SiLU-activated ones; (2) ReLU-activated models undergo an increasing trend of sparsity with the increasing data, while SiLU-activated ones undergo an opposite trend. Therefore, we demonstrate that ReLU is a more efficient choice considering both sparsity and performance. The "dying ReLU" may be true, but it is an advantage of ReLU that promotes sparsity without harming performance, rather than a problem.

Besides, our findings related to activation functions are of very important value in this LLM era, as most mainstream LLMs adopt SiLU without awareness of the potential benefits in activation sparsity provided by ReLU. What we want to achieve is to attract more attention to activation sparsity, and provide precious experience for LLM researchers to reconsider specific model settings for higher sparsity.

Question #2

We have released all our data used to fit the activation-data curves at this anonymous GitHub repository (link). It also includes codes to evaluate and visualize our fitting results.

Besides, we compare the fitting results of our power-law and the logarithmic function $A_0+\log(a*D+b)$. We find that for both ReLU and SiLU models, the fitting results of power-laws are much better than the logarithmic function according to either the Mean Square Error (MSE) or Mean Absolute Error (MAE). For example, the MSE of our power-law on 0.1B ReLU-activated model is $2.68\times10^{-7}$, while the MSE of the logarithmic function increases to $2.17\times10^{-5}$.

Notably, the logarithmic function is intuitively not suitable for our data and experiment setting. It cannot converge with $D\rightarrow\infty$, which is totally unreasonable. After all, the activation ratio is a bounded variable within $[0,1]$.

We use the curve_fit method from package scipy to fit our curves, which employ the Levenberg-Marquardt method.

Question #3

To obtain more insights into the "dying ReLU" problem, we study the dead neurons in our models. Specifically, we define the neurons that are activated by less than 0.1% tokens in the validation dataset on average as "dead neurons". For the 0.1B ReLU-activated model and SiLU-activated model, we obtain their trends of activations ratios and dead neuron ratios. The curves are drawn in the figure included in the anonymous GitHub repository (link).

For both models, the dead neuron ratios do not increase considerably throughout the training process. For example, the dead neuron ratio of 0.1B ReLU model increases to about 0.35% at the end of training. However, its activation ratio decreases by about 3.38% (from 10.47% to 7.09%). We can draw the following conclusion: The "dying ReLU" phenomenon does slightly exist, but is far from the fundamental cause of the decreasing activation ratio trends of ReLU-activated models.

Generalizability

Our empirical conclusions are generalizable to even larger models.

We have finally obtained the experimental results on a ReLU-activated 2.4B model, which has twice the number of non-embedding parameters as the previously largest model (1.2B). At this point, it has been pre-trained on about 291B tokens. The data used to fit the activation-data curves is available at this anonymous GitHub repository (link), including the results of the above 2.4B model.

By analyzing the 2.4B results, we find our conclusions generalizable to this larger model.

• The activation ratio also follows a logspace power-law with the amount of data: $A(D)=\exp(-(9.04\times10^{-6})\cdot D^{2.11}-3.82)+0.071$.
• The limit activation ratio is 7.1%, which is close to the activation ratios of smaller models (e.g., 7.8% for 1.2B and 7.2% for 0.8B). This is consistent with our conclusion that the limit activation ratio is weakly correlated with the number of parameters.
• By comparing the activation-data curves, we find that the activation ratio of the 2.4B model converges much slower than the 1.2B model. This is consistent with our observation that smaller models tend to converge faster than larger models to the limit activation ratio.

We are looking forward to your response! Your comments will certainly help us reflect more on our work and continue to forge ahead on the academic path! We have addressed your remaining concerns in detail with full experiments. Meanwhile, the data and codes are also provided to substantiate our studies further.

We have conducted comprehensive experiments and invested considerable computation resources to address the remaining concerns you raised. We are looking forward to your response, which is really important to us.

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Weakness #1

We believe that experiments on even larger models can further substantiate the generalizability of our research. However, we have long met with great difficulties collecting sufficient GPUs to run larger models. At present, we struggle to gather 64 GPUs and start running a 2.4B model to validate our findings, and we will try our best to present the results before the end of the rebuttal period.

Besides, note that even the most famous works on scaling laws [1] did not run models with more than 1B parameters due to the extremely expensive nature of such studies. As for our work, we have experimented on scales from 0.1B to 1.2B, where the largest model has 12 times the number of non-embedding parameters as the smallest one. Such a large gap can already provide some generalizability for our findings.

Weakness #2

Sorry for causing this misunderstanding. Actually, in our paper, the FFNs in all models are gated FFN, as specified in line 153 and line 237 (MiniCPM uses gated FFN). Therefore, the ReLU and SiLU refer to ReGLU and SwiGLU, respectively. We do not incorporate non-gated FFN, as it is seldom used in recent mainstream LLMs.

Besides, to increase the generalizability of our work, we have started experiments on gated GELU-activated FFNs, and already completed 0.1B and 0.2B settings. Similar to SiLU, we find a power-law relationship between activation ratio and data. The fitted curves are $A_{GELU}=-\frac{0.02}{D^{1.87}} + 0.333$ and $A_{GELU}=-\frac{0.14}{D^{1.15}} + 0.342$ for 0.1B and 0.2B respectively. The two limit activation ratios are also very close, and the smaller 0.1B GELU model converges much faster than 0.2B. These observations are consistent with existing results.

Question #1

Mathematically, the relationship between PPL and CETT is not rigorously monotonous. LLMs are statistic models with random output noises and fluctuations (PPL evaluates the deviation of model outputs from the original ground-truth training corpus). However, from the statistical perspective, we can state that PPL generally rises with the increase of CETT. This can be demonstrated through experiments.

As shown in Figure 13 of our updated paper, for both 0.1B and 0.8B, from ReLU to SiLU, the PPL always rises with the increasing CETT. The larger the CETT, the more our models deviate from the original dense checkpoint trained on the ground-truth corpus. This means that the model outputs will deviate more from the ground-truth and finally produce a larger PPL value.

Question #2

The activation pattern has already been widely used in the interpretation of LLMs. Specifically, the activation pattern reflects the behavior of neurons (FFN parameters) in response to given input tokens. By analyzing the correlation between neuron activation and inputs, we can explain the specialization of neurons (i.e., what input patterns a specific neuron is sensitive to) and even the grouping of neurons (i.e., neurons with similar specialization are clustered as intrinsic modules within LLMs). [2,3]

The next question is why higher sparsity can bring better interpretability. Let's take MoEfication [4] as an example, which is a work that finds and utilizes the intrinsic modules within LLMs. This work clusters neurons into groups (i.e., experts) by using the co-activation frequencies between neurons as a distance metric. According to Figure 3 of the MoEfication paper, the relative performance degradation from the original dense model to MoEfied model is clearly less significant on sparser models. In other words, the loss of neuron clustering is less on sparser models, indicating more significant neuron grouping. Therefore, we can assume that sparser models are potentially more interpretable from the aspect of neuron grouping, an important part of interpretation.

References

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[2] Li, Zonglin, et al. "The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers." The Eleventh International Conference on Learning Representations.

[3] Zhang, Zhengyan, et al. "Emergent Modularity in Pre-trained Transformers." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

[4] Zhang, Zhengyan, et al. "MoEfication: Transformer Feed-forward Layers are Mixtures of Experts." Findings of the Association for Computational Linguistics: ACL 2022. 2022.

We are looking forward to your response! Your comments will certainly help us reflect more on our work and continue to forge ahead on the academic path!

We have finally obtained the experimental results on a ReLU-activated 2.4B model, which has twice the number of non-embedding parameters as the previously largest model (1.2B). At this point, it has been pre-trained on about 291B tokens. The data used to fit the activation-data curves is available at this anonymous GitHub repository (link), including the results of the above 2.4B model.

By analyzing the 2.4B results, we find our conclusions generalizable to this larger model.

• The activation ratio also follows a logspace power-law with the amount of data: $A(D)=\exp(-(9.04\times10^{-6})\cdot D^{2.11}-3.82)+0.071$.
• The limit activation ratio is 7.1%, which is close to the activation ratios of smaller models (e.g., 7.8% for 1.2B and 7.2% for 0.8B). This is consistent with our conclusion that the limit activation ratio is weakly correlated with the number of parameters.
• By comparing the activation-data curves, we find that the activation ratio of the 2.4B model converges much slower than the 1.2B model. This is consistent with our observation that smaller models tend to converge faster than larger models to the limit activation ratio.

We are looking forward to your response! Your comments will certainly help us reflect more on our work and continue to forge ahead on the academic path! We have added more experiments on a larger model and more activation functions.

We have conducted comprehensive experiments and invested considerable computation resources to address the concerns you raised. We are looking forward to your response, which is really important to us.

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Weakness #1

In Table 1, we already provide accuracies on C.R. and R.C. benchmarks given by the dense model and PPL-p% at different p% levels (also, with different sparsity levels).

To further consolidate the advantage of our results, in Figure 14 of the updated manuscript, we give the Pareto curves of task performance v.s. sparsity. These curves show that PPL-p% obtains a better trade-off between task performance and sparsity compared to baseline metrics. Specifically, both baseline metrics start to show significant performance degradation (i.e., decrease by more than 5%) at a sparsity point consistently lower than PPL-p%. Meanwhile, this also demonstrates that high sparsity can co-exist with a low PPL increase ratio and minor performance degradation, especially in ReLU-activated models. Note that these evaluation accuracies and sparsity levels are measured on models after the decay stage, which includes SFT data.

Weakness #2

As for the width-depth issue, I'd like to state the most accurate suggestion we can give. That is, use the smallest width-depth ratio that ensures training stability.

Existing works have conducted studies on the trade-off between performance and width-depth ratio. [1,2] As revealed by these papers, deeper models (with smaller width-depth ratios) generally present better performance if unlimited precision is given. However, real-life training usually involves half-precision training, where an extremely high depth can considerably harm stability and thus cause performance degradation. These findings are consistent with our work, which states that a smaller width-depth ratio is better for higher sparsity, but an extremely small ratio significantly harms performance (due to training instability).

To sum up, if unlimited precision virtually exists, smaller width-depth ratios can simultaneously bring better performance and higher sparsity. Nevertheless, if limited precision and training stability are considered, the best value is hard to say, as the training stability depends on various factors, such as the training framework and hardware. The most accurate suggestion is to use the smallest width-depth ratio that guarantees training stability.

Weakness #3

We believe that experiments on even larger models can further substantiate the generalizability of our research. However, we have long met with great difficulties collecting sufficient GPUs to run larger models. At present, we struggle to gather 64 GPUs and start running a 2.4B model to validate our findings, and we will try our best to present the results before the end of the rebuttal period.

Besides, note that even the most famous works on scaling laws [3] did not run models with more than 1B parameters due to the extremely expensive nature of such studies. As for our work, we have experimented on scales from 0.1B to 1.2B, where the largest model has 12 times the number of parameters as the smallest one. Such a large gap can already provide some reliability for our findings.

Weakness #4

In both Figure 3 and Table 1, we have comprehensively demonstrated that whatever sparsity metric we use, enforcing too high activation sparsity can cause considerable performance degradation. In Figure 1, we also present experiments studying the effect of granularity and activation rate on the PPL of MoE models, which shows that larger activation ratios generally present lower PPL in MoE. Note that we apply standard load balancing loss for all MoE settings.

Besides, there are also works [4] studying the quantitative relationship between performance and the number of activated parameters in fine-grained MoE models, which clearly show that less activated parameters can negatively affect performance. Note that fine-grained MoE can be regarded as a special case of activation sparsity, and thus it is reasonable to assume similar laws in our scenario.

Finally, as you mention many kinds of MoE models can simultaneously obtain higher sparsity and better performance, we think this can usually be attributed to better training data, special training techniques (e.g., sparsity restrictions in training target), or other potential factors. A special case we find is Switch Transformer [5], where the top-1 routing performs better than top-2 routing. This may be attributed to many other improvements, such as differentiable load balancing, the setting of expert capacity, and many sophisticated training techniques mentioned in Section 2.4. If you have more recent cases, where models are of the same architecture and training recipe, you can provide us for further discussion.

We are looking forward to your response! Your comments will certainly help us reflect more on our work and continue to forge ahead on the academic path!

We have finally obtained the experimental results on a ReLU-activated 2.4B model, which has twice the number of non-embedding parameters as the previously largest model (1.2B). At this point, it has been pre-trained on about 291B tokens. The data used to fit the activation-data curves is available at this anonymous GitHub repository (link), including the results of the above 2.4B model.

By analyzing the 2.4B results, we find our conclusions generalizable to this larger model.

• The activation ratio also follows a logspace power-law with the amount of data: $A(D)=\exp(-(9.04\times10^{-6})\cdot D^{2.11}-3.82)+0.071$.
• The limit activation ratio is 7.1%, which is close to the activation ratios of smaller models (e.g., 7.8% for 1.2B and 7.2% for 0.8B). This is consistent with our conclusion that the limit activation ratio is weakly correlated with the number of parameters.
• By comparing the activation-data curves, we find that the activation ratio of the 2.4B model converges much slower than the 1.2B model. This is consistent with our observation that smaller models tend to converge faster than larger models to the limit activation ratio.

Thank you to the authors for their efforts in addressing my concerns. I think the sparsity law observed in this paper offers valuable insights for researchers to better understand the sparsity of large language models and inspire the design of sparse models. After reviewing the clarifications provided, I have decided to maintain my previous score.

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Weakness #1

Thank you for pointing out the over-fitting concern. The computation of PPL-p% does not intentionally minimize the PPL. Actually, it adopts adaptive thresholds for different layers while maintaining a layer-wise consistent L2 relative output error (i.e., CETT). The inactivated neurons are those with output magnitudes lower than the layer-specific threshold. The PPL is incorporated as a variable to make our metric more performance-aware, binary searching the CETT just to meet the PPL requirements to provide a performance-dependent sparsity value. The determination of inactivated neurons is weakly related to PPL itself.

Besides, existing works have already demonstrated that a monotonous relationship between loss (log PPL) and downstream performance generally exists. [1,2] Therefore, if PPL-p% can ensure lower PPL, this usually produces better downstream performance. As for specific results, Table 1 provides accuracies on C.R. and R.C. benchmarks given by the dense model and PPL-p% at different p% levels.

To further consolidate the advantage of our results, in Figure 14 of the updated manuscript, we give the performance-sparsity Pareto curves, which show that PPL-p% obtains a better trade-off between task performance and sparsity compared to two baseline metrics. Specifically, both baseline metrics start to show significant performance degradation (i.e., decrease by more than 5%) at a sparsity point consistently lower than PPL-p%.

Weakness #2 & #4

To consolidate our findings, we have already started experiments on a 2.4B model, which ties all our findings to pursue a highly sparsely-activated model. Specifically, it adopts ReLU activation, a small width-depth ratio (within the interval that ensures the lowest loss), and more training data to utilize the decreasing trend between activation ratio and data of ReLU models. Besides, its larger size can further demonstrate the generalizability of our work.

We have long met with great difficulties collecting sufficient GPUs to run larger models. At present, we struggle to gather 64 GPUs for the above 2.4B experiment. While works on scaling laws are extremely expensive, we will try our best to present the results before the end of the rebuttal period.

Weakness #3

As stated in line 237, we adopt the same architecture of MiniCPM [3], which adopts mostly the same architecture as LLaMA except for minor muP [4] adjustments for training stability. As for training strategies, the two-stage training paradigm is already accepted by cutting-edge models such as LLaMA3 [2]. Therefore, we think experiments on MiniCPM are already generalizable enough to cover most mainstream LLMs adopting the LLaMA-like Transformer decoder-only architecture.

If you have other LLMs worth concern (e.g., BERT, ViT, T5, or other LLMs dissimilar to LLaMA), you may kindly provide specific suggestions so that we can present experiments in time.

Question #1

The dataset details are already described very comprehensively in Appendix E and I.

References

[1] Owen, David. "How predictable is language model benchmark performance?." arXiv preprint arXiv:2401.04757 (2024).

[2] Dubey, Abhimanyu, et al. "The Llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

[3] Hu, Shengding, et al. "MiniCPM: Unveiling the potential of small language models with scalable training strategies." arXiv preprint arXiv:2404.06395 (2024).

[4] Yang, Greg, et al. "Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer." arXiv preprint arXiv:2203.03466 (2022).

We are looking forward to your response! Your comments will certainly help us reflect more on our work and continue to forge ahead on the academic path!

We have finally obtained the experimental results on a ReLU-activated 2.4B model, which has twice the number of non-embedding parameters as the previously largest model (1.2B). At this point, it has been pre-trained on about 291B tokens. The data used to fit the activation-data curves is available at this anonymous GitHub repository (link), including the results of the above 2.4B model.

By analyzing the 2.4B results, we find our conclusions generalizable to this larger model.

• The activation ratio also follows a logspace power-law with the amount of data: $A(D)=\exp(-(9.04\times10^{-6})\cdot D^{2.11}-3.82)+0.071$.
• The limit activation ratio is 7.1%, which is close to the activation ratios of smaller models (e.g., 7.8% for 1.2B and 7.2% for 0.8B). This is consistent with our conclusion that the limit activation ratio is weakly correlated with the number of parameters.
• By comparing the activation-data curves, we find that the activation ratio of the 2.4B model converges much slower than the 1.2B model. This is consistent with our observation that smaller models tend to converge faster than larger models to the limit activation ratio.
• This 2.4B model combines all our previous insights: more efficient ReLU activation, efficient training data, and a smaller width-depth ratio. Finally, as expected, it achieves a high activation sparsity level with only about 7.1% neurons on average to be activated by each token.

Performance-Awareness is the Key Property

In the paper, we propose three properties (line 85) that a good sparsity metric should have: versatility across model architectures, performance-awareness, and the precise recognition of weakly-contributed neurons. The Pareto curves only test the 3rd property. However, the first key to your concern lies in performance-awareness, without which a sparsity metric can hardly be applied in scaling law studies.

Specifically, PPL-p% is a performance-aware improvement of the original CETT (so they have the same Pareto curve), which distinguishes the CETT value that just makes PPL increase by p%. The consistency of PPL increase makes it possible to compare models of distinct architectures as well as different checkpoints of the same architecture. If you use metrics such as CETT, Top-$k$, or FAT-$\epsilon$, a critical question is: what hyper-parameter (i.e., the CETT, $k$, and $\epsilon$) should you choose when you measure the sparsity of different models and checkpoints? After all, they can have very different intrinsic properties, and thus it is unreasonable to apply the same CETT, $k$, or $\epsilon$ for all models. This makes it a very tough problem to use these metrics in the scaling law study and model comparison. However, PPL-p% makes this question solvable. To compare the sparsity of different models under the same increase PPL is the most reasonable practice we can conduct. Of course, we can also apply a similar performance-aware extension to FAT-$\epsilon$ (with an unchanged Pareto curve), but this practice fails in the 3rd property mentioned above, as demonstrated in Figure 3. For Top-$k$ and FAT-$\epsilon$, specifically, their lower accuracies in recognizing weakly-contributed neurons make the sparsity level given by them questionable in whether they accurately reflect the activation pattern of models.

Generalizability across Different Metrics

The second key to your concern addresses the generalizability across different metrics. Through experiments, we obtain the following conclusion: The consistency of our power-laws between the activation ratio and data mainly depends on the PPL increase ratio, rather than the sparsity metric.

To support the above statement, we add experiments on FAT-$\epsilon$. Note that different $\epsilon$ can lead to different PPL increase ratios and results. Our results on 0.1B ReLU and 0.1B SiLU settings are shown in the figure (anonymous link). As demonstrated in the figure, for 0.1B ReLU, when the PPL increase ratio is relatively small (i.e., $\epsilon=0,0.3,0.6$), the activation ratio follows a convergent decreasing logspace power-law with the amount of data, which is consistent with our paper. When the PPL increases considerably (i.e., $\epsilon=1.0$), the power-law does not hold any longer. Similarly, for 0.1B SiLU, the convergent increasing power-law holds when PPL increases slightly (i.e., $\epsilon=0.1,0.2$), while the laws are broken with a large PPL increase (i.e., $\epsilon=0.275,0.5$). Besides, in another figure (anonymous link), we demonstrate the consistency of our power-laws for different scales of ReLU models when $\epsilon=0.3$. The weak correlation between activation ratios and parameter numbers, as well as the faster convergence of smaller models, also holds according to the same figure.

Therefore, for both metrics including PPL-$p\%$ and FAT-$\epsilon$, under different numbers of parameters, our power-laws hold when the PPL does not increase considerably (e.g., below 5%), while not the case with a large PPL increase ratio (e.g., around 10%). Obviously, the former case is more reasonable and helpful, as a considerably larger PPL can harm the model's performance. This provides our insights with more substantiated practical value.

We have conducted comprehensive experiments and invested considerable computation resources to address the primary concerns you raised. We are looking forward to your response, which is really important to us.