Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Works in "Essential References Not Discussed"

Thank you for pointing out these two works! They are both related to our paper. We discuss these works here, and will cite them in our future versions.

CATS [1] seems to be a post-training sparsification method based on thresholds. However, after carefully reading the paper and codes, we find that CATS is exactly equivalent to the Top-$k$ setting in our experiments in Section 4.1 of our paper. Specifically, for each FFN layer, CATS will find a corresponding cutoff threshold that just controls the sparsity ratio at a desired level. The activations with smaller absolute values will be dropped. Therefore, this is the same as Top-$k$, which reserves the same ratio of activation values for each layer according to the absolute values.

TEAL [2] is a more interesting work. It proposes a completely different paradigm of activation sparsity. Concretely, TEAL focuses on the input sparsity, indicating that for each vector-matrix computation $\mathbf{x}\mathbf{W}^T$, the values at some positions of $\mathbf{x}$ are small, and computation corresponding to these positions can be skipped for acceleration. By contrast, our work focuses on output sparsity, indicating the highly sparse patterns in the output $\sigma(\mathbf{x}\mathbf{W}_{gate}^T)$ of FFN activation functions. Since these two paradigms have completely different definitions of activation sparsity, the sparsity ratios cannot be trivially compared, and the properties (e.g., scaling laws) of them are also different. We are willing to open a new work to study such input sparsity in the future.

Lack of Formal Proofs in "Theoretical Claims"

Thank you for pointing out the lack of formal math proofs in our work. We believe that more strict theoretical works are valuable and indispensable in the future. However, at present, LLMs are black-box systems, which are extremely difficult to interpret theoretically. Instead, mainstream works find empirical laws using statistical methods to make the LLM behavior more predictable. Strict math models are hard to build, as the huge parameter scales make LLMs a highly complex system. The study of human brains also encounters similar challenges, and the statistical analysis of signals like brain waves becomes the mainstream paradigm. Still, we admit the value of more rigorous and formal math modeling and will make our analyses more convincing.

Question 1

Considering the general ability, the PPL used to measure sparsity is evaluated on a general validation set with the same distribution as the training set, covering a wide variety of corpus. However, a lower average PPL on the general data cannot necessarily ensure better performance on a specific category of downstream tasks. In Table 1, the performance of reading comprehension consistently drops with an increasing PPL, but the scores of commonsense reasoning fluctuate. Such a phenomenon will be more significant for those tasks accounting for a smaller part of the training data.

Question 2

Some coefficients in our activation-data power-laws can potentially be extrapolated to larger models.

First, the limit activation ratio $A_0$ is clearly weakly correlated to the parameter scale. This is also an important finding already stated in this article.

Besides, by Table 2, for ReLU-activated models, the coefficient $\alpha$ monotonically increases, while $c$ monotonically decreases with the model scale. Coefficient $b$ seems to have little value fluctuation when the model scale is larger than 0.4B. These indicate that there probably exist quantitative relationships between these coefficients and the model scale, or in other words, we may incorporate the model scale into our scaling law. However, finding a well-fit law including the model scale is too expensive, as dozens of scales of models should be trained for data preparation. Even the most famous work on scaling laws by OpenAI [3] did not cover model scales larger than 2B.

References

[1] Lee, Donghyun, et al. "CATS: Contextually-aware thresholding for sparsity in large language models." arXiv preprint arXiv:2404.08763 (2024).

[2] Liu, James, et al. "Training-free activation sparsity in large language models." arXiv preprint arXiv:2408.14690 (2024).

[3] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Practical Acceleration using Activation Sparsity

In Section 6, we present an acceleration experiment based on our 2.4B sparsely-activated model, achieving a 4.1$\times$ speedup ratio. This is conducted on the machine with 1 NVIDIA A800 GPU and PowerInfer, a SOTA sparsity-based framework. As pointed out by the reviewer, there may exist gap between this setting and real-world application scenarios. Therefore, we conduct new acceleration experiments under the following setting: (1) We use the device "NVIDIA Jetson Orin NX", a real-world representative end-side device tailored for AI. (2) We combine activation sparsity with Q4 quantization, a mainstream acceleration technique.

The decoding speeds (token/sec) of the 2.4B model on NVIDIA Jetson Orin NX are shown in the following table:

Using activation sparsity, we achieve a considerable speedup compared to the dense setting. Moreover, activation sparsity can also be combined with quantization and achieve an even higher decoding speedup.

Finally, we admit that the combination of activation sparsity and some other acceleration methods is non-trivial. For example, we are already working on its combination with speculative decoding. In this case, we use a new auxiliary loss to increase the similarity between activation patterns of neighbor tokens. With such modification, we are able to effectively utilize activation sparsity to further promote the efficacy of speculative decoding.

Contributions of This Work

Our work is not a mere extension of previous work CETT. Instead, we expect this work to be the foundation of future works involving the measurement, analysis, and training of sparsely-activated LLMs. Specifically, we give answers to three important questions:

• How can activation sparsity be measured more accurately?

We present a new perspective on activation sparsity: sparsity is the function of performance and must be measured under a specific tolerance of performance drop (i.e., PPL increase). Most existing works focus on ReLU-based activation sparsity, considering sparsity a fixed value with no connection to performance. After that, "ReLU$^2$ Wins" proposes CETT [1], enabling us to evaluate the sparsity of non-ReLU LLMs, as sparsity is considered a function of CETT.

Admittedly, in terms of methodology, our CETT-PPL-1% mainly introduces a hyper-parameter search process, finding an appropriate CETT value under a PPL increase ratio. This conversion is simple but important, as sparsity is finally linked to performance, and we are able to inspect the sparsity of LLMs under a specific expectation of performance. After all, it is more intuitive and reasonable to measure sparsity from the lens of performance rather than the ambiguous CETT. Otherwise, it will be complicated to compare the sparsity of models with different architectures (e.g., measuring the sparsity at multiple CETT points and comparing the Pareto curves).

• How is activation sparsity affected by the model architecture and training process?

We present a systematic quantitative analysis of the influential factors of activation sparsity, including the activation function, amount of training data, parameter scale, and width-depth ratio. Analytical scaling laws are also found between sparsity and the amount of training data under ReLU and SiLU. These findings are the basis of the third question.

• How can we build a more sparsely-activated and efficient LLM?

As the most important long-term contribution of this work, based on the above findings, we propose the better approach to obtaining more sparsely-activated LLM: Use ReLU as the activation function with a larger amount of pre-training data, and a small width-depth ratio within the interval ensuring the training stability. We also pre-train a 2.4B model from scratch, with an extremely low activation ratio of 6.48%, to re-validate our findings. Our work can provide instructional values for designing and pre-training an LLM with greater activation sparsity, which helps produce more efficient LLMs.

References

[1] Zhang, Zhengyan, et al. "ReLU$^ 2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs." arXiv preprint arXiv:2402.03804 (2024).

Thank you for your excellent review. These will encourage us to further improve the quality of our work and continuously forge ahead on the research path.

Ablation Studies in "Claims And Evidence"

Thank you for reminding us of the ablation studies on the 2.4B model. As it is really expensive to pre-train a new 2.4B model from scratch, we'd like to re-state the results on smaller models for each ablation factor. Most of these are already presented in the current manuscript.

As mentioned in Section 6, we consider 3 factors: (1) ReLU as the activation function; (2) a larger amount of training data; (3) a small width-depth ratio within the interval ensuring the training stability.

Activation Function

For each size among "0.1B, 0.2B, 0.4B, 0.8B, 1.2B", as shown in Figure 7, replacing ReLU with SiLU can cause a considerable increase in the activation ratio by more than 30%, indicating worse sparsity. Note that these two activation functions do not have significant performance differences by task performance (Table 1) and training loss (Figure 14).

Specifically, we present the limit activation ratio (%) of the 0.1B/0.2B models with different activation functions in the following table. Clearly, ReLU has significantly higher sparsity than commonly used SiLU and GELU.

Amount of Training Data

For each size among "0.1B, 0.2B, 0.4B, 0.8B, 1.2B", as shown in Figure 4, the ReLU-activated models always display lower activation ratios (i.e., higher sparsity ratios) with the increase in the amount of training data. This fact is re-validated in the 2.4B model. Figure 11 indicates that 2.4B has a similar negative relationship between the activation ratio and the amount of training data. Therefore, a decrease in the amount of training data can cause a sparsity drop for ReLU-activated models.

Width-Depth Ratio

We conduct a systematic study on the relationship between sparsity and the width-depth ratio in Section 5.2. As shown in Figure 5 and Figure 6, for the 0.1B ReLU-activated model, the limit of activation ratios linearly increases with the width-depth ratio before a bottleneck point 114, and the lowest training loss with training stability can be achieved within the interval [74, 282]. The 2.4B model as well as other settings from 0.2B to 1.2B all adopt a width-depth ratio close to the smallest point of the training stability interval. In 0.1B experiments, this setting is demonstrated to have the lowest activation ratio under the premise of training stability. A smaller width-depth ratio than this can cause training instability and worse performance, and a larger one brings about a sparsity drop.