Optimal Brain Apoptosis

Question 4 L095 You mention that the Gauss.Newton approximation is in-sufficient what is the evidence?

That's a good question. In our importance calculation process, the importance score of a weight $\theta_i$ is the sum of all second-order derivatives multiplied with corresponding parameters $\sum_{j}\frac{\partial^2 \mathcal{L}}{\partial\theta_i\partial\theta_j}\delta\theta_i\delta\theta_j$. If we take the approximation, $\sum_{j}\frac{\partial^2 \mathcal{L}}{\partial\theta_i\partial\theta_j}\delta\theta_i\delta\theta_j$ would turn into $\sum_{j}\frac{\partial \mathcal{L}}{\partial\theta_i}\frac{\partial \mathcal{L}}{\partial\theta_j}\delta\theta_i\delta\theta_j=\sum_{j}\frac{\partial \mathcal{L}}{\partial\theta_j}\delta\theta_j\frac{\partial \mathcal{L}}{\partial\theta_i}\delta\theta_i = c \frac{\partial \mathcal{L}}{\partial\theta_i}\delta\theta_i$, where c is a constant value. This importance score is same to that of Taylor method, which is outperformed by OBA in nearly all results.

Question 5 L087 Do you keep track of the first order term (which is assumed zero in OBD/OBS)?

We don't keep track of the first order term in our algorithm.

We appreciate the valuable suggestions and efforts from the reviewer!

Weakness 1 The field is very busy with many results going back to the early 90s. Computational results and simplifying structure of the Hessian for feedforward network are found in many papers including • Wille, J., 1997, June. On the structure of the Hessian matrix in feedforward networks and second derivative methods. In Proceedings of International Conference on Neural Networks (ICNN'97) (Vol. 3, pp. 1851-1855). IEEE. • Buntine, W.L. and Weigend, A.S., 1994. Computing second derivatives in feed-forward networks: A review. IEEE transactions on Neural Networks, 5(3), pp.480-488. • Wu, Y., Zhu, X., Wu, C., Wang, A. and Ge, R., 2020. Dissecting hessian: Understanding common structure of hessian in neural networks. arXiv preprint arXiv:2010.04261 • Singh, S.P., Bachmann, G. and Hofmann, T., 2021. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems, 34, pp.23914-23927.

Thanks for pointing out these literatures that build the ground of the Hessian in the neural network. We've added them to our related work, merging the Hessian Matrix part into section 2 (preliminary) as part of the main paper. For more details please refer to section 2 of our revised manuscript.

Weakness 2 The paper presents a reduced complexity calculation of Hessian x vector, this is well-known - see work by Barak Pearlmutter for example; Pearlmutter, B.A., 1994. Fast exact multiplication by the Hessian. Neural computation, 6(1), pp.147-160.

We are very grateful for your introduction to this previously unknown work! While our research shares a fundamental concept with "Fast Exact Multiplication by the Hessian," the specifics of our approaches differ significantly. The work in 1994 introduces a differential operator, $\mathcal{R}v({f(\boldsymbol{w})}) = \left.(\partial / \partial r) f(\mathbf{w}+r \mathbf{v})\right|{r=0}$, for deriving second-order derivatives in single fully connected layer, recurrent layer, and Stochastic Boltzmann Machine. However, the use of single-layer networks has become quite rare in contemporary times. Our focus is on the more complex Hessian submatrices across layers. Our method, OBA, facilitates the computation of the Hessian-vector product in multi-layer networks, which are the predominant form of neural networks today, marking a clear departure from the techniques used in "Fast Exact Multiplication by the Hessian".

Question 1 Empirical evidence: While the results for CNNs are impressive, please consider transformers. There is much interest in pruning and redundancy in transformers e.g. a. Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X. and Chen, W., 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853. b. Lad, V., Gurnee, W. and Tegmark, M., 2024. The Remarkable Robustness of LLMs: Stages of Inference?. arXiv preprint arXiv:2406.19384.

Thank you for your recommendation! We have indeed performed experiments on ViT-B/16 and achieved promising results compared to magnitude pruning and first-order Taylor approximation (see Table 2). It would be worthwhile to conduct further experiments to test its effectiveness on other architectures such as BERT and GPT in future studies.

Question 2 L652 Consider setting the scene, motivation and related work as an integral part of the paper. With due refence to early work on pruning and Hessian structure

Thanks! We've revised the paper according to your suggestions. Please see the section 2 of our revised paper.

Question 3 L058-9 The OBS method is only different from OBD when the non-diagonal Hessian is used, i.e they do not ignore the off-diagonal terms in OBS

We apologize for the misunderstanding writing. OBS does not discard the second-order derivative information but rather approximates them with the Fisher information matrix. We've revised the content as follows: They either discard or approximate the second-order partial derivatives between all pairs of parameters, which capture the change of loss on one parameter when deleting another parameter.

Dear reviewer fJfY,

As the discussion period is nearing its end, we would like to know if you have any additional concerns regarding our paper. If so, we are happy to address them before the period concludes.

Best,

Authors

Dear reviewer fJfY,

we would like to summarize our opinions regarding the first-order term as follows.

1. The reason for not incorporating the first order term is to ensure fair comparisons with other Hessian-based methods, including OBD [1], OBS [2], EigenDamage [3], et al. They do not utilize the first-order term, either.

2. In our experiments, we found that the second-order term's average magnitude is approximately ten times greater than that of the first-order term per layer, indicating that the first-order term has minimal influence on the calculation of our importance score.

3. Through further experiments, as shown in the table presented in our "Reply to Reviewer fJfY (first order term)", we show that including the first-order term does not have a clear influence on the performance of the pruned network.

4. All our experiments do not leverage large regularization. Before pruning, all networks are fully trained into local minima, theoretically supporting the fact of the small first-order term.

The first-order term can be added back in other conditions by minimal code adjustment. Under our experiment settings, not incorporating it is valid and fair. We hope our response could resolve your concerns regarding the first-order term.

Best,

Authors

[1] LeCun, Yann, John Denker, and Sara Solla. "Optimal brain damage." Advances in neural information processing systems 2 (1989).

[2] Hassibi, Babak, David G. Stork, and Gregory J. Wolff. "Optimal brain surgeon and general network pruning." IEEE international conference on neural networks. IEEE, 1993.

[3] Wang, Chaoqi, et al. "Eigendamage: Structured pruning in the kronecker-factored eigenbasis." International conference on machine learning. PMLR, 2019.

Question 2 Minor suggestion. For LaTeX notations, the subscripts (especially subscripts of superscripts) can be wrapped in text format. What I mean is instead of $\mathbb R^{l_{out}}$, using $\mathbb R^{l_{\text{out}}}$ might give you a better looking symbol.

Thanks for your helpful suggestions. Wrapping these notations with "\text" do present a better looking. We've changed all these symbols into "\text{}" version in the revised manuscript.

Question 3 What do a, b, c, d, e respectively mean in Equation 3? Could the authors define them or point to the text where they are defined?

a, b, c, d, e represent the index of 5 dimensions, respectively the output channel/neuron dimension with length $l_{\text{out}}$, the flattened output feature size dimension with length $p_{\text{out}}$, the input channel/neuron dimension with length $l_{\text{in}}$, the flattened weight size for each input and output neuron/channel pair with length $p_{\text{weight}}$, and the flattened input feature size dimension with length $p_{\text{input}}$. You could refer to the beginning part of section 3.1 to better understand them with concrete examples.

We express our gratitude for your recognition on our work.

Weakness & Question 1 Since pruning is not my area of expertise, I am unsure whether the authors used fair baselines for comparison. The proposed method is mainly compared against 7 methods from 3 papers, respectively in 2016, 2017 and 2019. A simple literature search gave me a few methods that claim to have achieved better pruning performances and are fairly well cited and fairly highly stared: https://arxiv.org/abs/2203.04248, https://arxiv.org/abs/2208.11580, https://arxiv.org/abs/2210.04092, https://arxiv.org/abs/2112.00029. I would encourage the authors to either compare to some of the latest works, or provide a justification on not considering these more recent works --- meanwhile, I would resort to reviewers with more experience in pruning on their opinions regarding this matter.

Thank you for providing additional related work in the field of pruning. Our paper primarily introduces a novel Hessian-vector product method, where the Hessian matrix represents the second-order term in the Taylor expansion of the loss function. As such, we mainly focus our comparisons on works that similarly utilize either the Hessian matrix or the first-order term of the Taylor expansion. The seven methods you highlighted are from Table 3, which presents structured pruning results. For unstructured pruning, our results are also compelling, as shown in Tables 4 and 5, particularly against CHITA [1].

In the paper you recommended, both "Dual Lottery Ticket Hypothesis" and "Advancing Model Pruning via Bi-level Optimization" build on the Lottery Ticket Hypothesis, which seeks to identify an effective subnetwork prior to training, a concept distinct from our approach. "Pixelated Butterfly" does not use Hessian or Taylor expansion information, thus it is not a relevant comparison for our work. "Optimal Brain Compression," which utilizes the Hessian matrix for pruning, is indeed worth comparing. However, we were unable to replicate the author's results in our implementation (Our GMP achieves 73.16% accuracy under the same conditions described in OBC, which is significantly lower than the reported 74.86%), and the time constraints of the rebuttal period prevent us from implementing OBA within the OBC structure. We have expanded our unstructured pruning results on CIFAR10 using Resnet20 and included CBS [2] and WoodFisher [3] in our comparisons.

As demonstrated, OBA significantly outperforms other methods at high sparsity levels, confirming its efficacy.

[1] Benbaki, Riade, et al. "Fast as chita: Neural network pruning with combinatorial optimization." In ICML 2023.

[2] Yu, Xin, et al. "The combinatorial brain surgeon: pruning weights that cancel one another in neural networks." In ICML 2022.

[3] Singh, Sidak Pal, and Dan Alistarh. "Woodfisher: Efficient second-order approximation for neural network compression." In NeurIPS 2020.

The authors provided a disciplined response to my prior question on why they did not include comparisons against certain more recent pruning methods, and I am satisfied with the answer. Under the current form of the paper, I do not see pressing reasons for a major rating increase from 6 marginal accept to 8 clear accept, and I would feel more comfortable keeping my original rating. Nevertheless, I acknowledge the extra effort spent by the authors and wish them all the best.

Thanks for your reognition on our work!

Weakness The improvements in accuracy and speed do not seem overwhelming.

We acknowledge your concerns regarding some of the results presented. Specifically, in Table 2, you will notice that our method, OBA, shows significantly better results for structured pruning on the ViT-B/16 model compared to other methods. In terms of unstructured pruning at high sparsity levels, as shown in Tables 4 and 5, OBA notably outperforms CHITA and other competing approaches. However, we realize that the superiority of our method is less pronounced in Figure 3 and Table 3, though it still maintains a marginal advantage over other methods. Thank you for bringing this to our attention.

Question Considering that recurrent neural networks use back-propagation through time, for a given time window, how tractable would be to extend your method to such networks?

In an RNN, the concept of time is incorporated, requiring an expansion of series and parallel connectivity in networks. For layers that do not convey temporal information, series and parallel connectivity are confined to individual time steps. For recurrent neurons that propagate information through time, such as the basic RNN neurons whose hidden state is updated as $h_t=\tanh \left(W_{h h} h_{t-1}+W_{x h} x_t+b_h\right)$ and $y_t=W_{h y} h_t+b_y,$ the output $y_{t'}$ at time $t'$ is dependent on $x_{\tau}$ where $\tau \leq t'$. This dependency causes layers from earlier time steps, up to $t'$, to be in series connectivity with all subsequent layers at time step $t'$. Furthermore, since the parameters $W_{hh}$, $W_{xh}$, $W_{yh}$ interact to generate $y_t$, their Hessian submatrix between them is non-zero and must be calculated. The Hessian computation for a State Space Model is analogous to that in a basic RNN.

For more intricate recurrent layers like LSTMs, which incorporate multiple gates that affect the internal state, the network displays extensive parallel connectivities. Computing the Hessian for such a structure is significantly more complex. Nonetheless, through a systematic and categorized approach, we can still effectively address the complexities introduced by these advanced neurons using OBA. These are directions worth exploring in the future.

We deeply appreciate your recognition on our work! We would like to reply your questions and weaknesses as follows.

Weakness 1 It seems like calculating the full Hessian-vector product, even with optimizations, can still be computationally expensive for larger and more complex networks.

You are right. We recognize that our method for calculating the Hessian-vector product is still quite time-consuming, taking twice as long as OBS, as shown in Figure 6b. However, the time it takes for OBA to prune is still relatively minor compared to the extensive time required for training or fine-tuning the model. For very large models such as LLaMA 3.1, BLOOM, and others, it may be practical to prune only specific layers, similarly to common fine-tuning practices. This approach significantly reduces complexity by limiting the calculation of the Hessian-vector product to just the selected layers.

Weakness 2 Extending this method to newer architectures seems to require additional work. The method was tested on a specific set of architectures, and its generalizability across a wider range of tasks or domains is yet to be fully established.

Thanks for pointing out this. OBA can be applied to various parameter layers such as fully connected, convolutional, and attention layers, which are essential to most contemporary models in computer vision and natural language processing. This versatility shows that OBA can be seamlessly integrated into widely used models without any modifications. Moreover, since the importance score acquisition of OBA only relies on the loss of the output, which is task-invariant, OBA can also be directly utilized in other tasks to prune models. Exploring the application of the Hessian-vector product in other architectures like RNNs, SSMs, and additional tasks will be valuable to further assess OBA’s adaptability and generalization capabilities in future research.

Weakness 3 The experiments make the paper feel somewhat outdated, as the evaluations and datasets used are reminiscent of those from 7-8 years ago.

We appreciate your feedback and sincerely apologize if our experiments gave the impression of being outdated. Our intention was to compare our method fairly with others that also innovate on the Taylor expansion term, specifically Eigen Damage (2019) and CHITA (2023). Eigen Damage is a structured pruning method, while CHITA focuses on unstructured pruning. Although CHITA is more recent, the models and datasets used in both studies are quite similar, which may have contributed to the perception of our experiments being less current.

For fairness, we conducted our experiments under the same settings as these studies, ensuring a direct and meaningful comparison. However, we understand how this might make the experiments appear outdated. Moving forward, we plan to explore the application of OBA on more contemporary datasets and models to demonstrate its broader applicability and relevance. Thank you for pointing this out, and we will address it in our future work.