The authors propose a novel method of deep neural network unstructured pruning (sparsity). They claim that the best-performing pruning techniques use second-order methods for importance estimation. However, due to the size of modern neural networks, these methods are computationally too expensive. To address this limitation, the authors introduce FishLeg surgeon (FLS). The core idea is to leverage an accumulation of the gradients instead of storing them individually. This is achieved through tensor decomposition for an effective approximation. The authors mainly evaluate the proposed method on ResNet 50 trained on ImageNet.

This paper proposes FishLeg Surgeon (FLS), a pruning technique that uses the FishLeg under the hood (an already existing approach to estimate Fisher Information Matrix (FIM) in neural networks, initially used for optimization) in the Optimal Brain Surgeon (OBS) framework to perform unstructured pruning, more precisely one-shot and gradual pruning. The experiments are performed on an autoencoder for MNIST and ResNet-50 on ImageNet.

The authors proposed a Optimal Brain Surgery pruning technique where importance is based on the Fisher information matrix. Following the FishLeg optimization due to Garcia et al., they proposed a specific parameterization of the inverse Fisher information matrix, as well as optimization procedures. They further demonstrated with experiments on unstructured network pruning.

This work proposes a novel pruning mechanism that uses the FishLeg optimizer, which is based on the the inverse fisher information matrix. This work proposes a number of improvements to the fishleg optimizer to make it more amenable for unstructured pruning, such as modeling the full FIM, as opposed to its action of a parameter subspace, as well as a preconditioner for the auxiliary loss. When applied to pruning, the authors show that there are improvements on various benchmarks, outperforming other second-order methods, and shows potential for network quantization applications.

General response to all reviewers

We thank all reviewers for their time and effort in reviewing our paper. We have identified in the four reviews two common themes, and we are addressing them here jointly. More specific comments/questions are addressed separately for each reviewer.

First Question: About broader evaluation process and comparison with State-of-the-art algorithm.

This work does not aim to compete with successful state-of-the-art pruning approaches that are usually architecture-dependent and rely on sophisticated recipes and distillation strategies, such as SparseGPT[1] or Interspace pruning [2]. Our method does not enter this level of specialization; instead, it focuses on improving second-order pruning using FishLeg. Indeed, our algorithm could be combined with any of the state-of-the-art methods cited above to enhance their pruning capabilities further.

Our experiments use the same benchmarks as the previous second-order pruning papers, such as Woodfisher et al. [3]. In this work, first, we develop the FishLeg surgeon algorithm (FLS): we show how we can use FishLeg to better approximate the Fisher matrix, which translates to better second-order importance scores and, consequently, better pruning. Therefore, in this paper, we use simple autoencoders to illustrate the theory behind FLS and to compare this algorithm with other second-order pruning methods. Then, we apply FLS to ResNet-50, the standard benchmark used in second-order pruning papers, showing improved performance (as shown in Figure 2 and Figure 3 of the paper) with lower computational cost (see next question).

[1] Elias Frantar, Dan Alistarh. "SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot".arXiv:2301.00774. 2023 [2] Wimmer, Paul, Jens Mehnert, and Alexandru Condurache. "Interspace pruning: Using adaptive filter representations to improve training of sparse cnns." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022 [3] Sidak Pal Singh, Dan Alistarh. "WoodFisher: Efficient Second-Order Approximation for Neural Network Compression". Part of Advances in Neural Information Processing Systems 33. 2020

Second Question: About the Superiority of FLS with respect to other second-order methods, memory usage and clock time.

Our experiments show that FLS outperforms the other second-order pruning methods, even if this improvement is small in some experiments. However, as noted by one of our reviewers, previous second-order approaches saw smaller performance gains over each other, and FishLeg has other advantages that complement the performance improvement:

• Implementation flexibility: FishLeg directly amortize $F^{-1}v$ products in an easy-to-evaluate $Q(\lambda)v$ form. This approach enables both (i) easy application of FishLeg to different NN architectures and (ii) the development of different approximations for each type of NN layer to trade off scalability and accuracy.

• Faster wall-clock time and lower memory consumption: the dense layer and convolutional layer parametrization proposed in this work present a good balance of scalability and accuracy. Below, we provide a time comparison between FLS, oBERT and M-FAC as a function of number of samples collected. In these results, all algorithms implement the block (size 50) approximation for a fair comparison. Time is recorded during the re-computation of the Fisher inverse using the corresponding number of collected samples with batch size 16.

  Time Complexity \ Sample Size	512	2048	10k	50k
  FLS	0.096	0.239	0.981	4.755
  oBERT	0.781	3.040	14.091	74.852
  M-FAC	0.188	2.785	13.591	67.649

  Memory costs of the three algorithms are shown below during inverse-Fisher(Hessian)-vector-product computations as a function of the number of parameters.

  Memory Cost \ Network Size	500	40k	250k
  FLS	911MB	913MB	937MB
  oBERT	983MB	7107MB	OOM
  M-FAC	981MB	1003MB	1033MB

  We will add these results to our supplementary materials.

In this paper, the authors use a novel formulation of the inverse Fisher information matrix to prune deep neural networks. They then develop a block-wise approximation, reminiscent of the KFAC approximation, to achieve an efficient algorithm for doing the pruning. In experiments on an autoencoder on MNIST and a RESNET-50 on Imagenet, the authors demonstrate better accuracy at a specified sparsity than baseline approaches (oBERT and M-FAC). The reviews were overall mixed but leaning reject, with scores of 6, 5, 3, 3. The reviewers found the approach well-motivated as making large deep networks more efficient is obviously valuable and interesting to the community. The reviewers also found the approach technically sensible. However, the reviewers didn't find the empirical evaluation of the method convincing, and some important citations/baselines were missing. The major issue seems to be in the empirical evaluation. The reviewers asked for experiments that demonstrate the efficiency of the method and, in particular, didn't find the scale of the problems presented particularly convincing. While it's understandable that not all researchers have access to resources to train and evaluate the largest contemporary models, I agree that the presented problems aren't particularly motivating. While LLM experiments aren't necessary, they could be helpful to motivate the work. It's not hard to find reasonably tractable open-source models and a variety of interesting benchmark metrics. This seems like an exciting start but, unfortunately, this paper doesn't seem quite ready for ICLR. Strengthening the empirical evaluation would go a long way to improving the impact of the paper for a future submission.

Reject