We thank the reviewer for the close reading of our paper and the detailed comments. We have made most of the suggestions you proposed especially formatting related and provided an updated pdf. We address your concerns below:

Out of date baselines

Below we provide the new baseline (FALCON [1]) and include metrics for retraining with unstructured sparsity on on CIFAR-10 and MiniImageNet-1k. We provide the raw-one shot performance and performance after retraining. We retrain for 90 epochs on CIFAR-10 and 5 epochs on ImageNet-1k.

Note: for the retraining experiments, we retrain on the full ImageNet-1k dataset, as opposed to the sample of 3k samples from MiniImageNet-1k we used in our one-shot pruning experiments.

Also, note that FALCON does not actually prune to the target sparsity level given. It's sparsity is less (more dense) than the amount shown, since it uses a flop-aware formulation. As a result, these results are conservative with respect to our outperformance of FALCON.

ResNet20 on CIFAR10

MobileNetV1 on ImageNet-1K

We remark that though LayerMerge [2] is a good pruning technique, it should not be considered a one-shot pruning method, since the method has close to 10% accuracy following pruning (cf. Figure 3 of their paper), and requires extensive retraining to recover performance (they use 180 epochs of retraining on ImageNet-1k).

[1] Meng, X., Chen, W., Benbaki, R., & Mazumder, R. (2024). FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS).

[2] LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging, International Conference on Machine Learning, 2024.

We thank the reviewer for your positive review and comments. We found your questions to be relevant and useful for improving our work.

Motivation for the SNOWS loss function

We thank the reviewer for the question regarding the motivation of including subsequent operations $f^{\ell: \ell+K}$ in the reconstruction loss. Indeed, your intuition is correct that the motivation is in a way to minimize the residuals of the pruned network, relative to the dense network. To give you more intuition, we can return to the original motivation of the layer-wise reconstruction objective given in [1]. In their paper, they show that the error $\tilde{\varepsilon}^L = \frac{1}{\sqrt{n}} || \boldsymbol{Y}^L - \widehat{\boldsymbol{Y}}^L||_F$ of the entire pruned network at the final layer $L$ is bounded as:

$$\tilde{\varepsilon}^L \leq \sum_{\ell=1}^{L-1} \left( \prod_{r=\ell+1}^L \left| \left |\widehat{\boldsymbol{W}}^r \right|\right|_F \sqrt{\delta E^\ell} \right) + \sqrt{\delta E^L}$$

where $\delta E^\ell$ is the layer-wise error between the outputs at layer $\ell$, i.e., $\delta E^\ell = \frac{1}{n} \left| \left| \boldsymbol{X}^\ell \boldsymbol{W}^\ell - \widehat{\boldsymbol{X}}^\ell \widehat{\boldsymbol{W}}^\ell \right|\right|_F^2$. In other words, the difference between the final output of the dense and pruned network is bounded by a product of the layer-wise errors. This means if the layer-wise errors can be kept small, the overall error should be controlled.

However, there is an interesting subtlety to how the errors propagate. Say we are pruning layer $\ell$. It would seem as though the contribution by layer $\ell$ to $\tilde{\varepsilon}^L$ is confined to $\delta E^\ell$. However, notice that

$$\widehat{\boldsymbol{X}}^{\ell+1} = \widehat{\boldsymbol{Y}}^{\ell} = f^{\ell}(\widehat{\boldsymbol{X}}^{\ell}, \widehat{\boldsymbol{W}}^{\ell})$$

So pruning at the current layer changes the input at the next layer and moreover at all subsequent layers since $\widehat{\boldsymbol{X}}^{\ell+k} = \widehat{\boldsymbol{Y}}^{\ell+k-1} = f^{\ell+k-1}(\widehat{\boldsymbol{X}}^{\ell+k-1}, \widehat{\boldsymbol{W}}^{\ell+k-1})$, for all $k = 1, \dots, L - \ell$.

So after pruning layer $\ell$, we can observe the expression for the error in the subsequent layers

$$\delta E^{\ell+k} = \frac{1}{n} \left| \left| \boldsymbol{X}^{\ell+k} \boldsymbol{W}^{\ell+k} - \widehat{\boldsymbol{X}}^{\ell+k} \widehat{\boldsymbol{W}}^{\ell+k} \right| \right|_F^2$$

At the time of pruning layer $\ell$, $\boldsymbol{W}^{\ell+k} = \widehat{\boldsymbol{W}}^{\ell+k}$, since these weights have not yet been pruned, but notice that already $\boldsymbol{X}^{\ell+k} \neq \widehat{\boldsymbol{X}}^{\ell+k}$, since our pruning decision at layer $\ell$ has affected all layers after it.

The error term for the future layers becomes

$$\delta E^{\ell+k} = \frac{1}{n} \left| \left|(\boldsymbol{X}^{\ell+k} - \widehat{\boldsymbol{X}}^{\ell+k}) \boldsymbol{W}^{\ell+k} \right|\right|_F^2$$

Hence the error at the future layers is determined by the magnitude of the difference $\boldsymbol{X}^{\ell+k} - \widehat{\boldsymbol{X}}^{\ell+k}$ induced by pruning layer $\ell$.

Our formulation, by minimizing

$\sum_{k=0}^K| | \mathbf{Y}^{\ell+k} -\widehat{\mathbf{Y}}^{\ell+k}||_F^2$

$= \sum_{k=0}^K||\mathbf{X}^{\ell+k+1} -\widehat{\mathbf{X}}^{\ell+k+1}||_F^2$

can be interpreted as directly minimizing the subsequent error terms $\sum_{k=0}^K \delta E^{\ell+k}$ during the pruning of layer $\ell$ as opposed to only $\delta E^\ell$.

[1] Dong, X., Chen, S., & Pan, S. J. (2017). Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon. In Advances in Neural Information Processing Systems

The effect of $K$ on run-time

With regards to the ablations on the parameter $K$, we provide below a comparison of (1) run-time for pruning the entire network and (2) peak GPU memory usage, as the parameter $K$ varies for ResNet20 (R20) on CIFAR-10 and ResNet50 (R50) on ImageNet-1k. We have found this a useful ablation study and will include the comparisons in the final version of the paper. Our guidance is to choose the parameter $K$ to be as large as possible

Table 1: Memory and run-time using different $K$ in the SNOWS loss function.

Moreover, we centralize the values of $K$ used in all experiments with other hyperparameters in a single table for ease of comparison:

Table 2: $K$-step values, dampening factors, and CG tolerances used for $N$:$M$ sparsity and unstructured sparsity experiments across different architectures.

The reviewer's claim that [9] achieves 90% sparsity without accuracy loss

While it is true that [9] achieve 90% sparsity without dropping any clean performance for multiple architectures, these results are not directly comparable to the baselines provided in our work. This is because [9] uses ImageNet100, a subset of ImageNet with only 100 classes, which makes it easier for large networks to maintain good accuracy even at high sparsity levels after retraining. To verify this, we replicate the dataset from [9] with the exact classes they use and prune and retrain using SNOWS. We utilize the open-access ResNet18 provided by PyTorch, which is the same architecture used in their paper. Starting from a baseline accuracy of 88.36%, we prune the network to various sparsity levels and retrain for 50 epochs. The results are summarized in the Table below. As shown, even at 90% sparsity, the network attains an accuracy of 88.96% (an improvement of +0.6% over the dense model), and at 95% sparsity, the accuracy is 84.84% (a drop of -3.52%). The primary reason pruning works so effectively here is because the network is very large (11.7m parameters) relative to the number of classes since this is a small subset of ImageNet. In contrast, the networks we and other benchmarks used in our one-shot pruning experiments are generally closer to capacity on the datasets we prune on. For example, the ResNet20 we use on CIFAR-10 has 250k parameters and the MobileNet we use for the entire 1000 ImageNet classes has 4m parameters (less than half used for 100 classes in [9]).

Table 1: Pruning with SNOWS and Retraining with ResNet18 on ImageNet100, following the setup in [9].

[9] Hoffmann, J., Agnihotri, S., Saikia, T., & Brox, T. (2021). Towards improving robustness of compressed CNNs. In Proceedings of the ICML Workshop on Uncertainty and Robustness in Deep Learning

Run-time of our method

The run-time of our method depends mostly on the value of $K$ taken in the loss function. Below we show the run-time and peak memory consumption as the parameter $K$ varies for ResNet20 (R20) on CIFAR-10 and ResNet50 (R50) on CIFAR100 and ImageNet-1k.

Table 2: Memory and run-time using different $K$ in the SNOWS loss function.

Our guidance is to choose the parameter $K$ to be as large as possible for best performance. However, we have found that even small values of $K$ can outperform prior methods, while having faster run-times. Below we compare the runtime of our algorithm to the fastest (and best performing) competitor from our previous experiments, CHITA [6]. We remark that other competitors WoodFisher and CBS require hours to prune even MobileNet with a small sample size (see section 4.1.1 of [6]). We show below the run-time for pruning three networks of different sizes: ResNet20 (250k parameters), MobileNet (4m parameters), and ResNet50 (25m parameters). The main advantage of our method in terms of efficiency is scaling to large networks since we avoid computing, inverting, or storing the full Hessian or its approximation, and moreover not even storing or inverting the smaller layer-wise Hessian. Notably, our method outperforms CHITA even with moderate values of $K$ (though larger values used in our final experiments can obtain the best results). As shown in Table 1, SNOWS has just a slight run-time advantage for small networks but is more than 20 times as fast for ResNet50 which has 25m parameters, while maintaining a clear performance advantage even at small-moderate $K$.

Table 3: Run-time comparison of SNOWS and CHITA across different architectures pruning to 70% sparsity. For SNOWS, we use $K=10, 3, 3$ respectively for the networks shown. ResNet-20 is run on CIFAR-10, MobileNet on ImageNet-1k, and ResNet50 on CIFAR-100.

Note: we use 1000 samples for ResNet20 and MobileNet, but for ResNet50 we use 500 samples since this was the only sample size below which CHITA runs in $<$5 hours.

We thank the reviewer for the feedback on our paper. We appreciate your insights and addressed your concerns below:

Definition of one-shot pruning

In our introduction, we refer to one-shot pruning as "approaches that attempt to maintain network performance without requiring retraining." Later, in our section on related work, we provide more detail saying "Motivated by the cost of retraining, there has been a rise in methods for the post-training, one-shot pruning problem. In this setting, one typically assumes access to a limited sample of calibration data (e.g. a few thousand examples) and the goal is to compress the network with minimal accuracy loss relative to the dense model."

We now cite several works that share similar definitions of one-shot pruning

Optimal Brain Compression (OBC) [1] write "An alternative but challenging scenario is the post-training compression setup, in which we are given a trained but uncompressed model, together with a small amount of calibration data, and must produce an accurate compressed model in one shot, i.e., a single compression step, without retraining, and with limited computational costs."

SparseGPT [2] define one-shot pruning as: "compressing the model without retraining" and write "Post-training pruning is a practical scenario where we are given a well-optimized model $\theta^{\star}$, together with some calibration data, and must obtain a compressed (e.g., sparse and/or quantized) version of $\theta^{\star}$"

WoodFisher [3] define "One-shot pruning, in which the model has to be compressed in a single step, without any retraining"

We remark that these are only a handful of the most cited pruning papers but there are several others with similar definitions e.g. [4, 5, 6, 7].

The motivation behind one-shot pruning

We have no doubt that fine-tuning, retraining, or gradual pruning approaches can improve performance, as has been shown by the paper you provided and others e.g. [8], as well as the experiments we provide here in our response. However, this is typically considered a different setting where one has access to the resources required to compute gradients of the full model. The ImageNet dataset is 150GB in size, with over 1.28m images. We are able to accurately prune on this dataset with just 3k samples for ResNet50, and up to 20k for the VIT models, all using a single A100 GPU. Thus one-shot pruning approaches are well-motivated by the need for practitioners to prune without excessive resources. Similarly, a more accurate model requires less retraining and can be retrained to higher performance, as we show in our additional retraining experiments (see our response to reviewer TGxm "Whether retrainining with SNOWS improves performance").

[1] Frantar, E., Singh, S. P., & Alistarh, D. (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems

[2] Frantar, E., & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning

[3] Singh, S. P., & Alistarh, D. (2020). WoodFisher: Efficient Second-Order Approximation for Neural Network Compression. In Advances in Neural Information Processing Systems

[4] Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., & Gholami, A. (2022). A Fast Post-Training Pruning Framework for Transformers. In Advances in Neural Information Processing Systems

[5] Meng, X., Behdin, K., Wang, H., & Mazumder, R. (2024). ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models

[6] Benbaki, R., Chen, W., Meng, X., Hazimeh, H., Ponomareva, N., Zhao, Z., & Mazumder, R. (2023). Fast as CHITA: Neural Network Pruning with Combinatorial Optimization. In Proceedings of the 40th International Conference on Machine Learning

[7] Frantar, E., Kurtic, E., & Alistarh, D. (2021). M-FAC: Efficient Matrix-Free Approximations of Second-Order Information. In Advances in Neural Information Processing Systems

[8] Le, D. H., & Hua, B.-S. (2021). Network Pruning That Matters: A Case Study on Retraining Variants. In Proceedings of the 9th International Conference on Learning Representations

Dear Reviewer 96mw,

Thank you for taking the time to go through our responses individually. We very much appreciate it.

We wanted to respond as quickly as possible here to give the reviewers a chance to respond before the discussion periods ends. However, we will be finalizing the manuscript including adding new evaluations before the end of the discussion period.

Many thanks again for your time,

Authors

Dear Authors,

Thank you very much for your efforts and detailed responses.

I believe there to be some misunderstanding here when saying, "In our introduction, we refer to one-shot pruning as "approaches that attempt to maintain network performance without requiring retraining." Later, in our section on related work, we provide more detail saying "Motivated by the cost of retraining, there has been a rise in methods for the post-training, one-shot pruning problem."

The definition of one-shot pruning provided by me is indeed correct. Pruning can be either iterative, that is done it multiple shots: first prune 10% and re-train, again prune 10% and re-train, and again prune 10% and re-train. and so on, this is done lets say 5 times could give a model that is 0.90.90.90.90.9*100 % dense, that is ~59% dense which is 41% sparse compared to the original dense model. Or pruning can be done in a single shot, that is to obtain 41% sparsity, the model is pruned 41% in a single go. Now, most single-shot pruning methods require some retraining or finetuning after the one shot of pruning. However, this finetuning or retraining can turn out to be expensive, especially for large models, therefore your cited SparseGPT, WoodFischer, and some newer pruning methods try to propose one-shot pruning methods that do not require any finetuning. When doing so, this is explicitly mentioned, as done by both the SparseGPT and WoodFischer paper. I believe this has been misunderstood and therefore misrepresented in this submitted paper.

To further quote both the papers, exactly and with context, in WoodFischer, when the paper says "The first is one-shot pruning, in which the model has to be compressed in a single step, without any re-training.", here "without any re-training" is them highlighting their contribution and not them defining one-shot pruning. And by explicitly mentioning that their method does not do re-training, they are acknowledging that one-shot methods by default do "pruning + re-training".

Next, again in the SparseGPT paper they write as the first line of their abstract, "We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy." Here, they emphasize on "without any retraining, at minimal loss of accuracy" because this is a notable contribution, since most one-shot pruning methods require retraining, however, their proposed one-shot method does not require re-training and still does not lose performance, making it a contribution so notable that they use it as the first line of their abstract. Moreover, in the SparseGPT paper, in the introduction, they write "By contrast, the only known one-shot baseline which easily extends to this scale, Magnitude Pruning (Hagiwara, 1994; Han et al., 2015), preserves accuracy only until 10% sparsity,". Here Haniwara, 1994 is [R1] and Han et al is [R2]. [R2] even in their abstract define their method as "Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections." Also, [R1] in their work, in Section 2.4 Algorithm: Write step 7 as "Find and remove the least important weight." and then write step 8 as "Retrain the network.".

In conclusion of point 1, I believe even the works SparseGPT and WoodFischer do not define one-shot pruning in itself to not have any retraining. Rather, they acknowledge that one-shot pruning methods have retraining, and propose their one-shot pruning methods to be unique as they can perform one-shot pruning without having to fine-tune. Therefore the definition of one-shot pruning in this work is wrong and inconsistent with previous literature and must be changed. (Major Concern)

Efficiency during pruning vs Efficiency during inference

We observe that you made these statements:

This round:

“Since the claim of SNOWS is a gain in latency, I would suggest including actual latency numbers in the "response to Reviewer YAxu ("Inference Speed-ups")" table.”

Previous round:

"Lastly, the major motivation of the proposed work is gain in speed when performing pruning (as the entire hessian need not be calculated), however these no exists latency comparison to other pruning methods.“

This makes us think there is perhaps some confusion here around our usage of the term “efficiency” in the paper.

We would like to clarify that the aim of SNOWs is to improve pruning efficiency. This should be contrasted with a statement that SNOWS results in latency speedups or “inference efficiency”. We do not make such a claim in our paper.

We would like to distinguish between two forms of efficiency, the first we claim in our paper and the second we do not:

(i) Efficiency during pruning: The main claim of our paper is a gain in pruning efficiency. By that we mean that due to our optimization framework (new K-step loss function and the Hessian-free Newton approach to solve it), we can get higher model performance for the same run-time. Our claim is a more efficient mathematical framework to perform pruning, which provides lower accuracy loss than prior pruning methods. Performance comparisons given in the main text highlight this contribution. Additionally, we show runtime for pruning is faster than the fastest competing algorithm we benchmark against (CHITA) in Table 8 for the same “one-shot without retraining” setup. We also provide ablation studies on the run-time in Table 3.

(ii) Efficiency during inference: Once a sparse network is obtained, different sparsity patterns can realize different inference-time speed-ups on target hardware. Unstructured sparsity does not lead to strong speed-ups during inference; hence, N:M sparsity has been proposed in the literature as a pattern that leads to tangible speed-ups [7]. Our algorithm is agnostic to the mask, but we report accuracy for our method on common N:M sparsity patterns (1:4 and 2:4). Speedups available via our approach will be the same speedups from any existing N:M pattern. Following other papers, we report FLOPs as a proxy for true inference-time speed-up, for example see [8] Table 1 and [9] Table 1, though this is not the focus of our work.

We are happy to provide any additional clarification on our contributions.

We thank the reviewer again for the discussion. We have updated the manuscript with the amended introduction and related work section, which we believe has strengthened the paper.

Best regards,

Authors

[1] Frantar, E., Singh, S. P., & Alistarh, D. (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems (NeurIPS).

[2] Liu, Z., Sun, M., Zhou, T., Huang, G., & Darrell, T. (2019). Rethinking the Value of Network Pruning. In International Conference on Learning Representations (ICLR).

[3] Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations (ICLR).

[4] Hagiwara, Masafumi (1994). "A simple and effective method for removal of hidden units and weights." Neurocomputing 6, no. 2: 207-218.

[5] Frantar, E., & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning (ICML).

[6] Singh, S. P., & Alistarh, D. (2020). WoodFisher: Efficient Second-Order Approximation for Neural Network Compression. In Advances in Neural Information Processing Systems (NeurIPS).

[7] Sun, Y., Guo, H., Ning, S., & Yu, X. (2022). Accelerating Sparse Deep Neural Network Inference Using GPU Tensor Cores. Proceedings of the 2022 IEEE High Performance Extreme Computing Conference (HPEC).

[8] Zhang, Y., Lin, M., Lin, Z., Luo, Y., Li, K., Chao, F., Wu, Y., & Ji, R. (2023). Learning Best Combination for Efficient N:M Sparsity. Proceedings of the International Conference on Machine Learning.

[9] Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., & Li, H. (2021). Learning N:M Fine-Grained Structured Sparse Neural Networks from Scratch. Proceedings of the International Conference on Learning Representations (ICLR).

We thank the reviewer for reading our paper and in particular for noticing two typos. We have amended these typos in the main text.

Request for clarification on experiments

The reviewers strengths of the paper are that "The experiment setup is well-explained and shown in a detailed manner." while their first weakness includes "leave more space to explain the experiment section in the main paper." We would appreciate if the reviewer could provide more detail on exactly what remains to be explained in the experiment section.

Limitations and conclusions

While we were limited by space in the main paper, we have decided to add a limitations and conclusions section in the supplementary section. We thank the reviewer for this suggestion.

This work proposes a new layer-wise pruning framework, acting as a bridge between the layer-wise and global pruning loss functions introduced in prior work. The objective of our modified loss function induces a harder non-linear optimization problem compared to the standard layer-wise formulation. We propose an efficient Hessian-free method for optimizing this objective, and demonstrate its efficiency on large-scale networks. There are several limitations to the proposed work: firstly, SNOWS does not do mask selection. This makes it flexible since it can integrate with existing techniques for mask selection, but we believe using discrete optimization (as is done in ([1], [2]) on the SNOWS loss function may lead to stronger results. Moreover, the method has little dependence on the computational modules involved in the reconstruction loss. As such, the framework could readily be extended to Text-to-Speech or Large Language Models. However, this would require further work to specialize the formulation of the $K$-step problem as we did for e.g. VIT modules.

[1] Meng, X., Chen, W., Benbaki, R., & Mazumder, R. (2024). FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS).

[2] Benbaki, R., Chen, W., Meng, X., Hazimeh, H., Ponomareva, N., Zhao, Z., & Mazumder, R. (2023). Fast as CHITA: Neural Network Pruning with Combinatorial Optimization. In Proceedings of the 40th International Conference on Machine Learning.

We sincerely thank the reviewer for their positive and constructive feedback on our work. We address your questions and comments below.

True Hessian Information versus Fisher approximation

We ran some additional comparisons, replacing the true Hessian with the Fisher approximation, and comparing the objective value (Eqn 3 in our paper). Indeed, we do observe a performance improvement for using the true Hessian over the Fisher approximation. The main improvement in using the true Hessian via the Hessian-free approach is speed of convergence. Below we plot the objective value when using the Fisher approximation versus the exact Hessian-free steps for all layers in ResNet20 and the first 3 layers in ResNet50. We find that using Newton step as opposed to the Fisher approximation leads to much faster convergence (not just in iterations but in overall run-time). For the larger ResNet50 networks, running the Fisher approximation to convergence is actually prohibitive.

The figures can be viewed here:

ResNet20: https://github.com/anon-iclr-snows/SNOWS/blob/main/Rebutal-Figures/loss_plot_cifar10.png

ResNet50: https://github.com/anon-iclr-snows/SNOWS/blob/main/Rebutal-Figures/loss_plots_resnet50_layers.png

Why competing methods fail in some scenarios

This is generally due to either 1) very high sparsity and/or 2) a restrictive structured sparsity pattern. For example, in Figure 3 in the paper, the competing methods generally fail at 1:4 sparsity. This is a sparsity pattern in which at most 1 of every 4 weights in every contiguous block of 4 are retained. This induces both high sparsity and a restrictive pattern on the non-zeros, which leads to failure without weight readjustment. Intuitively, weight adjustments become more crucial as the sparsity level increases. In Table 1, VITs experience the same effect but fail at even lower sparsity than ResNet models.

Whether retrainining with SNOWS improves performance

Yes, we observe benefits to retraining even after pruning with SNOWS. Below we provide a new baseline (FALCON [1]) and include metrics for retraining with unstructured sparsity on on CIFAR-10 and ImageNet. We train for 90 epochs on CIFAR-10 and 5 epochs on ImageNet-1k. Retraining consistently improves performance across all sparsity levels. For example, on CIFAR-10 at 80% sparsity, retraining improves accuracy drop from -4.74% to just -1.50%.

Note: for the retraining experiments, we retrain on the full ImageNet-1k dataset, as opposed to the sample of 3k samples from MiniImageNet-1k we used in our initial experiments.

ResNet20 on CIFAR10

MobileNetV1 on ImageNet-1K

[1] Meng, X., Chen, W., Benbaki, R., & Mazumder, R. (2024). FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics