A Demon at Work: Leveraging Neuron Death for Efficient Neural Network Pruning

We thank the reviewer for the accurate summary of our work and their positive feedback on the soundess and usefulness of our approach and the simplicity and effectiveness of our method. We also appreciate their concrete suggestions to better articulate the motivation of our approach and to strengthen some of claims with further evidence.

Weaknesses:

1. On the motivation for accelerating neural death

Thank you for pointing out this confusing aspect of our paper. The motivation to accelerate and promote neural death is threefold:

• It alleviates the need for a score function and pruning intervention because the learning dynamics reach directly a sparser solution. It is also a notable difference from previous existing works. We now discuss further this point in the introduction.
• Actively promoting neural death and dynamically removing the dead units can lead to training speedup. We added training speedup results in Table 1 to better highlight this aspect.
• Because the updates are proportional to the loss, the update size diminishes as the model converges, significantly affecting the dying speed. Actively promoting neural death ensures that dying occurs in a fixed number of times steps. We added Appendix D1 to clearly expose this aspect.

2. Evidence that inactive neurons remain inactive

Thank you for pointing out this omission and the great suggestion. As per the reviewer’s suggestion, we complemented our previous results justifying dynamic pruning (now Appendix H1) with measurements of the overlap ratio in a new Appendix C (Fig. 8). We have added a reference to the latter to section 3.1.

3. On the role of noise

Thank you for highlighting the apparent inconsistency. Section 3.1 introduces — and tries to provide intuition behind — the driving hypothesis of our paper: Does the noise level during optimization affect the solution sparsity via dead neuron accumulation? Section 3.1 and Figure 2 show that SGD noise structure has the potential to amplify dead neuron accumulation, motivating further investigations. The comparison with Gaussian noise highlights the asymmetric difference between the two noise structures, an important aspect that is also part of our thought experiment.

We reworked thoroughly the intuition in section 3.1 and brought further clarification in the Figure 2 caption as well to better justify this section.

Minor:

1. Thank you, we clarified Figure 3.

Thank you for your thoughtful reconsideration and adjustment of our score.

We want to emphasize our agreement with the reviewer's point about the inherent sparsity-performance tradeoff. Achieving a favorable balance, where high sparsity levels are attained with minimal impact on performance, is a central challenge faced by all pruning methods. Our paper extensively evaluates and compares these tradeoffs, as illustrated in Figures 4-7. Our results demonstrate a more advantageous tradeoff for our method compared to the strongest competitors in the recent literature of structured pruning, despite the simplicity of our pruning criterion.

We appreciate the reference to another current submission at ICLR, but given the inherent variability in pruning methods' performance across different sparsity levels, a comprehensive quantitative comparison based on sparsity-performance curves would be needed for a fair assessment.

We thank the reviewer for their positive feedback on the presentation, soundness and novelty of our approach. More importantly, we thank the reviewer for their questions and concrete suggestions to clarify our contribution and improve the paper's presentation.

Questions:

1. Comparison with unstructured methods.

We appreciate the reviewer's valuable input regarding the comparison of DemP with unstructured pruning methods in Table 1. Our initial decision to include unstructured baselines was motivated by a desire for consistency with existing literature [1, 2], aiming to highlight the trade-off between structured and unstructured pruning approaches. However, we acknowledge the importance of contextualizing our comparisons within the domain of structured pruning.

To address this concern, we have revised our manuscript in Section 5 to explicitly clarify the rationale behind including unstructured baselines. Additionally, in response to the reviewer's observation, we have relocated Figure 6 to Appendix I to maintain focus on the main results. Importantly, we want to emphasize that the structured baselines chosen for comparison represent, to the best of our knowledge, the top-performing dynamics pruning algorithms available.

We refrained from introducing additional baselines that may offer subpar performance to prevent overcrowding the figures, ensuring a clear presentation of the most relevant and competitive comparisons. We believe these adjustments enhance the overall clarity and relevance of our comparisons, aligning more closely with the structured pruning context.

2. More speedup results

In line with the reviewer’s suggestion, we added in Table 1 extended results for training speedup as well as training/inference FLOPs for the ResNet-50 architecture on ImageNet. We intend to run analogous experiments for the smaller networks as well and add the corresponding results to the camera-ready version.

3. Difference with existing methods

Thank you for this important question. Dynamic pruning is an overloaded expression, that we use in accordance to how it is used in RiGL paper. It consists of methods that gradually prune neural nets to recover a sparse model at the end of training under a fixed trained budget. It contrasts with methods doing single-shot pruning at the end of training before fine-tuning, or with iterative magnitude pruning that repeats multiple cycles of single-shot pruning after initial complete training. Apart from the structured nature of DemP, further differences are:

• DemP acts on the learning dynamics to force the optimization process to generate a sparse solution. It does not require single or multiple pruning interventions during training.
• It alleviates the need to design a score function to identify the neurons to remove, monitoring the activations retrieved during the forward pass is sufficient.

We appreciate the opportunity to further highlight the originality of our approach. We made the necessary modifications in our paper’s introduction and in Section 4.

4. On the relation between DemP and the theorerical analysis

We appreciate this important feedback. Viewing the optimization process as akin to a biased random walk suggests that we can act on the learning dynamics to learn sparser solutions instead of relying on pruning intervention. DemP explores a single direction to affect the learning dynamics (regularization) while Section 3.2 was an attempt to further understand the impact of various hyperparameters on sparsity. We tried to clarify this connection by emphasizing this point of view in the introduction and section 4 and reworking the intuition. However we acknowledge that since the mathematical analysis is not central to the paper, it tends to distracts away from the main point of the paper. Following your suggestion, we recentered the discussion in section 3 on empirical evidence and moved the discussion of the Brownian motion model to Apppendix B, while incorporating a reference to it in the intuition subsection of Section 3.1. We appreciate the opportunity to improve the flow and clarity of this analysis section.

[1] John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, Stephan Günnemann: Winning the Lottery Ahead of Time: Efficient Early Network Pruning. ICML 2022: 18293-18309

[2] Stijn Verdenius, Maarten Stol, Patrick Forré: Pruning via Iterative Ranking of Sensitivity Statistics. CoRR abs/2006.00896 (2020)

We thank the reviewer for the accurate summary of our work and for recognizing the simplicity and efficiency of the method we propose. We also appreciate the questions and constructive feedback to improve the flow of the paper.

Novelty of Our Approach: We appreciate Reviewer K7GU's observation that the concepts of dead neurons and activation-based pruning methods have been explored in prior literature. However, we want to emphasize the distinctiveness of our contribution. To the best of our knowledge, our work is the first to demonstrate the viability and effectiveness of pruning based solely on units inactive across the entire dataset. This novel approach stems from our insights into the phenomenon, where we identified specific hyperparameters that make dead neurons prevalent enough for a successful pruning algorithm.

Unlike previous methods [1,2,3], our approach uniquely ensures that the pruning action does not disrupt the learning dynamics, eliminating the need for recovery interventions. While [2] and [3] focused on restoring capacity through pruning negative weights, their emphasis aligns more closely with studies addressing plasticity loss and subsequent restoration [4,5].

In reponse to the reviewer’s comment, we further highlighted these distinctive aspects in the revision, in both the introduction and Section 4, providing a comprehensive context for the originality of our approach. We appreciate the opportunity to clarify these nuances.

Questions:

1. Dead neurons definition for convolutional layers. Thank you for pointing out this oversight. We have added an explicit definition in Appendix A, referenced in section 3 when we introduce the general definition. In convolutional layers, ReLU is applied element-wise to the pre-activation feature map. We consider an individual neuron (filter) dead if all elements of the feature map post-activation are 0.

2. Brownian motion model. The purpose of this model is to highlight in a simple setting the specific role played by the noise structure in the weights dynamics. Specifically, the absorbing Brownian motion models a system subject to noise with ReLU activation structure; and our analysis highlights how such a structured noise induced sparsity, as the weight ends up in the inactive region with probability 1.

   We believe that this insight is important and reveals the pivotal role played by the stochasticity of the learning algorithm in the occurrence of dead neurons, as also illustrated in our empirical observations in Fig 2. However we acknowledge this is not central to the paper. As per Reviewer’s feedback, (also per Reviewer i9Pr’s suggestion) we’ve moved the discussion of the Brownian motion model to Apppendix B and incorporated a reference to it in the intuition subsection of Section 3.1. We appreciate the opportunity to improve the flow and clarity of this analysis section.

3. On the use of DemP’s regularization schedule for the baselines. Thanks for highlighting this confusing aspect. Our baselines trained for comparison do not use the one-cycle regularization schedule used by DemP, but rather the constant weight decay suggested by the training recipes. We clarified this important aspect in Section 5 and in Appendix J1-J4.

[1] Hu, Hengyuan, et al.: Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016).

[2] Whitaker, Tim, and Darrell Whitley: Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life. arXiv preprint arXiv:2302.05818 (2023).

[3] Liu, Shiyu, Rohan Ghosh, and Mehul Motani: AP: Selective Activation for De-sparsifying Pruned Networks. Transactions on Machine Learning Research (2023).

[4] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, Utku Evci: The dormant neuron phenomenon in deep reinforcement learning. ICML 2023: 32145-32168

[5] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, Marlos C. Machado: Loss of Plasticity in Continual Deep Reinforcement Learning. CoRR abs/2303.07507 (2023)

We thank the reviewer for their accurate summary of our work. We also thank the reviewer for insightful comments and concrete suggestions to improve the manuscript. We try to address as best as we can the issues raised below:

1. More speedup results. In line with the reviewer’s suggestion, we added in Table 1 extended results for training speedup as well as training/inference FLOPs for the ResNet-50 architecture on ImageNet. We intend to run analogous experiments for the smaller networks as well and add the corresponding results to the camera-ready version. Regarding the reviewer’s suggestion to refer to Mosaic ML’s results, it is worth noting that these leverage a number of methods and tricks, such as FFCV [1], in addition to pruning, so that fair, direct comparisons are not straightforward. Moreover, the provided framework requires more computational resources than what we use in our experiments. Finally, it is only available in Pytorch where our code relies on JAX.

2. Performances on CIFAR 10. We appreciate the reviewer’s feedback, which highlights the need for clarifications in our manuscript. First, it’s important to note that we report the test accuracy at the end of the training, while multiple works report the best accuracy reached during training (e.g., [3] sec. 5). Second, while SGD+momentum typically gives better accuracies, we focused our experiments on ADAM because of its popularity and its inherent alignment with our method (see Appendix E). It is also the choice made by our strongest structured pruning baseline [2]. The performance of our experiments seems aligned with the reported results in [3] (Fig. 5) for the ADAM optimizer.

3. On the use of DemP’s regularization schedule for the baselines. You are correct, our baselines do not use the (potentially suboptimal) regularization schedule used by DemP. We clarified this important aspect in Section 5 and in Appendix J1-J4.

4. DemP on transformers. While we acknowledge the importance of investigating the application of our approach to transformer-based models, we would like to highlight that our benchmarking was deliberately aligned with recent prior work on the topic (e.g., [2,4]). This alignment allows for a meaningful comparison with prior work, especially given our focus on gradual pruning during training to reduce computational costs.

5. On our comparison with unstructured pruning baselines. We thank the reviewer for these important comments and suggestions. We note that Diffenderfer et al. (2021) are targeting a different setup where pruning is done in a single shot after regular training, with additional fine-tuning to recover performance afterward. The goal we pursue with our method is to do dynamic pruning under a fixed training budget to reduce the computational load for learning the model. Also, they achieved this performance with SGD + momentum, while we used ADAM (see reply to question 2).

We agree with the reviewer that, relative to unstructured methods, the benefit of DemP is its ability to provide speedup. We included a quick comparison to unstructured methods that also prune the model throughout training to expose that the perceived trade-off between structured and unstructured methods (unstructured achieve better performance) is not always obvious and can require extensive fine-tuning. It also aligns with prior work [2].

Following your suggestion, we clarified further Fig. 6 with the points raised above. As per Reviewer i9Pr’c comment (question 1), we also moved this figure (Figure 16 in the new revision) to Appendix I to focus the content on structured methods.

6. More ImageNet results at different DemP sparsity levels Thank you for this great suggestion. We complemented Table 1 with further results, adding a comparison between methods at 90% sparsity as well. We also included the suggested figure in the revision (see Figure 7).

7. Implementation of Dense model Thanks again for pointing this inconsistency to our attention. Following your suggestion, we have trained our own dense baseline and added the results to Table 1.

Please continue to the comment below

Thank you for adding the new results and clarifications! The experiments appear more convincing now, and the speedups are particularly nice to see.

In main text Figures like 4b, 5b, 6b, and 7b (especially Figure 6b), I think a plot of training speedup vs. "neuron sparsity" would be more compelling than plotting Accuracy vs. "weight sparsity" (inference speedup results would be great too). For example, if you show a scenario where accuracy is not significantly harmed (<1% difference) and training time is improved (e.g., by 5+%), that would increase the relevance of this paper to the efficient training literature (e.g., see "Compute-Efficient Deep Learning", Bartoldson et al., 2023). Relatedly, extending the experiments to show speedups on transformer training could greatly raise the contribution of this manuscript.

I have raised my score. Going forward, I believe this paper can attain a significantly broader impact by expanding its focus in the suggested ways (improving on the MosaicML or another benchmark ResNet50 time, showing applications to transformer training, etc.).