Masks, Signs, And Learning Rate Rewinding

We sincerely thank the reviewer for their time and constructive feedback.

Response to weaknesses:

1. We further confirm our findings on the larger and more complex datasets: Tiny ImageNet (see Figure 2d) and ImageNet (see Figure 15 in the appendix).
2. So far, LRR has only been proposed as a variant of IMP, which is the most common pruning baseline for its simplicity and high accuracy. It was therefore restricted to magnitude as a pruning criterion. Nevertheless, we have transferred the same principle to random pruning (see Figure 2. b,c and Figure 13 in the appendix), for which our insights hold as well. To extend the analysis, we have further added experiments for iterative Synflow and iterative SNIP in Fig. 16. The superiority of LRR can be generalized to all these different criteria. Also, our additional experiments with a VGG16 architecture on CIFAR10 in Figure 17 in the appendix are in line with these results.
3. In Figure 8 and 9 in the appendix, we provide a comparison on the effect of different learning rate schedules and highlight the benefits of using warmup epochs at the beginning of every training cycle to improve performance of both IMP and LRR.

Minor: We have updated the draft with the corrected typo and added the references.

Response to questions:

1. Experimental insights:

• The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding (red line) this holds also true on CIFAR100. Furthermore, note that we can draw similar conclusions for Tiny ImageNet (Fig. 2d), ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).

• Fig 7(c, d) additionally indicates that LRR depends significantly on the learned sign information, as sign perturbations can greatly hurt its performance. Yet, it is robust to magnitude rewinding (IMP reset amplitude in Fig 7). LRR signs and masks seem to contain sufficient information to achieve the high LRR performance. This conclusion is supported by the results on all studied datasets.

2. The blue line in Fig 11 (Fig 12 in the updated draft) refers to experiments, where we fix several aspects that have been learned by LRR. Concretely, we train an architecture with LRR mask and initial parameters that have the same signs as the learned LRR parameters. The only part that is still learned in the iterative scheme is the parameter magnitude, which starts from the initial IMP magnitude. The fact that this blue line achieves a similar performance as LRR suggests that the high LRR performance can primarily be attributed to the learned signs and the mask.

3. Figure 12 (13 in the updated draft) plots the difference between the total number of signs flipped from the initialization. It shows that LRR flips many more signs in comparison with IMP. We have removed the conclusion in the caption, as not only this figure but the collection of different experiments provide evidence for the stated hypothesis.

We sincerely thank the reviewer for their time and constructive feedback.

Response to weaknesses:

1. Mask identification of IMP and LRR.

• Both IMP and LRR are trained until convergence in each prune-train iteration (All details on the procedure are provided in Appendix 3 (Table 1)). The mask is identified by pruning the smallest parameters at the end of each training level once the model has converged. Hence it stays constant for each training iteration between pruning levels. The mask can also be identified after fewer training epochs optimizing a trade-off between model accuracy and computational costs (see e.g. [1,2]). However, we chose to identify the mask after convergence, because our work is focused on insights into the mechanisms that make iterative pruning algorithms succeed.
• These insights might inspire computationally less expensive variants of LRR and IMP in future. For instance, we find the fact encouraging that LRR identifies good sign configurations relatively early in training.

2. Insights from Experiments are significant across learning tasks.

• The main insight we draw from Figure 4 is that LRR is able to improve parameter optimization for different sparse masks, like the mask found by IMP (IMP mask). This finding actually holds for multiple datasets and learning settings. As correctly noted, LRR with the IMP mask achieves a significantly higher performance than IMP on CIFAR10. If we modify LRR with BN parameter rewinding and use it to train a network with the IMP mask (red line) this holds also true on CIFAR100. Note that the yellow line is not designed to showcase the flexibility of LRR in optimizing different masks, the red line is. In summary, the comparison of the red and the orange lines support our hypothesis.

• Furthermore, note that we can draw similar conclusions for ImageNet (Fig. 15), different pruning criteria (Fig. 16), and VGG networks (Fig. 17).

3. We have added experiments for VGG16 in Fig. 17 in the appendix. The same general trends transfer also to this setting.

Response to questions:

1. We have added experiments for ImageNet in Fig. 15 in the appendix. The results are consistent with our previous findings on CIFAR10, CIFAR100, and Tiny Imagenet.
2. Figure 3 is without BN rewinding in LRR. However, note that IMP always rewinds BN parameters.

[1] You, Haoran, et al. "Drawing Early-Bird Tickets: Toward More Efficient Training of Deep Networks." International Conference on Learning Representations. 2019.

[2] Paul, Mansheej, et al. "Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?." International Conference on Learning Representations. 2022.

We sincerely thank the reviewer for their time and constructive feedback.

1. Both IMP and LRR appear to be similarly sensitive to hyperparameters. In general, generalization on CIFAR10 is less sensitive to different learning rate schedules. However, for the larger CIFAR100 dataset, generalization is more susceptible to a poorly tuned learning rate schedule for iterative pruning. Both IMP and LRR have been used with a step or cosine learning rate schedule (Renda et al. [1]). We find that adding a few warmup epochs (usually 10) at each pruning-training iteration (with linearly increasing LR during warmup) improves the overall performance marginally on CIFAR10 and significantly on CIFAR100, for both LRR and IMP (see Figures 8 and 9 in the appendix). Hence, we propose to use warmup epochs for all experiments.

2. LRR benefits from overparameterization.

• Our theoretical insights on the single neuron model highlight that LRR can leverage overparameterization in early prune-train iterations to enable sign switches of initial parameters and find stable sign configurations which ensure that LRR succeeds more often than IMP. While an increasing input dimension does not harm LRR, at some point its positive impact saturates and further overparameterization is not required anymore (see Figure 18 in the appendix).
• Also experimentally, ResNet architectures tend to benefit from an increased width [2]. Yet, this benefit saturates at a certain point (so that the performance does not improve significantly by further increasing the width). This benefit also translates to pruning the corresponding architectures.
• We are not aware of a case, in which width overparameterization harms performance. However, we could imagine that it could cause issues with trainability and machine precision in extreme cases, but this would be a different topic of investigation.

[1] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.

[2] Golubeva, Anna, Guy Gur-Ari, and Behnam Neyshabur. "Are wider nets better given the same number of parameters?." International Conference on Learning Representations. 2020.