PaI is getting competitive by training longer

-The presentation need to be more clear. For example, SCULPT-ing may be presented more formally as Algorithm 1 with pseudocodes. The bottom line of Figure 1 illustrates that SCULPT-ing requires two cuts, while the discussion some times indicates pruning only happens at initialization.

-The experiments are not comprehensive enough. Only small CNN models are involved. How about Vision/Language Transformers? Note that pruning is more important for those large-scale models, right?

-The work suggests that PaI can be competitive. But how competitive it is exactly? What are the SOTA pruning methods? It will be useful to compare SCULPT-ing with SOTA pruning methods with various model architectures on various benchmark tasks.

-This paper lacks theoretical analysis.

Overall, I think the paper requires major revision to clearly frame the results and their relation to prior work. Furthermore, while the paper heads in some interesting directions, I have reservations about the significance of the new results.

When discussing pruning at initialization, there are two questions that have been of general interest:

1. Do there exists masks that, when applied to the network at random initialization, produce a subnetwork that is trainable to the same test accuracy as the original dense network?
2. Can such a mask be found with no or limited training? (The same question is also of interest for the mask applied to the network early in training.)

In essence, to be deemed successful, training with the masks produced by a pruning at initialization method should produce a test accuracy vs. sparsity curve at or above the performance of weight rewinding, e.g. the curves shown in Fig. 2A,B of this paper. These curves have a sparsity level above which the accuracy quickly drops below that of the dense network; before this, they are consistently at or above this accuracy. The reason for interest in these two questions is to drive down the training cost of finding very sparse networks that make no performance compromises compared to training the full network.

Section 4: The experiments in Section 4 most straightforwardly engage with these questions. Fig. 4 shows how cyclic training boosts performance for three different methods of pruning the network at initialization: random, SNIP, and Synflow. However, as clearly acknowledged by the paper, the performance falls off with increasing sparsity much earlier than LRR, especially in the case of ResNet-18 on ImageNet where performance falls below the dense network before 50% sparsity. Thus, while cyclic training boosts performance, there is still a significant gap in the key high sparsity regime. (Note: I would reword Line 350: "Having established that the learning rate schedule of LRR drives most, but not all of its performance, we are left to wonder what constitutes strength in the high sparsity regime.": I think this a misleading summary of the results in Section 4. The summary in line 328 is more accurate: "we conclude that cyclic training can significantly boost PaI methods and even outperform LRR in low sparsity regions.")

Section 5: Section 5 then explores the relation between the mask and the parameter initialization. In particular, it is shown that starting with the LRR mask from a warmup initialization and then performing cyclic training matches the performance of LRR but not with random initialization; similar results are obtained with the mask from weight rewinding. This is then discussed in terms of coupling of mask and parameter initialization being crucial to improve performance at high sparsity. However, this does not seem to substantially build on previous results. As shown in Fig. 5, WR and LRR have similar performance to begin with, and Appendix D, Fig. 8 of (Paul et al., 2023) shows that the mask obtained from LRR can be retrained from an early rewind point and match the performance of LRR. Thus, the only difference to previous work is cyclic training is used, which I think provides limited additional insight about the coupling of mask and parameter initialization.

The paper does ask in the text whether cyclic training is required to achieve this performance, but no comparisons are made to non-cyclic training with the LRR mask. Rather the loss landscape comparisons in Fig 6 simply compare the warmup init and random init for just cyclic training. Given that the same result held in previous work, my takeaway is that the cyclic training is relatively unimportant here compared to the fact that "an initialization that is coupled to the mask and is task specific starts in the final loss basin or close to it." The two more novel results are: (1) cyclic training with a random mask outperforms the LRR mask at random initialization and (2) the signs of the LRR mask are sufficient with the warmup initialization.

The section concludes with the following:

Cyclic training alone is not sufficient to succeed at high sparsity but requires an initialization that is well coupled to a mask. Our analysis is inconclusive whether LRR masks alone are better aligned with a learning task than PaI masks and poses the potential universality of lottery tickets in the high sparsity regime as an open question.

In my view, this conclusion has not pushed us forward on either of the two questions for pruning at initialization. For 1, it is saying we need a rewind point that is not random initialization which has been explored at length in multiple papers, including (Frankle et al., 2020, "Linear Mode Connectivity and the Lottery Ticket Hypothesis") and (Paul et al., 2023). And then for 2, it essentially says that other ways of finding a mask at high sparsity other than WR and LRR remains an open question.

Section 6: Section 6 then presents SCULPT-ing as a new algorithm that prunes in 2 steps. First the model is pruned to 20%, 50% or 70% sparsity via one of the PaI methods and then cyclic training is performed. Then the network is pruned to the final sparsity by magnitude pruning and one more cycle of training is performed. For many cases, SCULPT-ing still underperforms LRR at high sparsity, and in the case of ResNet-20 on CIFRAR-10, underperforms cyclic training with PaI methods across all sparsities (this is hypothesized to be caused by the small parameter count). Figure 8c shows the method is successful at high sparsities for ResNet-50 trained on ImageNet.

I see SCULPT-ing as more akin to a continual pruning method than PaI. The final mask is not assumed or tested to be effective at random initialization, but rather at the end of a cyclic training procedure. Thus, I would recommend the emphasis of this section be on the any training FLOP win as discussed in the "Training Time" section.

Given this paper is framed around pruning at initialization, several works are missing from a background discussion, including (Sreenivasan et al., NeuRIPS 2022, "Rare Gems: Finding Lottery Tickets at Initialization.") Table 1 of this paper also lists more relevant works that should be discussed.

Editing note: While the green band is Figure 2, 4, 5, 7, 8, and 9 is described in the text, it is never labelled in the figures.

1. While the general focus on the impact of cycling training on the performance of PaI training is interesting, I find that a significant part of the paper emphasizes the importance of aligning the mask with proper initialization for high sparsity in PaI. This point has already been extensively discussed in prior research on sparse training (e.g., Frankle & Carbin, 2019; Zhou et al., 2019; Chen et al., 2020). What new insights does this paper contribute to this topic? While it is certainly valid to confirm previous findings through empirical studies, I would be cautious about presenting such a validation as one of the main contributions of the paper (as suggested in the second bullet of the "contributions" section). Overall, I appreciate the authors' comprehensive Related Work section, but I am uncertain whether this paper has been effectively positioned within the existing research context in terms of this topic.
2. From an empirical standpoint, the paper focuses exclusively on ResNet models and three datasets: CIFAR-10, CIFAR-100, and ImageNet. Additionally, using ResNets originally designed for ImageNet when training on CIFAR-100 is known to result in overparameterized models, making it relatively straightforward to operate at quite high sparsity levels without significant performance degradation. The authors might consider using CIFAR-specific ResNet architectures, as they do for CIFAR-10, to ensure a more balanced evaluation.
3. Moreover, this focus raises the question of whether the paper’s findings are transferable to models beyond ResNets. While it would be ideal to extend the analysis to architectures such as Transformers (e.g., ViT), even experiments on smaller fully connected networks, or other convolutional architectures (EfficientNet, MobileNet) could add valuable context and broaden the scope of the study.

Fundamentally, I have three major concerns with this work: 1). Lack of novelty for primary contributions and claims, 2). Low model architecture and dataset diversity; and, 3). Low performance of SCULPT relative to existing methods which achieve better generalization with much lower total training FLOPs. Below I expand on these concerns and include actionable requests that, if satisfied, would enable me to raise my initial score.

Lack of novelty

• Cyclical LR schedulers are commonly employed in training of SNNs and its benefits have been previously established [1-2, 6]
• Similarly, the benefits of extending training durations have been established across a wide range of SNN training methodologies [3-5].
• Based on the above points, I believe the primary contribution of this work is the rigorous evaluation of extended training with cyclical learning rates specifically for the PaI paradigm. Unfortunately, I believe this contribution is of modest significance given its poor performance compared to existing methods as discussed below.

Model architecture and dataset diversity

• Empirical evidence for the primary claims of this work are motivated through the training of sparse ResNet CNNs on CIFAR-10, CIFAR-100, and Imagenet-1k. Given the prevalence of the transformer architecture, extending these results to a small ViT (DeiT-Tiny for instance) and also exploring alternative CNN architecture such as MobileNet or EfficientNet would improve my confidence that these results would generalize to other modalities and models.
• The primary evidence offered for the benefit of cyclical LR schedules is based on a single ResNet-20 / CIFAR-10 study (Fig 2). In my opinion, this dataset alone is not sufficient to draw strong conclusions. Confirming the benefit of cyclical LR schedules on datasets more indicative of “real-world” data such as ImageNet would improve my confidence in the claims made w.r.t. cyclical LR vs. one-cycle. Given that the rest of the work depends on this result, this point is of critical importance for me to increase my score.

Performance of SCULPT

• The authors state that SCULPT is a sparse training method; however, SCULPT requires training with very low initial sparsities (20%) to obtain good results on ImageNet. This stands in stark contrast to methods such as RigL and other dynamic sparse training (DST) algorithms that initialize the mask to the final sparsity level (90% for instance) and maintain that sparsity throughout the entire training duration. RigLx5 (~500 epochs) obtains 76.4% with a 90% sparse ResNet-50 on Imagenet vs. SCULPT 20% with 450 epochs obtaining ~75.5% accuracy.
• The above weakness is further exacerbated by methods such as RigL outperforming SCUPLT while maintaining the final target sparsity throughout the entire training process, resulting in a very large decrease in total FLOPs required compared to SCULPT. Even dense-to-sparse methods such as AC/DC likely outperform SCULPT in an iso-flop comparison when accounting for total training flops.
• Further to this, while the authors claim that this initial sparsity level for SCULPT yields potential for memory / computational benefits over LRR, at sparsities such as 20% it is unlikely that any computational benefits can be realized in practice. The most efficient possible condensed representation for a 20% SNNs is to use bitmasks to compress the weights, adding a 1-bit/parameter overhead. Assuming 16 bit floating point weights during training, this means the total memory overhead of a 20% sparse network, in terms of parameter storage, is 86.25% of the dense network ((16 * 0.8 + 1) / 16). Further, at this level of compression, sparse matmul kernels are not efficient so it would be required to scatter the sparse compressed weights back into a sparse, dense tensor, adding some latency overhead as well.
• To better determine the affect of the the initial sparsity, I’d like to see a comparison of SCULPT 20% with GMP* and its accelerated cubic learning rate scheduler.

Minor concerns / typos

• L404 ‘upto’ -> up to
• LRR description in A.5 should explicitly discuss learning rate rewinding.

[1] L. Yin et al., “Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training,” Jun. 21, 2022, arXiv: arXiv:2205.15322. Accessed: Jun. 28, 2022. [Online]. Available: http://arxiv.org/abs/2205.15322

[2] T. Jin, M. Carbin, D. M. Roy, J. Frankle, and G. K. Dziugaite, “Pruning’s Effect on Generalization Through the Lens of Training and Regularization,” presented at the Advances in Neural Information Processing Systems, May 2022. Accessed: Feb. 09, 2024. [Online]. Available: https://openreview.net/forum?id=OrcLKV9sKWp

[3] K. Sreenivasan et al., “Rare Gems: Finding Lottery Tickets at Initialization,” Jun. 02, 2022, arXiv: arXiv:2202.12002. Accessed: Jul. 02, 2022. [Online]. Available: http://arxiv.org/abs/2202.12002

[4] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen, “Rigging the Lottery: Making All Tickets Winners,” arXiv, arXiv:1911.11134, Jul. 2021. doi: 10.48550/arXiv.1911.11134.

[5] G. Yuan et al., “MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2021, pp. 20838–20850. Accessed: Oct. 14, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/ae3f4c649fb55c2ee3ef4d1abdb79ce5-Abstract.html [6] E. Kurtic and D. Alistarh, “GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods,” Dec. 08, 2022, arXiv: arXiv:2210.06384. doi: 10.48550/arXiv.2210.06384.

[7] S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both Weights and Connections for Efficient Neural Networks,” Oct. 30, 2015, arXiv: arXiv:1506.02626. doi: 10.48550/arXiv.1506.02626.