Revisiting the Lottery Ticket Hypothesis for Pre-trained Networks

• The proposed technique has a limited novelty. Most of the findings are empirical-based without deeper theoretical underpinning. The paper has made a trivial extension of the GMP model which may not be enough in terms of novelty.
• The authors may need to provide more descriptions of IMP, EB, One-Shot, and GMP techniques in order to make the paper self-contained.
• The main advantage of using GMP over IMP is in terms of computational cost as the performance is comparable in most cases. The author may need to include a separate subsection in the experimentation section to compare the computational costs of different baselines along with IMP and its variants.
• The authors have mentioned in the paper that using GMP in the pre-trained network, a smaller pruning period, and a higher learning rate are preferred in order to achieve a better performance. A more comprehensive study may be required to support the above claim. Specifically, in Table 2 it would be interesting to see the performance variation on a wide range of learning rates along with the different pruning periods.

1. The conclusion drawn regarding PAI methods in Section 3 may be subject to potentially unfair comparisons. It's important to note that these sparse training methods are originally designed to derive improved pruning masks from random initialization, rather than from fully-trained networks. Therefore, their effectiveness should be gauged based on their ability to outperform random pruning. Notably, in Figure 1, most PAI methods, with the exception of GraSP, still exhibit superior performance compared to random pruning (depicted by the black line) in both random initialization and pre-trained weights. This suggests that these methods remain effective in the context of transfer learning. Conversely, the practice of employing magnitude-based pruning on pre-trained dense weights is often categorized as pruning after training (Han et al., 2015a). Although the pre-trained networks were not specifically trained on the downstream datasets, the pre-trained weights still encapsulate valuable information and differ significantly from randomly initialized weights. Consequently, from my personal perspective, comparing PAI methods with magnitude-based pruning may be considered an inequitable assessment, even within the framework of transfer learning. Thus, it would be inaccurate to claim that saliency-based PAI methods are ineffective for pre-trained networks.

2. The distinction between this paper and prior works (Chen et al., 2020, Chen et al., 2021) is somewhat vague. As of now, the primary dissimilarity lies in the fact that this paper places a distinct emphasis on leveraging downstream datasets for PAI. Thus, the novelty of this paper is kind of limited.

3. The ablation studies about the learning rate and the pruning periods are missing. Although Table 3 does provide results involving different learning rates and pruning periods, the performance within the range of pruning periods spanning from 100 to 1000 remains ambiguous. This paper only show that $\bar{GMP}$ outperforms GMP in Table 1 and Table 2. However, the underlying rationale for the choice of these specific hyperparameters requires more comprehensive elucidation.

I have major concerns regarding its motivation and the real contribution.

1. I am rather confused by the settings of this work - The paper states "Our research stands as the first to systematically apply and compare diverse pruning-at-initialization methodologies to pre-trained networks." Pruning-at-initialization means the model is supposed to be randomly initialized, however, the paper says the method is applied to a pretrained network. Then why call it "pruning at initialization"?

2. Limited true technical contribution.

2.1 The method, GMP_bar, is basically identical to GMP, just using a large LR. I am not sure this can be claimed as a "new" method...

2.2 Meanwhile, the observed effect that using a larger LR can boost pruning performance in Tab. 2 & 3 is not new, either -- Many works have observed this phenomenon (see the discussions in https://arxiv.org/abs/2301.05219).

Most of the performance improvements go back to the usage of a larger LR, even for the counterpart method ("it is observed that EB occasionally outperforms IMP, which can be attributed to the use of larger learning rates in EB"). I do not think this can be a novel point for this paper as finetuning LR is essentially not about pruning (especially when this has been reported in the community).

In short, the real contribution of this paper, in my view, is to exploit an already-known phenomenon (a larger LR can help boost pruning performance) in the setup of PaI for transfer learning. To me, this seems not sufficient for an ICLR paper.

I have many significant concerns with the novelty and necessity of this work. There are several claims which are over-sold in the paper e.g. 1. we are the first to demonstrate that a cutting-edge pruning method can also achieve state-of-the-art performance in the LTH domain, 2. first to systematically apply and compare diverse pruning-at-initialization methodologies to pre-trained networks, among many others. I highly recommend the authors to carefully dig up LTH/pruning for pre-trained models papers. The limited role of IMP for pre-trained models in comparison to one-shot is also established (check https://arxiv.org/abs/2306.03805). GMP vs GMP-BAR with difference of large learning rate and a small pruning period - I do not understand why this is important and shown as one of the main contributions. Isn't these are hyperparameters that can be tuned depending on intuitive performance? The addition of some large-scale models like LLMs can bring some new insights PaI which I think is not sufficiently explored. Also, the paper writing can be significantly improved, making smaller sentences to improve readability and convey a clear message.