Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning

[W1] (Re. SWAMP+ in low sparsity regime): Thank you for pointing out the effectiveness of SWAMP+ in low sparsity regimes. We agree with the reviewer that the effectiveness of SWAMP+ fades under low sparsity regimes. That said, IMP is a SOTA algorithm for extremely high sparse networks (sparsity over 90%). We thus suggest two alternatives instead of SWAMP+ for low sparsity regimes. Firstly, dynamic sparse methods (e.g., RigL) can also benefit from SWAMP as shown in Table 9 which exhibits significantly lower training costs than IMP. Secondly, leveraging the robustness of SWAMP (refer to Table 7), one can employ higher pruning ratio which can still boost the training speed even in low sparsity regimes. We believe the general response could further resolve your concerns regarding the training cost of the algorithm. Additionally, as per your great suggestion, we hereby provide the results of IMP+ on ImageNet in Table R.10. Please note that the results in Table R.10 differ from Table 3, as the experiment was conducted earlier with a different set of hyperparameters.

[Q1] (Re. superior performance of interpolated networks in Fig. 4): We appreciate the reviewer for highlighting empirical analysis in Figure 4. Following [Paul et al., 2023], Figure 4 illustrates that there exists a linear path on the loss landscape connecting two successive IMP/SWAMP solutions. In case of both IMP/SWAMP, as the reviewer pointed out, the interpolated weights exhibit lower loss compared to obtained solutions which is also corroborated by several prior works [Garipov et al., 2018; Wortsman et al., 2021; Nam et al., 2022; Paul et al., 2023]. This finding suggests that one can obtain a better solution interpolating successive solutions which is the main motivation for Lottery Pools [Yin et al., 2023]. Further, while we only highlight the existence of linear connectors in the paper, we here provide additional experimental results in Tables R.11 and R.12 suggesting that SWAMP can also benefit from such a phenomenon, i.e., combining SWAMP and Lottery Pools leads to further performance enhancement.

[Q2] (Clarification: learning rate setting): Thank you for the attentive review. In our IMP experiments, we explored two learning rate schedules: cosine annealing (initiating from 0.1 and decaying to 0.001) and step learning rate (adopted from SWA; dropping to a constant value of 0.05). The cosine annealing schedule turned out to be the best for IMP, while SWAMP exhibited better performance with step lr schedule. In short, we employed the cosine annealing schedule for IMP and the step lr schedule for SWAMP.

[Q3] (Clarification: rewinding epochs): Yes, indeed. We set the same matching epoch for both IMP and SWAMP.

Table R.10. Results of SWAMP+ on ImageNet.

Table R.11. Further comparison between (a) IMP, (b) SWAMP, (c) Lottery Pools, and (d) SWAMP + Lottery Pools algorithm on CIFAR-10.

Table R.12. Further comparison between (a) IMP, (b) SWAMP, (c) Lottery Pools, and (d) SWAMP + Lottery Pools algorithm on CIFAR-100.

References

• Garipov et al., “Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs.” NeurIPS 2018.
• Wortsman et al., “Learning Neural Network Subspaces.” ICML 2021.
• Nam et al., “Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation.” ICML 2022.
• Paul et al., “Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?” ICLR 2023.
• Yin et al., “Lottery Pools: Winning More by Interpolating Ticket without Increasing Training or Inference Cost.” AAAI 2023.

[W1] (Re. lack of novelty): Our primary contribution extends beyond the development of an algorithm; it involves empirically demonstrating that sparse networks trained with varying SGD noise can benefit from weight-averaging, especially as they become sparser – a finding that is not immediately obvious and has been acknowledged by Reviewer-tk5N. Moreover, as Reviewer-MGna highlighted, our work is inspired by the theoretical framework of [Paul et. al, 2023] which enabled us to empirically validate whether SWAMP retains the key property of IMP; we kindly refer the reviewer to Section 3.3.

[W2] (Marginal improvement of the algorithm): Please refer to General Response B.

[W3, Q1] (Analysis under lower sparsity regime): Thank you for highlighting the failure of sparse weight averaging under low sparsity regimes. We can see that weight averaging fails at the earlier stages of IMP due to the highly non-convex nature of the landscape. However, as sparsity increases, particles tend to locate in the same wide basin which enables weight-averaging. Such a finding is in line with [Frankle et al., 2020; Paul et al., 2023] demonstrating the ease of finding a low-loss curve with a smaller network compared to a larger one, i.e., a sparse network tends to be more stable. Additionally, it further supports that our algorithm benefits more with sparser networks.

References

• Frankle et al., “Linear mode connectivity and the lottery ticket hypothesis.” ICML 2020.
• Paul et al., “Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?” ICLR 2023.

[W1] (Extension to different hyperparameters): We thank the reviewer for recognizing the novelty of our work, where our main objective is to empirically validate sparse weight averaging technique and whether it preserves the characteristics of IMP rooted in the theoretical framework of [Paul et al., 2023]. As the reviewer pointed out the work of model soups [Wortsman et al., 2022], we hereby provide extended results of SWAMP with multiple particles obtained from various hyperparameters settings (weight decaying rate, learning rate) beyond random seeds in Table R.5. Our findings indicate that weight averaging with particles perturbed by means other than random seeds do not markedly improve performance. We acknowledge that further optimization of hyperparameters, given enough time, could potentially boost SWAMP's performance making it an interesting direction for future research. That said, one should also consider whether the improvements of SWAMP variants justify the additional computational cost they demand.

[W2, Q2] (Heavy training cost & extension to other pruning methods): As per your constructive feedback, we will revise the final manuscripts to include the SWAMP-extended dynamic pruning results in the main body of the paper, not the appendix. As shown in Table 9, SWAMP is indeed applicable to other sparse training methods. During the rebuttal phase, as time permits, we aim to investigate whether dynamic pruning methods, aside from RigL, can also benefit from SWAMP. Moreover, we choose to focus on IMP in order to verify whether SWAMP preserves the theoretical characteristics outlined in [Paul et al., 2023] as discussed in Section 3.3. Also, we believe that General Response A could further resolve your concerns regarding the training cost of the algorithm.

[W3] (Marginal gain in ImageNet experiments): Please refer to General Response B.

[Q1] (Re. ensemble of dynamic pruning methods): As per your detailed feedback, we hereby provide additional results on the ensemble of RigL and DST solutions. Tables R.6 and R.7 provide clear evidence that SWAMP exhibits comparable generalization to test sets when compared to ensembles of other baseline methods.

Table R.5. Further comparison on Cifar-10 with ResNet-20 between (a) SWAMP particles with varying weight decaying rates, and (b) SWAMP particles with varying learning rates. Reported values are averaged over three random seeds.

Table R.6. Additional accuracy results on the ensemble of two particles of (a) VGG13 with RigL, (b) VGG13 with DST on CIFAR-10. And (c) is the performance of a single SWAMP solution. Reported values are averaged over three random seeds.

Table R.7. Additional accuracy results on the ensemble of two particles of (a) VGG16 with RigL, (b) VGG16 with DST on CIFAR-100. And (c) is the performance of a single SWAMP solution. Reported values are averaged over three random seeds.

References

• Wortsman et al., “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.” ICML 2022.
• Paul et al., “Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?” ICLR 2023.

Dear reviewer-tk5N,

We sincerely appreciate your time and the constructive feedback. With the Author/Reviewer discussion deadline approaching, we would be grateful if you let us know whether our responses have addressed your concerns. This will greatly assist us in enhancing our work. For any further clarification, we are more than happy to answer.

Warm regards,
Authors

[W1] (Ablation study w.r.t. SWA & multiple particles): Thank you for pointing out the issue. We indeed provided an ablation study regarding SWA in Table 5. In detail, the second and the third row each correspond to SWAMP without SWA and IMP with SWA, respectively. We can readily find that each component, SWA and multi-particles, contributes similarly to the performance gain, and the optimal performance is attained when used in combination.

[W2] (Clarity: performance comparison to ensembles): We greatly appreciate the reviewer for pointing out unclear points. In Figure 1, we show that SWAMP matches the performance of an ensemble of two sparse networks but not surpassing that of an ensemble of three networks, which is still a significant performance gain considering impressive performance of Deep Ensembles [Lakshminarayanan et al., 2017; Fort et al., 2019]. Further, in the right side of Figure 1, we demonstrate how SWAMP offers advantages over ensembles in terms of memory costs.

[Q1] (Re. best seed performance): As per your insightful suggestion, we provide the performance of an IMP ensemble, wherein each member is chosen as the best-performing model from multiple random seeds. Tables R.8 and R.9 below verify that exploring more random seeds does not result in additional improvements.

Table R.8. Further comparison between (a) IMP, (b) IMP selected from best seed, and (c) SWAMP algorithm on CIFAR-10.

Table R.9. Further comparison between (a) IMP, (b) IMP selected from best seed, and (c) SWAMP algorithm on CIFAR-100.

References

• Lakshminarayanan et al., “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.” NeurIPS 2017.
• Fort et al., “Deep Ensembles: A Loss Landscape Perspective.” arXiv preprint, 2019.

Thank you for your rebuttal. My concerns have been addressed. I request the authors to fix imprecise statements in the manuscript such as that mentioned W2. I also request the authors to provide recent and relevant citations and a more precise replacement for their statement on IMP being state-of-the-art in their abstract (eg. IMP is SoTA in unstructured pruning at high-sparsity). Because this is not true in the structured sparsity regime [a].

I will maintain my score.

[a] Neural Pruning via Growing Regularization, Wang et al. ICLR 2021

[W1] (The choice of model architectures): We appreciate your attentive review. Throughout the experiments, we aim to verify the effectiveness of SWAMP via various architectures including ResNet (Tiny-ImageNet, ImageNet), Wide-ResNet (Cifar-10, Cifar-100), VGG (Cifar-10, Cifar-100), RoBERTa (GLUE). As per your great suggestion, in Tables R.3 and R.4, we additionally provide experimental results of ResNet-20 on Cifar10 and ResNet-18 on Cifar100, which is a frequently employed experimental setup in prior works [Paul et al., 2023; Yin et al., 2023]. We can readily find that SWAMP works well on not only wide-networks but also on parameter-efficient architectures which highlights that overparameterization is not a crucial factor for the success of SWAMP.

[W2] (Marginal gain in Tables 2 and 3): Please refer to General Response B.

[W3] (Re. heavy computational costs): Please refer to General Response A.

[W4] (Re. transformer-based architectures): Thank you for highlighting the observed performance drop of RoBERTa models beyond 50% sparsity. As shown in Table 8, while both IMP and SWAMP struggle to recover the matching performance, SWAMP still surpasses IMP in terms of accuracy throughout all sparsity levels. In other words, such performance drop stems from inherent limitations of IMP, not a drawback of SWAMP. Moreover, [Chen et al., 2020] supports our findings by empirically confirming that BERT networks suffer to find winning tickets compared to vision task models (See Table 3 in [Chen et al., 2020]). Also, we certainly agree with the reviewer that exploring IMP on GPT-like architectures would be a valuable addition to the community. However, as the focus of our work lie in exploring sparse weight averaging techniques along with empirical verification of the theoretical framework of IMP, we leave it as a future research.

Table R.3. Additional results on Cifar-10 with ResNet-20. Reported values are averaged over three random seeds.

Table R.4. Additional results on Cifar-100 with ResNet-18. Reported values are averaged over three random seeds.

References

• Chen et al., “The Lottery Ticket Hypothesis for Pre-trained BERT Networks”, NeurIPS 2020.
• Paul et al., “Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?” ICLR 2023.
• Yin et al., “Lottery Pools: Winning More by Interpolating Ticket without Increasing Training or Inference Cost.” AAAI 2023.

Dear reviewer-MGna,

We sincerely appreciate your time and the constructive feedback. With the Author/Reviewer discussion deadline approaching, we would be grateful if you let us know whether our responses have addressed your concerns. This will greatly assist us in enhancing our work. For any further clarification, we are more than happy to answer.

Warm regards,
Authors

Table R.1. Improved classification accuracy against vanilla IMP in terms of percentage. The percentage points on the left and right correspond to the results for Cifar-10 and Cifar-100, respectively, as detailed in Table 2.

Table R.2. Additional results on ImageNet datasets in terms of classification accuracy. Experiments are repeated over three random seeds.