Sign-In to the Lottery: Reparameterizing Sparse Training

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. We appreciate your positive comments regarding the clarity of our work, theoretical support and extensive experiments. Please find a detailed point-by-point response to your comments below. In case of any open questions, we would be happy to discuss them.

Rebuttal of Weaknesses

1. In the revised manuscript, we would be happy to extend the table and figure description as suggested to highlight sign flipping dynamics and that the reported results are for Top-1% accuracy on ImageNet.

2. To make the distinction between DST and PaI in Figure 1 clearer, we will add two headers saying DST and PaI. Moreover, we will make the current vertical line that indicates the distinction bolder.

3. We agree with the reviewer and will replace “training sparse neural networks from scratch (PaI)” with the full name Pruning at Initialization (PaI) in the abstract and define it early on.

4. If the gradient is positive ($\nabla_{\theta}f > 0$) and the parameter theta is negative ($\theta < 0$), this means that the sign is perceived to be already correct (assuming that the gradient is pointing in the correct direction). This is also true for standard gradient descent. Note that Sign-In controls the relative magnitude but not the direction itself. In practice, we would work with stochastic gradient estimates. Noisy gradients could push the parameter away from zero wrongly then, as in case of standard gradient descent. Sign-In would simply not add merit in this case. The main benefit of Sign-In is realized in the case when the gradient and parameter are both pointing in the same direction (eg: $\nabla_{\theta}f > 0, \theta > 0$), which would imply that the gradient is pushing the parameter towards 0 (because $\theta \leftarrow \theta - \eta \nabla_{\theta}f$). Sign-In speeds up this movement towards 0 and promotes thus a sign flip.

5. We will explicitly state that both the teacher and student are single neuron neural networks in a revision of our manuscript.

6. To strengthen the our conclusions from Figure 1, we provide additional experiments by directly manipulating the sign as suggested by the reviewer. We also report results where Sign-In improves state-of-the-art pruning methods like dynamic sparse training with RiGL. See tables below.

7. Our theory combines insights from two lines of work, namely, Riemannian gradient flows induced by overparameterization and dynamical systems theory. The combination of both is a novel idea and requires non-trivial proof steps. Moreover, the theory captures and provides insights into a novel practical algorithm, Sign-In, which translates to large scale experiments and presents a step forward towards solving the PaI problem. For that reason, we see potential that our conceptual insights could inspire further theoretical investigations and new method design.

Answers to Questions:

1. Yes these are indeed the top 1% accuracy values. We will make sure that this is clear from the table description in the revised manuscript.

2. Please see Point 4 in weaknesses (above).

3. Different $\beta$ would lead to different effective learning rates (see Eq.(1) for example). Setting $\beta =1$ allows us to keep the effective learning rate close to the original learning rate of standard optimizers. To illustrate this point, we have conducted an experiment for different $\beta$ on CIFAR100 with a Resnet18 for random pruning with $90$% sparsity:

We observe that higher $\beta$ could lead to slightly better performance, most likely due to more opportunities for correct sign flips. Yet, it also induces a higher variance due to the higher effective learning rate. Across different experiments, $\beta=1$ required less tuning of other hyperparameters. Hence, we choose this value for $\beta$ in all our experiments.

4. In Table 1 in the main text, we have studied the effect of manipulating signs at initialization. In principle, it might also be possible to provide the right signs later during training, as parameters need to move less given the right signs. However, this is an orthogonal question to solving PaI.

In addition, we report results for manipulating the signs as suggested by the reviewer, extending the results from Table 1 for ACDC. We find the reducing the number of correct signs progressively worsens performance.

To strengthen our point that Sign-In improves contemporary sparse training methods, we provide experiments with RiGL + Sign-In. This highlights that learning with a dynamic mask as in RiGL, Sign-In also helps with the sign flipping mechanism, which can improve performance at high sparsity, especially.

We sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications.

We sincerely thank you for your thoughtful feedback and for engaging in this discussion. We are happy that we were able to address most of your comments and questions.

To answer your question, our analysis relies on the standard assumption that the gradient points in the correct direction. This is a necessary precondition for any first-order optimizer—with or without Sign-In—to function correctly.

Therefore, in the specific scenario you described (a large negative parameter with a positive gradient), the gradient correctly indicates that the parameter should become more negative. In this case, a sign flip is undesirable, and our method is designed not to force one.

Toy Example: Let us make this more concrete with our toy example from Figure 3a, given a single neuron network with a scalar input and a ReLU activation, $f(x; a, w) = a \sigma (wx)$. This network, $f(x; a, w)$, is trained with gradient descent and LR=$0.005$. In our analysis, we have considered all four possible initializations for $a, w$ parameterized with Sign-In, with a balanced initialization ($a^2 = w^2$). We construct a case which generates a positive gradient for a parameter which is negative by assuming a ground truth $y = -10 \sigma (x)$ and discuss the four possible cases. Note that the cases when $w < 0$ will fail due to the ReLU activation which is in line with Figure 2 (Sparse + Sign-In case). Experiments for the remaining two other cases are presented by the next table.

• In Experiment 1 (in the table above), the sign of $a$ is flipped by Sign-In to reach the ground truth. This would not happen without Sign-In.
• Experiment 2 replicates the scenario described by you. In this case, $a$ does not need to switch its sign to reach the ground truth. With and without Sign-In, $a$ correctly converges to the ground truth and becomes more negative.

In both cases, learning with Sign-In finds the ground truth, as verified by the final product $a_{\text{learnt}}w_{\text{learnt}} = -10$.

In addition, note that Sign-In only modifies the magnitude of the gradient step by multiplying with $\sqrt{\theta^2 + \beta}$ to facilitate a sign flip but does not change the sign of the gradient of the original parameter to force a sign flip.

Regarding our comment on SGD: The gradient updates used in practice are based on mini-batches, making them noisy approximations of the true, full-dataset gradient. As a result, the direction of these gradient estimates can sometimes be suboptimal. This is a universal property of SGD that affects all first-order methods, including our own, and is not a new limitation introduced by Sign-In.

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Please find a detailed point-by-point response below. In case of any open questions, we would be happy to discuss them.

Rebuttal of Weaknesses

1. We thank you for pointing us to the related work [1]. While [1] draws the insight that the signs learnt by Iterative Magnitude Pruning (IMP) and thus the initial signs are informative on small scale experiments (CIFAR10), our work is concerned with the question when signs become informative during (and the end of) training, how something similar can be achieved by PaI also in the context of larger scale experimental settings (like ImageNet). Our work is closer to [4], which analyzes how this sign learning mechanism enables LRR to outperform IMP. As mentioned, we show that this insight is more universal and holds for various mask topologies and sparsification methods. We will be happy to add this discussion.

2. As you have pointed out, dense to sparse methods flip parameter signs early in training, which stabilize during subsequent training.

   • However, this is not the case for PaI methods as highlighted by Theorem 5.4, due to the lack of overparameterization. To mitigate this, according to Theorem 5.2, Sign-In introduces an orthogonal sign-flipping mechanism that enables useful sign flips also in the absence of overparameterization.
   • Hence, in the case of PaI, where early sign flipping followed by sign stabilization is not possible, Sign-In induces a mechanism to learn the correct sign flips (as shown in Figure 2).
   • Specifically, Sign-In facilitates more useful sign flips, as shown in the toy example by recovering the ground truth.
   • To motivate this further, consider our theoretical example, where we want to train a single neuron $a \phi( w x)$, where the sign of $a \in \mathbb{R}$ is wrongly initialized. The parameter $a$ cannot be flipped due to the balancedness property. Nevertheless, all parameters in the weight vector $w \in \mathbb{R}^d$ have a chance of flipping while the activation stays non-zero for some data points. This will not necessarily bring us close to the ground truth but will lead to more sign flips (in $w$) as in line with the general instability of the sign flipping in PaI, whereas Sign-In allows for a correct additional sign flip.
   • Therefore we use the amount of sign flips as a proxy for the amount of additional correct sign flips on top. Further experiments with ACDC + Sign-In (Table 10) and Rigl + Sign-In (shown below) show that Sign-In performs useful sign flips which are orthogonal to the sign stabilizing mechanism in dense-to-sparse methods.
   • So the flow of argumentation is that we first showed empirically that correct signs plus mask is enough for recovering dense performance then this is followed by a method that can provably induce a part of these correct sign flips.

3. The additional experiments below highlight the utility of Sign-In for state-of-the-art dynamic sparse training methods like RiGL and Mest. On Imagenet with a ResNet50, we see relevant improvements starting from the 90% sparsity level, where the sign optimization starts to matter more.

Furthermore, we show the utility of Sign-In with other PaI methods such as SNIP and Synflow in Table 7 in Appendix H. Note that [2,3] showed that the random baseline often performs better, which was the main reason to highlight it in the main part of the manuscript. We would be happy to move the additional results on SNIP and Synflow to the main part of the manuscript given an extra page. Additionally, we have conducted experiments with the recently proposed PaI method NPB [5], as reported below.

Question

We report results with additional PaI methods as well as highlight that Sign-In can improve dynamic sparse training methods like RiGL via its sign flipping mechanism. Similar improvements on sparsification with ACDC are also reported in Table 10.

We sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications.

[1] Zhou, Hattie, et al. "Deconstructing lottery tickets: Zeros, signs, and the supermask." NeurIPS 2019.

[2] Liu, Shiwei, et al. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. ICLR 2022.

[3] Gadhikar, Advait Harshal, Sohom Mukherjee, and Rebekka Burkholz. "Why Random Pruning Is All We Need to Start Sparse." ICML 2023.

[4] Gadhikar, A. H., & Burkholz, R. (2024). "Masks, signs, and learning rate rewinding." ICLR 2024.

[5] Pham, H., Ta, T. A., Liu, S., Xiang, L., Le, D., Wen, H., & Tran-Thanh, L. (2023). Towards data-agnostic pruning at initialization: What makes a good sparse mask? NeurIPS 2023.

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. We appreciate that you find our work well written, easy to follow and consider our results to be strong. Please find a detailed point-by-point response below. In case of any open questions, we would be happy to discuss them.

Rebuttal of Comments:

1. We are happy to switch to the more common terminology of the field and mainly distinguish sparse training methods and pruning at initialization.

2. By choosing the signs corresponding to warm-up or some preliminary epoch we are effectively selecting partially correct signs. To analyze the dependence on partially correct signs in more detail, we have performed additional experiments where we vary the number of correct signs for a mask identified by ACDC on a ResNet50 for ImageNet. The table below shows that fewer correct signs at initialization progressively degrade performance.

3. We would be happy to include such a figure in the main manuscript. Table 12 shows that Sign-In’s added overhead is negligible compared to the method it is applied to. Therefore, Sign-In pushes the Pareto frontier of PaI, as it achieves significantly better generalization with a similar overhead.

4. The table below presents the effect of combining Sign-In with RiGL for ResNet50 on ImageNet. We conclude that also dynamic masks can benefit from improved sign learning at higher sparsity.

5. We would be happy to sparsify the introduction by putting less emphasis on all the individual results in the revised manuscript.

Questions:

1. That is a really interesting idea. We also believe that improved sign learning could be an important mechanism contributing to the success of RiGL. To analyze this hypothesis, we have perform RiGL with two ablations in addition to re-initializing weights with 0. They are a) initializing with the weight value at pruning and b) switching the sign of the weight upon re-initalizing. We find that both perform worse than initializing at 0. This is in line with the findings of the authors of RiGL and in line with your hypothesis that initialization at 0 promotes flexibility of parameters to learn signs. (Furthermore, it perturbs the model less.) Interestingly, reinitializing with the opposite sign performs better that the original sign, suggesting that a sign flip could be helpful. We report these results for a ResNet50 on ImageNet at 90% sparsity.

In addition to the mechanism of pruning and regrowing in RiGL, our results suggest that training RiGL in combination with Sign-In can lead to additional gains especially at higher sparsity (see previous section on weaknesses), as also the weights that are not rewired learn signs more effectively.

2. Sign-In also helps in the high sparsity regime. Inevitably, there will be a point where the connectivity between layers will break down depending on mask topology. As we see in experiments (Table 3 and 7), the performance gains are comparable for different sparsity levels. Sign-In cannot counteract the connectivity issue. However, as Sign-In is agnostic to the mask structure and the correct signs become more important for sparser masks, Sign-In should lead to similar gains in the high sparsity regime. To verify this intuition, we have conducted an additional experiment for random pruning at initialization, SNIP, and Synflow on CIFAR10 and CIFAR100. According to the table below, Sign In still outperforms the baselines. However as expected at 99% sparsity, due to mask topology instability, the differences between Sign-In and the baseline are within noise.

We sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications.

We sincerely thank you for your thoughtful feedback and for engaging in this discussion. We are happy that we were able to address most of your comments and questions. Please find our reply to your remaining concerns below.

1. On Parameter Signs and Our Contributions

   1.a. Context and Novel Insights

   We would like to highlight that previous work on parameter signs by [1,2] showed that the signs learnt at the end of training with Iterative Magnitude Pruning (IMP) and Learning Rate Rewinding (LRR) contain task relevant information. In contrast, we have shown that training a good mask with a good sign initialization from scratch is in fact competitive with sparse training (in Table 1).

   Our novel insights regarding signs are two-fold:

   • (a) Early Sign Alignment: We show that signs are learnt early in training and important signs are already learnt as early as the 10th epoch.
   • (b) Universality of Sign Alignment: We also show that this sign alignment phenomenon extends to different kinds of sparse training methods including magnitude pruning (ACDC), continuous sparsification (STR) and dynamic sparse training (RIGL). This highlights sign alignment as a more universal phenomenon.

   1.b. Theoretical Limits and Broad Applicability

   While signs largely explain the performance gap between Dynamic Sparse Training (DST) and Pruning at Initialization (PaI), our Theorem 5.4 proves that achieving the correct signs without overparameterization is a fundamental challenge for PaI. We consider identifying this limitation a strength of our work.

   To address this, our method, Sign-In, introduces a sign-flipping mechanism that is orthogonal to overparameterization. As you highlighted, this allows Sign-In to improve not only PaI but also a broad range of other sparsification methods, including DST (RIGL, MEST) and ACDC.

2. Motivation and Versatility of Sign-In

   Our motivation to induce correct sign flips with Sign-In follows from the sign alignment observed across different sparse training methods. However, as this is not possible in PaI (as shown by Theorem 5.4), we use Sign-In to introduce an orthogonal mechanism for sign flipping as shown in Figure 2.

   As you rightly noted, this mechanism's benefits extend beyond PaI. We are grateful for your positive feedback and will revise the main text to highlight this versatility, incorporating our results on RIGL, MEST, and ACDC (Table 10) to underscore our method's broad applicability.

3. The Relationship Between Sparsity and Performance

   We agree that Sign-In's benefit grows with sparsity, and our reasoning is as follows:

   • At low sparsity, the network remains overparameterized, allowing standard optimization to be effective, so Sign-In's improvement is modest.
   • As sparsity increases, the lack of overparameterization makes it difficult to flip signs. Sign-In directly alleviates this bottleneck, significantly improving performance.
   • At extreme sparsity, the mask's topology itself becomes the primary constraint on expressivity, limiting the gains from any method, including ours.

   We will gladly add this discussion and our supporting high-sparsity experiments to the paper.

We would like to thank you again for your valuable suggestions leading to the improvement of our work.

---

[1] Zhou, Hattie, et al. "Deconstructing lottery tickets: Zeros, signs, and the supermask." Advances in neural information processing systems 32 (2019).

[2] Gadhikar, A. H., & Burkholz, R. (2024). Masks, signs, and learning rate rewinding. In The Twelfth International Conference on Learning Representations (ICLR 2024).

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

Rebuttal of Weaknesses

1. Pruning at Initialization (PaI) is an active research field (see e.g. [1,2,7]), as solving this problem would be considered a significant breakthrough for multiple reasons:

   • From a practical side, achieving competitive performance when training a fixed mask from scratch would enable training neural networks in low resource settings (like edge devices) - even lower than what is possible with dynamic sparse training (DST).
   • Training models locally could also have positive implications for data privacy.
   • Training a sparse mask would cost less memory and compute (in particular on specialized hardware such as [5]) in comparison to training denser models (or using denser gradients or larger batch sizes as in some DST approaches). The fact that the mask is fixed and known from the start in contrast to DST, would also allow for utilizing optimized kernels (or even hardware) that exploit the specific sparsity patterns to gain computational speedups during forward and backward propagation.
   • From a conceptual side, improving PaI likely requires novel optimization approaches that provide fundamental insights into the interplay between overparameterization or sparsity and neural network optimization. Our method Sign-In, for instance, addresses the limitation of standard optimizers to effectively learn to switch parameter signs. In overcoming this limitation, it provably solves an optimization problem that is hard in the standard setting.

2. We would like to highlight Table 7 in the appendix, which presents results for Sign-In applied to SNIP and Synflow alongside random masks. In addition to these PaI methods, we also report new results for NPB (a recent PaI method) and RiGL (a dynamic sparse training method) below.

3. Many other methods do not outperform random pruning at initialization (PaI) as shown in [3,4]. For that reason, we focused on demonstrating the effectiveness of Sign-In on random pruning in the main manuscript. With an extra page of space, we would be happy to move our results on SNIP and Synflow to the main part of the paper. In addition, as suggested by the reviewer, we perform experiments on PaI with the NPB method [7], results shown below on both CIFAR10 with a ResNet20 and CIFAR100 with a ResNet18:

4. We assume your comment refers to Table 8 in the appendix.
   • The purpose of Table 8, is to investigate how Sign-In performs for different fixed mask topologies that are derived from running different sparse training methods. Each method in Table 8 is initialized with a fixed mask which has been identified by a specified pruning method and then trained from scratch. The experiment is designed like this to assess how close we come to solving PaI (where only the mask is given). Matching a target performance (obtained by the sparse training method) would mean solving PaI. Hence, the numbers in Table 7 highlight this gap between PaI and other sparse training methods like RiGL or ACDC. The actual performances for these methods are reported in Table 1 for comparison (which for RiGL is 73.75%).
   • As per your comment, we can still ask the question if our method Sign-In could improve dynamic sparse training. To this end, the table below for ResNet50 on ImageNet shows that Sign-In can boost RiGL especially at high sparsity.
   • It can also improve Mest, as we demonstrate with a ResNet50 trained on ImageNet.

5. Neither the premise of our work nor our theory is based on the lottery ticket hypothesis (LTH).

   • We cite the LTH as an important line of work that highlights the difficulty of training sparse neural networks from scratch. The fact that lottery tickets might not exist or are more difficult to find in large-scale settings supports our point. [6] hypothesizes that the main challenge for lottery tickets is in fact to learn parameter sign flips. It argues that learning rate rewinding (LRR), which gives up on the lottery ticket idea, succeeds by improved sign identification. Table 1 demonstrates that the superior sign learning ability is actually a more universal property of contemporary sparsification and DST methods, which do not find lottery tickets but utilize different forms of overparameterization.
   • Our proposed parameterization Sign-In further boosts sign learning. Our theory even proves this in the setting of [6]. Another way to frame our innovations could be to cast them as exploration of controllable Riemannian gradient descent for better generalization. While we demonstrate its utility for PaI, the tools can also be used for different purposes.

6. We will correct the typo in the revised manuscript. Thank you for pointing it out.

In summary, we have argued why PaI is a worthwhile endeavour to enable computing on edge devices and gain theoretical insights into the role of overparameterization in (sparse) deep learning. Moreover, we have referred to more results with other PaI methods (Sign-In applied to SNIP and Synflow in Table 7) and added experiments with NPB and the dynamic sparse training (DST) method RiGL, finding that our method Sign-In leads to consistent improvements. Furthermore, we have clarified our theoretical foundation and motivated it from an optimization angle.

We sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications upon request.

[1] Iurada, Leonardo, Marco Ciccone, and Tatiana Tommasi. "Finding lottery tickets in vision models via data-driven spectral foresight pruning." CVPR 2024.

[2] Adnan, Mohammed et al. “Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry.” ICML 2025

[3] Liu, Shiwei, et al. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. ICLR 2022

[4] Gadhikar, Advait Harshal, Sohom Mukherjee, and Rebekka Burkholz. "Why Random Pruning Is All We Need to Start Sparse." ICML 2023.

[5] Thangarasa, Vithursan, et al. "Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency." International Conference on Machine Learning. PMLR, 2024.

[6] Gadhikar, A. H., & Burkholz, R. (2024). Masks, signs, and learning rate rewinding. ICLR 2024.

[7] Pham, H., Ta, T. A., Liu, S., Xiang, L., Le, D., Wen, H., & Tran-Thanh, L. Towards data-agnostic pruning at initialization: What makes a good sparse mask? NeurIPS 2023

We sincerely thank you for your thoughtful feedback and for engaging in this discussion. We appreciate your comments that allow us to clarify the contributions and context of our work.

Below, we highlight a) why PaI is an important research direction and how it creates a hard test bed for new training algorithms that can benefit sparse training in general. b) Note that our proposal, Sign-In, does not only benefit PaI but also improves other sparse training methods, including DST.

• Sign-In Improves Existing DST Methods: Our work is not limited to Pruning at Initialization (PaI). New results show that Sign-In also enhances contemporary Dynamic Sparse Training (DST) methods:

  • At 90% sparsity: RIGL improves from 73.7% to 74.27%.
  • At 90% sparsity: MEST improves from 73.1% to 73.32%.
  • These results are consistent with our findings for ACDC (Appendix, Table 10), demonstrating the broad benefit of improving sign learning.

• PaI Remains a Critical Research Area: While we agree DST currently achieves higher accuracy, PaI's core constraint—a fixed sparse mask—makes it a unique and valuable research problem. It is a different training principle that is useful in specific very low resource settings. Note that altering the mask during training introduces additional overhead and is not free of cost.

  • Challenge: The fixed mask makes optimization fundamentally harder, requiring novel techniques to succeed with less overparameterization (as corroborated by our Theorem 5.4).
  • Opportunity: A static mask enables significant computational speedups via specialized kernels [1], an advantage that is less pronounced for dynamic masks in DST.

• Our Core Contribution: We identify the importance of sign alignment in sparse training and propose Sign-In, a simple and effective reparameterization to improve performance by introducing useful sign flips in the absence of overparameterization. As shown above, its benefits are applicable across different sparse training paradigms.

• Context from the Lottery Ticket Hypothesis (LTH): The LTH established the core challenge in sparse training: successfully training a sparse mask.

  • It demonstrated that standard optimizers require special initializations that are practically unattainable without first training an overparameterized model.
  • Our work builds on this by explaining why it is hard to optimize in the absence of overparameterization and, based on that insight, proposing an improved training approach.

• Proposed Action: To reflect this broader impact, we will move the results for ACDC, RIGL, and MEST into the main body of the paper.

We thank you again for helping us strengthen our work.

[1] Thangarasa, Vithursan, et al. "Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency." International Conference on Machine Learning. PMLR, 2024.