DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity

We sincerely appreciate the reviewer for providing valuable and constructive feedback on our work. We also thank you for bringing to our attention typos in Table 8. Below, we address the questions and concerns raised by the reviewer.

W1, Q3. Clarification on our Framework and DASH

We acknowledge that there may be some confusion regarding our theoretical framework. To clarify, we provide a high-level intuition and an example to the global response to better explain our framework. We hope this will help in understanding our approach. In our next version, we will include these explanations.

Regarding Q3, in our framework, the frequency of a feature in dataset is estimated by the number of data that contain the feature, divided by the total number of data. For example, in Figure 1 of the attached pdf, the feature represented by the violet has a frequency of 5/6. Also, we emphasize that the concept of learning by the frequency of features in our framework is intended to capture the characteristics of learning in real-world in abstract manner.

We hope this clarifies any confusion.

W2, Q2, L2. DASH is practical method

We believe that there may be some confusion. DASH is a practical algorithm that has been evaluated on real-world datasets such as CIFAR-10, CIFAR-100, and Tiny ImageNet. We suspect that we didn’t deliver this well enough and made the reviewer think that DASH operates only within the feature learning framework introduced in Section 2. However, that framework is intended to mathematically model the training procedure and hence analyze the test accuracy and training time trade-off between cold-start and warm-start. We propose a method which is ideal in our framework and show that the method mitigates the trade-off. Motivated by this ideal method, we propose DASH as a practical method which intuitively captures the idea of the idealized method.

Q5. What is meant 'This allows weights to easily change its direction.'?

It means that the mechanism enables weights to change their direction with even small updates. Here’s why this is important:

In unseen data, features are more likely to reappear, while noise from old data is less likely to be present. As a result, learning these features effectively enhances generalization. In other words, it is beneficial for neurons to align with class-relevant feature directions. By shrinking weights that are less aligned with these features, we make it easier for the model to adjust and align with the feature directions and enhance generalization performance.

W3 & L1 Underperformance DASH in SOTA setting

We agree that there was a lack of discussion in the main text regarding the underperformance of DASH compared to cold-starting in the SOTA setting, which we had only addressed in the appendix. We plan to include this in the main text in the next revision. As described in Appendix A.3, we believe that this underperformance is partially due to using hyperparameter optimized for cold-starting. Even though we did not tune the hyperparameters for DASH, it remains comparable to cold-start, which in fact better aligns with our theory (Theorem 4.1.). We believe that specifically tuning the hyperparameters for DASH might further enhance its performance and potentially surpass the cold-start method in the SOTA setting. The fact that our method performs better with naive training suggests it might be particularly effective for understudied datasets where SOTA settings are not well-established, as you pointed out.

Q1. The setting in which data is continuously added

While it is not entirely clear to us what the reviewer meant by “continuously added data”, we conducted an additional experiment that might address your concern. In this experiment, we added new data at each epoch for the first 500 epochs, then continued training for an additional 500 epochs (similar to Igl et al. (2021)). Cold-started models are trained for 1000 epochs using the entire dataset from the beginning. We used the CIFAR-10 on ResNet-18 with five random seeds.

As shown in the result plot in Figure 7 in attached PDF, there is a clear performance gap between warm-starting and cold-starting methods. We applied DASH and S&P every 50 epochs periodically until the 500th epoch, as you mentioned it cannot be done every time new data is introduced. Interestingly, DASH preserved learned features while S&P did not, even when we used the same $\lambda$ value of 0.3. As a result, DASH showed an increasing trend in train accuracy as the number of epochs increased, while S&P failed to learn anything during shrinking. Consequently, DASH converged faster than S&P while achieving better test accuracy.

Q1. The setting in which data is added from non-stationary distribution

DASH is designed for stationary data distributions, assuming features of the new data remain consistent with previous data. Because of this, the negative gradient has a high cosine similarity with the neurons that already learned features, allowing learned features to be retained. However, in non-stationary cases, the features may change continuously as new data is introduced. This means that it might be challenging to retain already learned features in such scenarios. The effectiveness of DASH may be limited in these non-stationary environments.

Adapting DASH for non-stationary scenarios (e.g., reinforcement learning) might require incorporating techniques similar to S&P, where perturbations are introduced to reactivate dead neurons. This approach could be crucial for maintaining network adaptability in changing data distributions. However, addressing these challenges in non-stationary environments is beyond the scope of our current work. We believe that this is a very important future direction.

Thanks for your time and consideration.

Best regards,

Authors

---

Reference

• Igl et al. Transient Non-Stationarity and Generalization in Deep Reinforcement Learning. In ICLR 2021

Dear Reviewer rFKW,

We sincerely appreciate your time in reviewing our work. Your feedback has helped us improve our research. We understand that you may have a busy schedule, but we'd be grateful if you could take a moment to look over our responses and ensure we've addressed your concerns adequately. If you have any additional questions or suggestions, please don't hesitate to let us know. We're ready to address any further points you may have.

Thank you again for your valuable contribution to our research.

Thank you for your feedback and for reconsidering the score.

• While we currently don't have the time to fine-tune hyperparameters for DASH in the SOTA setting, we intend to address this in our next revision. In the upcoming update, we will perform more comprehensive hyperparameter tuning, tailored specifically to each method, and incorporate these improved results into our findings.
• Thank you for your additional comments on our feature learning framework. We now understand the concerns you raised in W2 and Q2. As you pointed out, the learning of features can also be influenced by the strength of the features, especially when distinct features vary in strength. Our theoretical results and framework can be directly extended to this scenario by slightly altering our approach to consider $(\text{frequency}) \times \text{(strength)}$ as learning criteria instead of just frequency. Additionally, we believe our analysis can be extended to scenarios where feature strength varies across data by treating the set of features as a multiset, where multiple instances of the same element are allowed. Since the analyses in these cases are nearly identical to the one we have considered and follows the same underlying principles, we have assumed all features have identical strength for notational simplicity, without loss of generality. We hope these discussions adequately adress your concerns. We plan to include these discussions as a remark in the next revision to provide a clearer explanation to readers of our learning framework.

We welcome any further questions or comments you may have.

We sincerely appreciate the reviewer for providing valuable and constructive feedback on our work. Below, we address the questions and concerns raised by the reviewer.

W1 & W3

Check out our global response where we address these issues. We hope our explanation and ablation study resolve all concerns you may have.

W2, Q1. Proxy for the number of training steps

We believe there may be some confusion, let us clarify this point. For all our real-world experiments including Figure 1(b), we count the actual number of training steps until near-convergence (99.9% training accuracy), not the initial gradient norm. In contrast, we use the number of non-zero gradient data as a proxy of the training time ONLY in our abstract learning framework, for the sole purpose of theoretical analysis.

The reason for adopting a proxy of the training time is to capture the concept of “training time” within our abstract learning framework in which features/noise are learned/memorized in an “on/off” manner. Due to the binary (zero vs non-zero gradient) nature of our framework, we think the number of non-zero gradient data can serve as a reasonable proxy of the norm of gradient in practice. In addition, since the initial gradient norm is positively correlated with the number of training steps (in practice), we believe that the number of non-zero data point in our abstract framework can role as proxy for the number of training steps in practice. While the relationship between gradient norm and training time is not strictly linear, we think the number of non-zero gradient data to be the best possible measure of training time in our framework.

W4, Q2. Evaluation on ImageNet-1K

We appreciate your comments on scalability for larger datasets like ImageNet-1K. However, including experimental results for such datasets are challenging, as it would require repeating the training process 50 times until convergence—an extremely time-consuming process. As an alternative, we trained ImageNet-1K on ResNet18 using a setup similar to Figure 2 in Section 3.2. We used 50% of the data for pretraining and the full dataset for fine-tuning. Results in Figure 3 (attached PDF), show DASH outperforming both cold and warm starts in test accuracy and convergence speed, demonstrating its effectiveness on challenging datasets. Due to time constraints, we couldn't include results from training ImageNet-1K divided into 50 chunks in the paper. However, we plan to add results in the future using more than two chunks.

We also investigate the effect of pretrain epoch on warm-starting in ImageNet-1K classification with ResNet18. We conducted experiments using different pretrain epochs: 30, 50, 70, and 150. Figure 4 (attached PDF) shows a decline trend in accuracy for warm-started models as pretrain epoch increases. Interestingly, we noted a similar phenomenon in Figure 2 in Section 3.2, where warm-starting does not negatively impact test accuracy when training resumes from earlier pretrained epochs, such as 30 in this case. This observation aligns with our theoretical framework, which suggests that neural networks tend to learn meaningful features first and then begin to memorize noise later. We believe this phenomenon is persistent across different datasets, model architectures, and optimizers.

W5. Inconsistency in outperformance of DASH compared to warm-start

We acknowledge that there is some case that DASH underperform warm-start in the test accuracy at last experiment. However, It is important to note that the performance differences among all methods are not large, with most falling within standard deviation ranges. Also, we emphasize that performance at every experiment is crucial in practice. Therefore, we suggest examining not only the test accuracy at the final experiment but also the average test accuracy across all experiments. We ask the reviewer to revisit Table 7 in Appendix A.3., where we show that DASH consistently outperforms warm-starting in terms of average test accuracy, with improvements ranging from as little as 0.45% to as much as nearly 3%.

Minor Questions

• Q2. You're right that this is a matter of training data size. If a network isn't provided with sufficient data to learn enough features during the initial experiment, it may struggle to learn feature even when new data is introduced at following experiments. This concept is central to our Theorem 3.4. In scenarios where data is sampled from a stationary and class-balanced distribution (like real-world image data), there is less distribution shift due to class imbalance, as the samples are drawn iid.
• Q3. DASH is applied at the start of each experiment, not at every training step. When DASH's hyperparameter $\alpha$ is set to 1, only new data is needed for shrinking. Figures 7-8 in Appendix A.4 show DASH behaves similarly across different $\alpha$ values, including when $\alpha=1$. This indicates that DASH can be effective even when only the new data chunk is accessible. Furthermore, DASH can be utilized in online learning settings, where only newly introduced data is available during training. We evaluate DASH in such a setting using CIFAR-10 divided into 50 chunks and trained on ResNet18. Each experiment was conducted on a single chunk of data before moving to the next, with all other settings consistent with those detailed in Section 5.1. The results, shown in Figure 8 (attached PDF), show that periodically applying DASH outperforms baselines including S&P in terms of test accuracy. This confirms DASH's effectiveness with limited data access.
• Q4. Our use of $S_y$ is intended to represent the set of features associated with the true label $y$.
• Q5. We consider the case where $\tau$ is strictly smaller than the total number of features within a class. If $\tau = K$, then only data points containing all possible features would be correctly classified during testing.

Thanks for your time and consideration.

Best regards,

Authors

Thanks for your response. We are glad to hear that some of your concerns have been addressed. We want to respond to your additional comments regarding our supplementary experiment in the online learning setting.

1. Clarification on “Catastrophic Forgetting”

Before directly answering the questions one at a time, we want to clarify what we meant when discussing the reduction of “catastrophic forgetting” in an IID setting. In our experiment setup, we considered an online learning scenario where all data chunks are drawn from an identical distribution. While catastrophic forgetting is typically less severe in this IID setting as you pointed out, the inability to access previously introduced data in later experiments can still lead to significant forgetting issues. This is particularly problematic with shrinking-based methods, as they intentionally force the model to forget previously learned information through the shrinking process [3]. We considered the inability to retain learned information as a form of catastrophic forgetting. We apologize for any confusion this may have caused. To clarify, when we referred to "catastrophic forgetting" in our previous response to Reviewer LBum, we meant to highlight that shrinking-based methods inherently face this “forgetting” issue. Our point was that applying DASH and S&P for every chunk doesn't actually solve this problem. Instead, we attempt to mitigate the issues associated with shrinking-based methods by applying shrinking with some intervals.

2. Degradation of DASH and S&P in Figure 8

Every point at which degradation of test accuracy occurs in DASH and S&P coincides with the shrinking point of each method. The reason for different shrinking cycles is that we report the best learning curve (in terms of the last test accuracy) out of all $3\times3=9$ hyperparameter configurations (the interval between shrinkages $\in \\{10, 15, 20\\}$ & the shrinkage factor $\lambda \in \\{0.05, 0.1, 0.3\\}$) for each method. The performance drops immediately after applying parameter modification (i.e., resetting or shrinking) and subsequent performance recovery is common in the literature of resetting techniques [1,2].

Additionally, in our online learning setting, we only used a newly introduced chunk of data for training at each experiment, rather than training on all previously seen data. This resulted in slower accuracy recovery due to the limited amount of training data. Moreover, when applying shrinking (DASH or S&P) too frequently, the performance degradation caused by shrinking outweighs the learning effect from the limited number of new data points, leading to a decrease in overall test accuracy.

We would like to express our appreciation to the reviewer for your valuable and constructive comments. In the following, we address the points raised by the reviewer.

W1, Limitation. Clarification on our framework and link between the framework and the algorithm

We apologize for the unclear explanation regarding the framework and connection to our algorithm, DASH. We have clarified this in the general response. In it, we explain the role of cosine similarity and how shrinking with cosine similarity effectively captures feature retaining and noise forgetting. We hope this addresses any concerns.

W2. Additional results with different seeds

We understand your concerns about the statistical significance of our experimental results. While we aim to include as many random seeds as possible, our incremental learning framework is computationally intensive, which initially limited us to three random seeds. To address this issue, we're actively working on increasing the number of random seeds to enhance statistical significance. In response to your concerns, we can now provide results from two additional seeds (bringing the total to five) for experiments trained with SGD. As shown in Table 1 attached PDF in the global response, these new results demonstrate trends similar to our previously reported values. We hope this improvement addresses part of the concerns you raised.

Unfortunately, we cannot currently provide results for Tiny-ImageNet on VGG16 with SGD, as these experiments take approximately a week to train. However, we'd like to draw your attention to the results table in Appendix A.2, which shows our approach with Tiny-ImageNet on VGG outperforming other baselines. We hope this helps alleviate some of your concerns. In our revised version, we plan to include results with five random seeds for all experiments to ensure comprehensive statistical significance across our findings.

W2. Additional results with different hyperparameters

In our experiments, we used the same hyperparameters across multiple datasets and methods. It's important to note that these hyperparameters were not specifically tuned for DASH; rather, this approach was chosen for fair comparison, following the methodology of Ash & Adams 2020. Ideally, as you pointed out, we would have preferred to use the best hyperparameters for each method. However, due to limited computational resources, we couldn't exhaustively search for optimal values of base hyperparameters (such as learning rate and batch size) or method-specific parameters, since some runs take as long as a week. Given these constraints, we concluded that the best way to compare baselines was to maintain consistent hyperparameters across all methods.

In our original vanilla setting described in Section 5.1, the only hyperparameters available for tuning were learning rate and batch size. To address your concerns, we conducted additional experiments with CIFAR-10 on ResNet18 where we varied the learning rate while keeping the batch size fixed at 128. You can see these results illustrated in Figure 5, 6 and Table 2. Our findings confirm what we previously mentioned: there is little to no effect when using L2 Init, resetting, layer normalization, SAM, or reviving dead neurons. L2 Init and resetting show worse generalization performance compared to warm-started neural networks.

Questions

We have provided high-level intuition and explanations with examples for our framework in our global response, and we encourage you to refer to it. We hope this will clarify any misunderstandings.

Q1. What does the feature direction mean exactly in a rigorous way?

As detailed in our global response, there are two ways to minimize training loss: learning features and memorizing noise, with only learning features being beneficial for unseen test data. Therefore, the negative gradient can be decomposed into components of feature and noise. A feature direction represents the direction of updates that is effective for both training and test data.

Q2. What is the definition of feature?

We emphasize that the notion of features in our work is totally different from what is commonly referred to as 'features' in the context of neural network outputs, specifically the last hidden layer outputs. As discussed in our global response, we consider features to be the information contained in the input that is relevant to the label and therefore useful for generalizing to unseen data.

Thanks for your time and consideration.

Best regards,

Authors

---

Reference

• Jordan Ash and Ryan P Adams. On warm-starting neural network training. In NeurIPS 2020

Dear Reviewer t732,

Thank you for your time and effort in reviewing our work. We greatly appreciate your valuable and constructive feedback. Given your busy schedule, we would be grateful if you could review our responses to ensure we've adequately addressed your concerns. If you have any further questions or suggestions, please let us know, and we'll be happy to address them.

Thank you again for your valuable contribution to our research.

We would like to express our appreciation to the reviewer for your valuable and constructive comments. In the following, we address the points raised by the reviewer.

W1, Questions. Detailed description of DASH

We acknowledge that our description of DASH may have caused some confusion regarding the algorithm. DASH is a shrinking method that can be applied whenever new data is provided. As a result, the running average gradient computation (Q1) and shrinking (Q2) are performed at the beginning of the “experiment” (i.e., when a new data chunk is added). We realize that this description was omitted from Section 4.2 and was only mentioned in the Experimental Details (Section 5.1, line 359-360).

Q3: Since DASH is an initialization scheme performed independently of the optimizer, it can be combined with any optimizer, such as SGD, Adam, or SAM. Indeed, we have provided results using two different optimizers: SGD and SAM.

Q4: DASH calculates the weighted average gradient in training data sequentially, resulting in a memory complexity proportional to the batch size and model size, as only the gradient needs to be stored. In a scenario where gradients are computed one at a time, DASH's memory complexity would be equivalent to the model size. Please refer to our global response for more details, where we have included experimental results about both the memory requirements and computational overhead of DASH.

We plan to include these detailed explanations in our next revised version.

W2. Evaluation in online learning

Thank you for suggesting an evaluation of DASH in an online learning setting. We believe this is a great suggestion, as it addresses situations where storing all data is not feasible (e.g., due to memory constraints) and helps us explore how our algorithm can be applied to a wider range of scenarios. However, we'd like to clarify that our current setting is indeed realistic. It reflects real-world scenarios where new data continuously accumulates, such as in financial markets or social media platforms. This type of data accumulation is also studied in the work of Ash & Adams (2020).

While we are not entirely sure of the specific online learning setting you have in mind, we have conducted experiments with a setup where, instead of accumulating and storing data, the model learns from new data as it becomes available. For example, we divided CIFAR-10 into 50 chunks and performed experiments with ResNet-18, where each experiment was conducted on a single chunk of data before moving to the next. All other settings remained consistent with experiment details provided in Section 5.1.

To address the issue of catastrophic forgetting, which is common in online learning, we configured our experiments to apply DASH at specific intervals (e.g., every 10, 15, or 20 experiments), rather than after each experiment. Since there is no previous data available, we set $\alpha = 1$. After the 40th experiment, when the model had sufficiently learned, we stopped applying the shrinking process. We conducted similar experiments with the S&P method for comparison. It's worth noting that this variant is feasible because both DASH and S&P are algorithms that focus on adjusting the initialization of the model.

We tested DASH and S&P with intervals of 10, 15, and 20 epochs. For both methods, we explored shrinkage parameter ($\lambda$) of 0.05, 0.1, and 0.3. Notably, DASH's test accuracy consistently surpassed that of S&P across all hyperparameter configurations. In Figure 8 of our PDF, we plotted the results using the best hyperparameters for each method. These results demonstrate that DASH outperforms both the warm-starting baseline and the S&P method in terms of test accuracy. Based on these findings, we can conclude that DASH is suitable for online learning scenarios. If you could provide more details about the specific online learning setting you mentioned, as well as any additional baselines, we'd be happy to conduct further experiments to explore this area.

W3. Reasons for improved performance of DASH over cold-starting

The reason our theoretical framework does not explain DASH's superior performance in cold-start scenarios may be due to the binary nature (learn/not learn, memorize/not memorize) of our framework. However, we can provide an intuitive explanation based on our approach of retaining features and forgetting noise. When applying DASH in practical scenarios, even features previously considered "learned" can be further "improved" (i.e., model learns the feature with a greater strength) after noise is forgotten through shrinking, due to the continuous nature of real learning scenarios. We believe further improvement of feature has contributed to the performance improvement.

Thanks for your time and consideration.

Best regards,

Authors

---

Reference

• Jordan Ash and Ryan P Adams. On warm-starting neural network training. In NeurIPS 2020

Dear Reviewer LBum,

Thank you for taking the time to review our work and for providing such insightful and constructive feedback. We understand that you may have a busy schedule, but we wanted to follow up to ensure that our responses have sufficiently addressed your concerns. If you have any further questions or comments, we would be glad to hear them.