Experimental Design for Nonstationary Optimization

We thank the reviewer for their comments. We are glad that the reviewer appreciates the insights our work provides on trainability vs generalizability and designing resource constrained experiments. We address the reviewers' questions and concerns below.

[Various typos]
Thank you for pointing all of these out. We have fixed them and worked to improve the writing quality of the paper.

[Protocol 3 comparison]
Thank you for your suggestion. We have added a comparison of Protocol 3 to the other protocols to Figure 3. We see that the method rankings generated by Protocol 3 are either worse or within the confidence interval of Protocol 2. Furthermore, even if Protocol 3 manages to generate an accurate method ranking, in section 4.5, we see that it is not good identifying the best configuration for each method. Doing well in a setting like Protocol 3 is necessary to create “lifelong” agents that have unbounded lifetimes. While we do not claim that our version of this protocol is optimal, the underperformance at least invites further research.

[Why is the number of seeds and tasks different in different types of networks, same for the gradient steps?]
The number of seeds and tasks are lower for the ResNet experiments just for practical compute reasons, as it takes longer to train ResNets and you can fit fewer runs onto a single GPU. The number of gradient steps per task were set based on approximately the amount of time it took for training to converge.

We thank the reviewer for their valuable comments and helping improve our work. If we have addressed your questions and concerns, we would appreciate it if you could further support our work by increasing your score. If there are more questions/concerns, please let us know.

We thank the reviewer for their comments. We are glad that the reviewer finds our work timely, understudied and important. We address the reviewers' questions and concerns below.

[Inconclusive Results]
We have updated the paper with more concrete takeaways (specifically for sections 4.1, 4.2, and 4.3). If there are other parts that the reviewer would like expanded on, please let us know.

[Non-aggregated analysis]
We thank the reviewer for their useful suggestion. We have added Figure 15 and Table 3 that do a non-aggregated analysis of the results in Section 4.3 and Figures 10 and 11 that show a non-aggregated analysis of the results in Section 4.4 (please see below for further discussion). Furthermore, Figure 12 (present in the original manuscript) shows a non-aggregated analysis of the results in 4.6.

[Improvement of Figure 2]
We have moved this Figure to the appendix (Figure 14), and replaced it with a plot showing the actual values of the train and test accuracies used to generate the rankings, and a plot showing how much the different rankings correlate with each other. From this Figure, we can see that while there are a few overlaps in the error bars (representing 95% empirical confidence intervals), most of the results would stay fairly stable.

[Figure 3 results]
We have updated Figure 3 to be a bar plot and added the missing legend. We also added Protocol 3 to the evaluation. Each bar represents the performance of one of the protocols described in Section 3.3. You are right that perhaps our original claim was too strong. Protocol 2 is not outright better than the other protocols, but it is the only one that is not outright beaten by another protocol (taking into account the confidence intervals), and it is the outright best on Shuffled CIFAR-10. It outright beats Protocol 1 on 2 of the 6 settings, and approximately matches performance (within CI) on the other 4. Protocol 3 similarly is outright worse on 2 settings compared to Protocol 2, and within the CI on the other 4. Thus, we argue that there is not much advantage to be gained from using Protocols 1 or 3, and at least a few settings where it is disadvantageous to do so. We have updated Section 4.1 with this discussion.

[Clarification for statistical bootstrapping]
With statistical bootstrapping, we sample $B$ datasets of size $n$, compute some statistic using each of these datasets, and use the empirical distribution over the statistic to create confidence intervals or compute standard errors for the sample mean of the statistic. In our case, we are sampling partitions over the seeds, $P$, and using the partition to estimate some statistics that are a function of this partition, $f(P)$. Specifically, statistics such as the rank correlation between the rankings generated by two protocols or the binary random variable indicating whether the best config was selected by a protocol are a deterministic function of the partition. In our case, since when people do hyperparameter search, they usually only sample one partition, we set $n=1$ and $B=1000$. Thus, we are sampling $1000$ different partitions (with replacement), calculating the statistic for each of the partitions, and displaying the 95% empirical confidence interval that results. We have added a discussion of this to the appendix.

[Clarifying seeds in Section 4.1]
Of the 20 seeds, 10 are used to create a held out ranking. These are not available to any of the protocols when doing model selection or evaluation. The idea is that they represent unseen task sequences, and we can see how the method rankings generated by each protocol transfer to unseen task sequences. Protocol 1 uses the validation accuracy of all 10 of the available seeds to select the best configuration for each method and uses the test accuracy of those same seeds to rank/evaluate them. Protocol 2 uses the test accuracy of 5 of the 10 available seeds to select, and the test accuracy of the other 5 to rank/evaluate. We also add Protocol 3 to our evaluation, which uses the test accuracy of all 10 seeds over the first 10% of tasks to select, and the test accuracy of all 10 seeds over the latter 90% of tasks to rank/evaluate. We have updated the writing in 4.1 to make this procedure more clear.

We thank the reviewer for their comments. We are glad that the reviewer appreciates that our work provides a unified and independent evaluation of the works done by the plasticity community. We address the reviewers' questions and concerns below.

[Use of unrealistic benchmarks]
Our use of the datasets and nonstationaries studied was motivated by the fact that these are the types of settings that are in fact used in plasticity research, and we wanted to give empirical recommendations to the plasticity community. Several recent works [1,2,3,4,5] have used these settings as a way to simplify and study the core problem, including Dohare et al., 2024. We also do acknowledge the need for newer and more difficult settings in our discussion of our results in Section 4.3. We also would like to point out that the Noisy distribution shift is representative of a fairly realistic distribution shift where you get higher quality data over time, and is encountered in both the supervised setting and the reinforcement learning setting. Given that these are commonly used settings and we do include a realistic setting, we do not think this should be a reason to reject the paper.

[Use of Train/Test sets]
Our work aims to facilitate the research that people have started doing in nonstationary optimization/plasticity. While the setting with only a stream of data is a valid continual learning setting, there are several use cases, both in research and in real systems, where there would be a train/test split in order to quantify the generalization capabilities of the system. Many works [1,2,3.4,5] in both the general continual learning field and in the plasticity field have also introduced such a split, as it allows us to better probe the abilities of these systems.

[Use of random seeds]
Similar to our point above, we agree that when continual learning systems are deployed, it probably would not make sense to have random seeds. The goal of our work, however, is to facilitate the research on these systems, and being in order to evaluate whether the methods we are developing are overfitting to a single task sequence or initial random conditions, we need to evaluate with multiple seeds, as has been done throughout the continual learning literature [1,2,3.4,5].

[Continuous hyperparameter selection] Continuous hyperparameter selection is a separate research problem within continual learning. It would most likely improve the performance of the methods we study, but as we are trying to facilitate and provide recommendations for research in plasticity, applying those methods in our work when they are not used in the plasticity community would make our recommendations less applicable and helpful to this community.

[Limited settings]
Our goal with this work was not to create a realistic continual learning system or to provide empirical guidelines on how to create a realistic system. It was to provide empirical research recommendations to the growing plasticity community. The work in this community tries to create methods that can deal with nonstationary optimization and to study it, we need to make simplifying assumptions and make sure we are providing repeatable results. This could lead to limited settings, but is also useful to make progress on the problem.

[Small scale of experiments]
As a part of this study, we ran 27,600 trials across multiple datasets, distribution shifts, architectures, and methods. The ResNet architecture is also one of the larger architectures studied in plasticity research. We believe that this is an acceptable scale for such an empirical study.

[Online algorithm doing better than baselines]
We are not sure of how well the online baseline was tuned in prior work, but one of the contributions of our work is showing that a well tuned online baseline (with L2 regularization, we allow L2 regularization for all methods except L2-Init) can be quite competitive and beat out several existing plasticity methods.

[Gap between train and test accuracies for different methods]
We have moved the original Figure 2 to the appendix (it is now Figure 14), and replaced it with a bar chart (Figure 2a) showing the train and test accuracies achieved by each method. Here, we can see that it is not just the ranking but the actual performance that can change significantly going from train to test. For example, methods such as L2-init and Hare and Tortoise are worse than the best methods on training accuracy by 10-20% on both Shuffled CIFAR-10 and Permuted CIFAR-10, and yet they are the best and second best methods for test accuracy on those two settings. We can also see in Figure 2b that several of the method rankings generated when selecting for the best train accuracy are actually anti correlated with method rankings generated when selecting for test accuracy.

I appreciate authors’ responses and their modifications, which I think has improved the paper’s presentation. Specifically, the revision has addressed most questions, except for the most important question above (i.e., my first question).

On the negative side, I think the revision (and the response) falls short of concrete improvements towards addressing the weaknesses mentioned above. In particular, the first contribution of the paper, as mentioned is the abstract, is the evaluation and comparison of existing algorithms, which I think can be potentially valuable; however the authors performed this comparisons on some of the simplest and most unrealistic benchmarks available in the literature. This significantly weakens the intended contribution (given that this is a fully empirical paper). The argument that these benchmarks are also used in existing works is not convincing, because those works also include several better and more realistic benchmarks that could have been used here (refer to the weaknesses above for pointers and detailed discussion).

The fact that the online baseline in the reported experiments perform as well as an average LoP method, potentially suggest that no Loss of Plasticity occurs in the tested environments, rendering LoP studies irrelevant on the tested benchmarks (otherwise the online-baseline’s relative performance would have considerably decayed over time due the lost plasticity, by definition of LoP).

As another weakness regarding contribution, the last line of the abstract claims to “provide concrete recommendations and analysis that can be used to guide future research”. While the paper provides some interesting insights that can be useful for future research, I think many of these recommendations are not actually relevant for realistic continual settings for the reasons detailed in the weakness section above; and overall they do not direct the field towards addressing challenges that really matter in long term.

On the plus side, some of the insights and recommendations are interesting, especially for the simplified setting of non-stationary supervised learning. The gap between train and test can be important (I have personally observed it in some of my older continual learning experiments). I also think sensitivity analyses like Fig. 8 are useful for future use of these methods.

Regarding my score, the grounds for rejection are not due to mistakes or inaccuracies in the paper but rather my judgment of the overall contribution of the paper, considering the rather high bar for ICLR. This judgment is, of course, influenced by my subjective perceptions of what matters in LoP research, which may prove inaccurate with the test of time. I actually lean toward a score of 4 (but unfortunately the possible scores are quantized at 3 and 5). Given all the improvements in the revision and the remaining concerns, I am keeping the score of 3 and reducing my confidence to 2, favouring judgment about the contribution from other expert reviewers.

We thank the reviewer for their comments. We are glad that the reviewer finds our work novel, timely, and useful for the plasticity community both in terms of what we should focus on and in terms of helping make efficient experiments. We address the reviewers’ questions and concerns below.

[Fixing the writing errors]
We thank the reviewer for pointing out the writing errors in our manuscript. We have gone through and made the corrections.

[Permuted CIFAR experiment with ResNet]
While we agree that permuted CIFAR-100 with ResNet is not typically seen in the community and that it would seem ResNets would not be well suited for the task, we chose to include them because we saw the networks did seem to be able to achieve perfect training accuracy and a nontrivial test accuracy. The permuted input task is already a very artificial one, only used to probe the abilities of continual learning methods. Despite this, there is still some structure there that ResNets are able to learn. If the reviewer strongly feels that this experiment detracts from the paper, we can do so, but we would like to emphasize that doing so would not affect our claims and analysis.

[Figure 2 explanation]
In Figure 2, we perform Protocol 2 for model selection and evaluation. We use half the seeds to find the best hyperparameter configuration for each model, and the other half to evaluate the best configurations and create a ranking of the methods. We show the ranks of the methods for each of the settings in our paper. The left plot shows the method rankings when selecting/evaluating for best training accuracy, and the right plot shows the rankings when selecting/evaluating for best testing accuracy. In the updated manuscript, we have moved this Figure to the appendix (Figure 14), and replaced it with a plot showing the actual values of the train and test accuracies used to generate the rankings, and a plot showing how much the different rankings correlate with each other.

[Updating the claim in Figure 4 about train and test accuracy not correlating with each other]
The reviewer is correct that we need to be more precise about this claim. What we meant to convey is that trainability correlates with generalizability only up to a point, after which continuing to improve trainability does not end up correlating with the end goal of improving model performance. This suggests that (1) for the types of settings presented in this study (which are representative of what is currently studied in the literature), we should shift our focus from improving trainability to the problem of improving generalizability. (2) Studying trainability could still be a valuable problem, but we should find harder settings to do so.

We have updated the discussion of this claim (in Section 4.3) in the paper and thank the reviewer for the valuable suggestion.

[Clarification on “How many tasks do you need to include in your training sequences?”]
Yes, we meant how many tasks do you need to include when doing hyperparameter selection. This has been updated in the manuscript.

[Legend for Figure 3]
We have updated Figure 3 to be a bar plot and added the missing legend. We also added Protocol 3 to the evaluation. Each bar represents the performance of one of the protocols described in Section 3.3. Thank you for pointing out the missing legend.

[Figure 4 with loss instead of accuracy]
We have added this Figure to the Appendix (Figure 12). While some of the trends are different, there are still only two settings that have a statistically significant positive correlation between train and test accuracy for the top configurations. Thank you for the suggestion.

[X axis label on Figure 5d]
Yes, sorry for the mixup. We have updated the label.

We thank the reviewer for their valuable comments and helping improve our work. If we have addressed your questions and concerns, we would appreciate it if you could further support our work by increasing your score. If there are more questions/concerns, please let us know.