Towards Realistic Hyperparameter Optimization in Continual Learning

• The work is not really about an assessment of HPO in CL. The only HPO strategy that is considered here is grid search. What is being assessed is how different ways in which you can pick up the validation set(s) as part of datasets received continual-style, influence the predictive performance of CL strategies whose hyperparameters are not selected having a continual HPO setting in mind. Continual HPO is actually about how you perform positive information transfer between tasks at the level of the hyperparameter adaptation strategy, without resorting to inefficient grid-like search and while catering for knowledge retention between tasks. Which is pretty much the HPO equivalent of CL for parameter update. This choice is evident from the fact that while the paper reference actual attempts at performing continual HPO, it does not benchmark any of them. The sheer effect of this is that there is little "realism" in all the tested strategies, in CL sense.

• An additional limit to the claimed generality of the study is that the benchmarking exercise is not about CL. It is about a very specific family of methodologies, i.e. replay-based ones, which are arguably the ones whose predictive performance depends less on model-selection/HPO choices. Before coming down to the conclusions that existing benchmarks datasets are not sufficient to differentiate the effects of different HPO strategies, one would need to explore the effect on regularization-based and of architectural CL methodologies. Those are the ones which are likely to be heavily influenced by the choice of hyperparameters. Consider for instance the weight of the regularization terms to avoid forgetting, and their impact on the stiffening of the model ability to adapt its parameters on the current task. Even more so, if one considers architectural approaches where HPO may entail topological changes to the network which may render the adaptive model discontinous (at inference time) between task experiences (and hence strategies such as current-task HPO will loose meaning at all).

• The empirical analysis, while robust, focuses on very few hyperparameters and does not provide any insight into the differential effect of different hyperparameters (e.g. how much they tend to vary across experiences). Additionally, the empirical setting assumes partitioned classes. While being a common assumption, this is highly unrealistic, which contrasts with the claimed realism targeted by the work.

The authors should have discussed in more detail the pros and cons for the different HPO strategies. In the following some ideas what could be discussed or my opinion on each of the strategies:

• first-task HPO: many CL methods, e.g., DER++, have hyperparameters that only matter if there is more than one task. how can you tune them in this case?
• current-task HPO: this is useless since it does not allow to measure for important metrics such as forgetting.
• seen-tasks HPO: this increases the required memory size since this is data we cannot train on. some scenarios don't allow for any memory. is it better to use the additional data for HPO or for training?

Besides the clear disadvantages of the methods discussed above, we don't observe this empirically.

The authors imply that we care about "end-of-training" performance. They completely overlook that we actually care about anytime performance and that the concept of "end" doesn't exist in CL. It would be great if the authors would show how performance changes over time. Besides this fact, it's a common plot in many CL papers as well as a nice ablation study.

What about other metrics such as forgetting? As I've mentioned above, some HPO strategies would completely ignore it and we'd assume to see this in this metric.

The authors argue that all tested methods perform equally and therefore "first-task HPO" should be preferred since it is computationally cheap. In my opinion such a recommendation should not be made. The authors should make clear when exactly this choice is justified.

The novelty of this work is questionable. The paper surveys the different HPS approaches in the CL setting and describes the limitations and challenges of training CL with HPO. This is most likely a benchmark and datasets papers submitted to the main track of ICLR. The key contribution of this work is the code, that can be used to test further and novel HPO methods in the CL paradigm.

• The paper claims that none of the compared HPO methods seems to be superior to the others. I found it a bit disappointing that there was no further examination of why that was the case. Does it really make no difference (in terms of learning performance) which HPO method one uses, or could it be that certain HPO methods are better suited to CL methods with some particular characteristics?
• The benchmark contains just five CL methods, and the paper does not really explain why these particular methods were selected.
• I think three runs are too few to have a relatively accurate picture of how the various HPO methods compare to each other.
• I think the paper would benefit from a short section on its limitations. For instance, there are online CL settings and CL settings without crisp transitions between tasks, which might not be compatible with the CL definition followed within the paper. Also, the experiments are performed on only one data modality (images) and with only one neural network architecture.
• I think that having access to validation sets from previous tasks is arguably an unrealistic assumtpion, not only for the reason given in the paper, but also because it might be more beneficial to actually use the data in these sets during training instead of just using them as validation sets.
• How did you decide on the value of 0.5%, which you use to decide which table entries to write in bold.