Predicting the Susceptibility of Examples to Catastrophic Forgetting

Thank you for your thoughtful review. We appreciate your recognition of our extensive empirical study and the practical value of our findings.

Regarding the connection between learning speed and catastrophic forgetting, our primary contribution is identifying and characterizing this phenomenon. While a formal theoretical analysis is an exciting future direction, many fundamental insights in machine learning (e.g., double descent, simplicity bias) were first observed empirically before theory followed. Despite the lack of theory, our results have several more intuitive explanations: later-learned examples likely depend on more complex or composite features, making them more susceptible to forgetting when new tasks are introduced, as those features are more likely to break first. Another intuition is that a similar pattern occurs in human learning, where foundational skills persist longer than more complex ones.

On the choice of hyperparameters $q$ and $s$, our goal with SBS is to demonstrate that leveraging learning speed can improve continual learning across different algorithms and settings. While a more theoretically derived selection method could be valuable, it is non-trivial and beyond this paper’s scope. Instead, we propose a practical heuristic -- using a related self-supervised task (e.g., RotNet) to tune these parameters without additional labels. While RotNet was chosen arbitrarily, our experiments show that this approach consistently finds the best hyperparameters across all the setups we evaluated.

Thank you for your thoughtful review and for your willingness to consider raising your score. Below, we address your concerns in detail.

concern 1

The advantages of picking $q=s=20\%$ across different settings are scattered throughout the figures of the paper: see Figs 4(a-b), 16(a-b), 17(a-b), 18(a-c), 19(a-c), 20(a-c), 21(a-c) and 24(a-c). However, we will add a dedicated appendix section explicitly showcasing results for picking $q=s=20\%$, hopefully making this point clearer.

Additionally, we will add a new row to Table 1, reporting the results of SBS with these fixed hyperparameters, showing the advantages that can be gained by SBS with and without the hyper-parameter search. The results that will be added are:

to compare, the baseline results of random sampling (Table 1 in the original manuscript) are:

While removing hyperparameter tuning slightly reduces performance, SBS still consistently outperforms other methods across all settings. This demonstrates that SBS is effective even without extensive tuning.

We will also add computational cost for both the RotNet-based version and the fixed-hyperparameter version in the appendix to provide a complete picture.

concern 2

Thank you for catching this oversight. You are correct that Table 1 (main paper) and Table 2 (supplementary) both report results for TIL. We also conducted experiments for CIL, and while they were qualitatively similar (though lower in absolute performance due to the increased difficulty of CIL), they were mistakenly omitted from App. B.

We will correct this in the camera-ready version by including the CIL results in App. B.

concern 3

This is an interesting question. While we do not have a formal theoretical proof, we can provide an intuitive explanation based on our observations.

While the training error of networks tends to go to $0$, meaning that every example in the dataset is going to be learned at some speed, the test error often does not. The training examples that are learned slowest are often those that correspond to points near test points that the model simply did not learn, meaning they are inherently difficult for the model to generalize to, even in non-continual settings. Therefore, in the CL settings, where the replay buffers are limited, we want to focus on examples that the model can generalize from, and removing the slowest-learned examples helps allocate space to more useful ones, improving overall performance. However, we note that the better the model can perform on the original task, the smaller the number of these unhelpful slower-to-learn examples, allowing us to remove less of them to get better performance.

comparison to other methods

You suggested comparisons with methods that retain complex or diverse samples, specifically Rainbow Memory (RM) [1] and LARS [2]. Below, we summarize how our method relates to them and provide additional quantitative comparisons.

Rainbow Memory (RM)

The original RM paper focuses on blurry-CIL settings, whereas our study considers disjoint settings. The RM paper itself notes that RM does not consistently improve performance in disjoint settings, often performing similarly to random sampling. Since SBS outperforms random sampling, this suggests that SBS is superior to RM in these settings.

That said, we acknowledge RM’s potential value and have already discussed its relation to SBS in App. G. We will make this connection more explicit in the main paper and include RM results in Table 1 for direct comparison.

LARS (from BAGS of Tricks)

LARS is presented in [2] as one of several "tricks" to improve CL and was not presented as a stand-alone sampling method, which is expected to improve performance across different settings. LARS suggests to sample the buffer randomly, but to remove examples from it in a suggests loss-aware, where examples with low loss will be removed more frequently. We agree that adding a comparison to LARS can strengthen our work, and we added such a comparison to the paper. To compare to SBS, we isolated LARS from the rest of the "tricks" suggested in [1], and evaluated it on the different datasets and buffer sizes in Table 1. Below are the results:

While LARS improves over random sampling in most cases, its gains are smaller than those achieved by SBS. Following your suggestion, we will add both RM and LARS results to Table 1 in the camera-ready version for a clearer comparison.

---

[1] Bang, Jihwan, et al. Rainbow memory: Continual learning with a memory of diverse samples. (CVPR 2021)

[2] He, Tong, et al. Bag of tricks for image classification with convolutional neural networks. (CVPR 2019)

Thank you for your thoughtful review. We appreciate that you found our analysis comprehensive and our results well-supported. Below, we address your comments in detail:

Comparison with non-uniform sampling in continual learning methods

Most competitive continual learning methods rely on random sampling [1] because non-uniform sampling strategies generally fail to provide consistent improvements across continual learning methods and settings. SBS is, to our knowledge, the first sampling method that consistently enhances different continual learning methods, making this a key contribution of our work.

Some methods, like iCaRL [2], incorporate specific non-random sampling strategies (herding for iCaRL), but these are tightly coupled to the method itself and do not generalize well. For instance, herding does not work effectively with most competitive continual learning methods, while iCaRL is designed around herding and performs poorly with other sampling methods, including random sampling. However, for completeness, we will follow your suggestion and include additional results on continual learning methods with different sampling functions in the supplementary material and briefly discuss them in the main paper.

Max entropy and IPM performance

Max entropy (Max Ent) selects examples with the highest entropy in network logits. While a common baseline, it is rather a naive baseline, and it is known to perform poorly in practice, as also observed in our results.

IPM [3] selects a diverse subset of examples from a dataset, and tries to maximize the information that this set has. The original IPM paper was not focused on continual learning, and showed advantages of this subset in other fields, such as active learning, representation learning, and GAN training. However, prior work [4] indicates that its effectiveness in continual learning is inconsistent, which aligns with our findings. A possible explanation is that IPM prioritizes "informative" examples, which often overlap with slower-to-learn examples. Since our analysis shows that slower-to-learn examples are more prone to forgetting, this selection bias may explain IPM’s weaker performance. Interestingly, IPM performs better in settings where learning is easier or buffer sizes are large (e.g., CIFAR-100-20 with a 10k buffer), which is consistent with this hypothesis.

Choice of hyper-parameters

As noted in our response to reviewer hXyj, the performance of $q=s=20\%$ is already present in multiple figures throughout the paper, including Figs. 4(a-b), 16(a-b), 17(a-b), 18(a-c), 19(a-c), 20(a-c), 21(a-c), and 24(a-c), showing its advantages over random sampling. However, to make this clearer for future readers, we will add a dedicated paragraph consolidating these results in the camera-ready version. Additionally, we will include SBS results with these fixed hyper-parameters in Table 1, demonstrating that they still significantly outperform random sampling without requiring additional tuning or additional computation. The exact numeric results for $q=s=20\%$ that will appear in Table 1 are also provided in our response to reviewer hXyj.

---

[1] Wang, Liyuan, et al. "A comprehensive survey of continual learning: Theory, method and application." (PAMI 2024)

[2] Rebuffi, Sylvestre-Alvise, et al. "icarl: Incremental classifier and representation learning." (CVPR 2017)

[3] Zaeemzadeh, Alireza, et al. "Iterative projection and matching: Finding structure-preserving representatives and its application to computer vision." (CVPR 2019)

[4] Brignac, Daniel, Niels Lobo, and Abhijit Mahalanobis. "Improving replay sample selection and storage for less forgetting in continual learning." (ICCV 2023)