Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling

Can you consider applying aL-SAR on more diverse datasets (e.g., text, multimodal data) to demonstrate the approach’s generalizability?

$\to$ As suggested, we applied the proposed aL-SAR to a multi-modal concept-incremental task, using LLaVA-1.5-7B as the architecture and multi-modal data such as Bongard-OpenWorld and Bongard-HOI (details are provided in Sec. A.13), and we summarized the results in Tables H and I. We compare aL-SAR with ER, the best-performing baseline, under the total constraint.

Table H. Comparison between aL-SAR and ER on the Bongard-HOI under total constraint (Memory budget : 27.05GB)

Table I. Comparison between aL-SAR and ER on the Bongard-OpenWorld under total constraint (Memory budget : 27.05GB)

[1] Lee et al., Online Continual Learning on Hierarchical Label Expansion, ICCV 2023

[2] Prabhu et al., Online Continual Learning Without the Storage Constraint, arXiv 2023

Can the authors compare the al-SAR with other methods under the epoch-based setup to show the effectiveness of the method with larger memory constraint?

$\to$ Yes, we compare aL-SAR with baselines under the epoch-based constraint and a large memory constraint, summarizing the results in the table below. Specifically, we assume a memory infinite setup [2], which allows storing all samples in episodic memory, as well as logits and previous models for distillation.

While ER outperforms other methods except our proposed aL-SAR in total constraint, ER-CCLDC and MIR outperforms baselines. However, MIR and ER-CCLDC consumes 4.12 times and 4.66 times more FLOPs than ER, as MIR requires additional forward and backward passes to retrieve maximally interfered samples from episodic memory, while ER-CCLDC requires 10 times more forward passes for collaborative learning and distillation using augmentations. Additionally, ER-CCLDC requires a sibling network for collaborative learning, which further increases memory storage requirements. In contrast, aL-SAR outperforms baselines with less computational cost and negligible additional memory budget.

As discussed in Line 39 and Fig. 1, we argue that comparing methods under the same epoch constraint does not provide useful information about computational cost as a method can do a lot of computation during a single epoch which may take much longer time than other methods. Furthermore, we would like to emphasize that we compare aL-SAR with baselines under the large memory constraint (i.e., memory infinite setup) in Sec. A.21, where aL-SAR demonstrates superior performance compared to the memory constraint.

Table E. Comparison between aL-SAR and other baselines on the CIFAR-100 Gaussian Setup under total constraints, using 3 epochs of training per sample encounter with an infinite memory budget.

Table F. Comparison between aL-SAR and other baselines on the CIFAR-100 Disjoint Setup under total constraints, using 2 epochs of training per sample encounter with an infinite memory budget.

Table G. Comparison between aL-SAR and other baselines on the ImageNet Gaussian Setup under total constraints, using 0.125 epochs of training per sample encounter with an infinite memory budget.

Though designed to be efficient by only computing class based gradient, the similarity-aware retrieval approach still requires gradient similarity calculations, which could introduce overhead in large-scale applications.

$\to$ We can minimize the computational overhead in large-scale applications by introducing a hierarchical structure that groups similar classes together. This allows us to focus on group-wise similarity rather than class-wise similarity, thereby reducing computational overhead. Since many real-world scenarios involve hierarchical relationships between old and new tasks [1], incorporating class groups enables aL-SAR to better address these real-world challenges. Moreover, as mentioned in Sec.A.17, by utilizing only 0.05% of the entire model’s parameter gradients, we believe that aL-SAR will maintain computational efficiency in large-scale applications.

The authors should compare the method with model based like iCaRL.

$\to$ Sure. We add comparisons with iCaRL in the table below. In CIFAR-10 experiments, we set the computational budget and memory budget as 293.6 TFLOPs and 25.16 MB, respectively. In CIFAR-100 experiments, we set the computational budget and memory budget as 440.4 TFLOPs and 25.16 MB, respectively. Note that aL-SAR has lower FLOPs by virtue of proposed adaptive layer freezing.

Table A. Comparison between aL-SAR and iCaRL on CIFAR-10 Gaussian Setup under total constraint

Table B. Comparison between aL-SAR and iCaRL on CIFAR-10 Disjoint Setup under total constraint

Table C. Comparison between aL-SAR and iCaRL on CIFAR-100 Gaussian Setup under total constraint

Table D. Comparison between aL-SAR and iCaRL on CIFAR-100 Disjoint Setup under total constraint

We sincerely appreciate your effort in reviewing our submission. We gently remind the reviewer that we tried our best to address your concerns via our replies and manuscript revision. As the discussion period is nearing its end, we would be glad to hear more from you if there are further concerns.

The method doesn't address catastrophic forgetting directly, instead focusing on resource efficiency. It's unclear how the method balances the trade-off between resource efficiency and forgetting prevention.

$\to$ Thank you for your insightful question. The proposed adaptive layer freezing inherently (or naturally) balances the trade-off by preventing unnecessary parameter overwriting. In other words, our proposed adaptive layer freezing provides advantages in both resource efficiency and forgetting prevention. Specifically, we believe this advantage stems from preventing parameter overwriting, which is one of the primary causes of catastrophic forgetting [4, 5]. Specifically, when model parameters are continually overwritten by new data, fully updating the model at every iteration can result in severe forgetting. On the other hand, freezing parameters can limit the model’s plasticity to learn new concepts [6]. To address this stability-plasticity dilemma, our proposed aL adaptively freezes layers based on the informativeness of each training batch, updating parameters only when the batch contains sufficiently informative data. Since each training batch includes both recently encountered and past data from episodic memory, aL freezing may freeze fewer layers for batches containing undertrained or forgotten data, while freezing more layers for well-trained batches. This approach prevents unnecessary parameter overwriting and maintains a balance between stability and plasticity.

Furthermore, we account for forgetting in similarity-aware retrieval (SAR) by using discounted use frequency, as described in Sec. 3.2. If a sample was frequently used in the past but seldom used in recent iterations, its knowledge may have been forgotten despite its high use frequency. To account for this, we apply an exponential decay to the use frequency, inspired by the exponential decay model of forgetting. To demonstrate the effectiveness of forgetting prevention in aL-SAR, we include a comparison of forgetting rates between aL-SAR and the baselines in Sec. A.24, showing reduced forgetting across various benchmarks and setups.

The computational overhead of maintaining class-wise gradient similarities might become prohibitive for problems with a large number of classes or frequent distribution shifts.

$\to$ We can minimize the computational overhead in large-scale applications by introducing a hierarchical structure that groups similar classes together. This allows us to focus on group-wise similarity rather than class-wise similarity, thereby reducing computational overhead. Since many real-world scenarios involve hierarchical relationships between old and new tasks [7], incorporating class groups enables aL-SAR to better address these real-world challenges. Moreover, as mentioned in Sec.A.17, by utilizing only 0.05% of the entire model’s parameter gradients, we believe that aL-SAR will maintain computational efficiency in large-scale applications.

[1] Li et al., Smartfrz: An efficient training 415 framework using attention-based layer freezing, ICLR 2023

[2] Yuan et al., Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training, NeurIPS 2022

[3] Liu et al., AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning, arXiv 2021

[4] Zhang et al., Continual Learning on Dynamic Graphs via Parameter Isolation, SIGIR 2023

[5] Kirkpatricka et al., Overcoming catastrophic forgetting in neural networks,PNAS 2017

[6] Zhao et al.,SAFE: Sl ow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models, NeurIPS 2024

[7] Lee et al., Online Continual Learning on Hierarchical Label Expansion, ICCV 2023

[8] Lee et al., Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning, AAAI 2023

[9] Yuan et al., Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training, NeurIPS 2022

Using the layer freezing technque for recuding the model params would always block the shallow layers (i.e., layer 1-n) and their representation would be outdated. This would be problematic for frequent distribution shifts.

$\to$ Thank you for the insightful comments. But our proposed adaptive layer freezing method (i.e., aL) does not encounter such issues during frequent distribution shifts, since when temporal distribution shifts occur, the adaptive layer freezing (called ‘aL’) tends not to freeze layers, including shallow ones. Specifically, as shown in Fig.8 in Sec.A.10, we observe that batch freezing criterion (BFC) values, which quantify the net benefit in the information gained by the model when we freeze layers given an input batch $z_t$, tend to remain negative on average, even in shallow layers (note that the plot is significantly smoothed to highlight overall BFC trends, while actual values fluctuate depending on the input batch). This indicates that freezing is not beneficial for most batches, likely due to insufficient training in continual learning and the continuous introduction of new samples. However, for input batches with little information, BFC temporarily becomes positive, allowing for freezing.

Furthermore, at the task boundaries, we observe a sharp drop in BFC values, indicating that the layers are less likely to be frozen, allowing them to learn from the new, previously unseen data. This demonstrates that BFC effectively handles distribution shifts, even though our method does not rely on any task boundary information. Additionally, while shallow layers exhibit higher BFC values compared to deeper layers, suggesting they are more frequently eligible for freezing, they are not always frozen, especially when faced with shifted distributions.

Moreover, we demonstrate that aL-SAR performs effectively under frequent distribution shifts by conducting experiments in the Gaussian-scheduled setup, where the data distribution follows a Gaussian distribution and shifts at each time step, unlike the disjoint setup, which assumes distribution changes only at explicit task boundaries.

Although authors have illustrated the process of calculating Effective Use Frequency in Sec.3.2 and Sec.A.17, it is still vague for me that how to calculate cosine similarity of the gradients between classes? In A.17, do it that using 0.05% of the model parameters means that randomly sample params among unfrozen layers?

$\to$ No, not just among the unfrozen layers but across all layers before training. Specifically, we first randomly select 0.05% of the model parameters. Then, for each iteration, we use the gradients of the pre-selected, which are unfrozen by adaptive layer freezing, to update the class-wise gradients. Specifically, we employ an EMA estimation of the class-wise gradients, updating them every batch iteration using the gradients of selected parameters in an EMA manner. While not all EMA-estimated class-wise gradients are updated every iteration (since gradients of frozen layers are not computed), all elements of the estimated class-wise gradients are updated every $m$ iterations, as we unfreeze all layers every $m$ iterations, as mentioned in Sec. A.1. To form the similarity matrix, we first compute the element-wise product of class-wise gradients and then average them over all class-wise pairs.

Note that the reason for using only 0.05% of the model parameters for the similarity calculation is to reduce the computational cost, as the gradient distribution of this randomly selected subset is similar to that of the entire weight set [1]. We have added these details in Sec. A.17 in the revision for further clarification. Thank you!

The paper's adaptive layer freezing strategy treats layers somewhat independently when making freezing decision. However, neural networks typically have complex inter-layer relationships that this approach might not fully capture. Could the adaptive layer freezing strategy be extended to handle cross-layer dependencies, potentially leading to more optimal freezing decisions?

$\to$ This is a terrific point. Yes, we treat the layers independently but we can extend it to account for cross-layer dependencies by incorporating non-diagonal elements of the Fisher Information Matrix (FIM), which involves computing the Hessian (i.e., second-order derivatives). However, given our goal is computationally efficient CL, we intentionally limit our focus to the diagonal components of the FIM (i.e., treating each layer independently). Note that calculating these diagonal components only requires first-order derivatives, which are naturally obtained during gradient descent. Considering layer-combinatorial methods that are computationally efficient is a very promising future research avenue.

We sincerely appreciate your effort in reviewing our submission. We gently remind the reviewer that we tried our best to address your concerns via our replies and manuscript revision. As the discussion period is nearing its end, we would be glad to hear more from you if there are further concerns.

Dear Reviewer VVoh

We sincerely appreciate your effort in reviewing our submission. We gently remind the reviewer that we tried our best to address your concerns via our replies and manuscript revision. As the discussion period is nearing its end (closing in a few days), we would be glad to hear more from you if there are further concerns.

Dear Reviewer VVoh

We sincerely appreciate your effort in reviewing our submission. We gently remind the reviewer that we tried our best to address your concerns via our replies and manuscript revision. As the discussion period is nearing its end (closing in a few days), we would be glad to hear more from you if there are further concerns.

Fisher Information (FI) is calculated based on the current mini-batch. However, the calculation results based on the current mini-batch may not generalize across diverse batches in continual learning. For instance, a layer might appear less informative for one mini-batch but could be essential for others due to shifting data distributions, especially under continual learning settings. Hence, the paper needs to justify this.

$\to$ Although we compute the FI on the current mini-batch, we use the exponential moving average (EMA) of the expectations computed from mini-batches over past iterations to account for the entire dataset (Sec. A.1). Since EMA estimates continuously incorporate FI of new data, it allows us to account for shifting data distributions while still considering previously encountered data (note that current mini-batch includes both newly encountered data and data from episodic memory).

Writing Issues - Line 071: 'ImageNet)' -> 'ImageNet'. Line 087: 'ER' has no citation and explanation for the abbreviation. Fig. 1: No explanation of the y-axis.

$\to$ Thank you for the findings! We have corrected typos, added a citation for ER, provided an explanation of the y-axis in Fig. 1, and additionally ran multiple rounds of revision

The strategy might prioritize rare samples in the dataset but may need to be more informative. This overemphasis can lead to inefficient learning, as these rare samples may not enhance generalization and introduce noise. The paper needs to justify that such a retrieval strategy would maintain the acquired knowledge during training.

$\to$ We guess the 'rare sample' you mentioned as the stored samples in episodic memory and of the classes with a few examples. We argue that such rare samples are particularly valuable during training in CL for the following reasons.

Training a model with an imbalanced dataset without accounting for the imbalance would lead to biased predictions and suffer from slow adaptation to novel classes. The biased prediction refers to the samples from minor classes (e.g., novel classes) are predicted as belonging to major classes (e.g., previously encountered classes) [8, 9]. The slow adaptation leads to poor anytime inference performance [11]. To this end, we prioritized less-trained and forgotten samples to train a model with an imbalanced dataset, inspired by over-sampling (i.e., prioritizing minor classes) [6] and under-sampling (i.e., deprioritizing major classes) [7] strategies from imbalanced setups.

Moreover, our proposed retrieval strategy (i.e., SAR) does not always prioritize rare samples. Specifically, if samples from certain classes are repeatedly used for training while others are not, the use frequency of the unused classes decays continuously through our proposed discounted use frequency. As a result, the priority of unused samples increases, preventing any particular class from being excessively prioritized during training.

[1] Wang et al., SparCL: Sparse Continual Learning on the Edge, NeurIPS 2022

[2] Ghunaim et al., Real-Time Evaluation in Online Continual Learning: A New Hope, CVPR 2023

[3] Csordas et al., SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention, NeurIPS 2024

[4] Chen et al., VanillaNet: the Power of Minimalism in Deep Learning, NeurIPS 2023

[5] Tian et al., Towards Higher Ranks via Adversarial Weight Pruning, NeurIPS 2023

[6] Zhang et al., Learning Fast Sample Re-weighting Without Reward Data, ICCV 2021

[7] Yu et al., Reviving Undersampling for Long-Tailed Learning, arXiv 2024

[8] Caccia et al., New insights on reducing abrupt representation change in online continual learning, ICLR 2021

[9] Seo et al., Learning Equi-angular Representations for Online Continual Learning, CVPR 2024

[10] Koh et al., Online continual learning on class incremental blurry task configuration with anytime inference, ICLR 2022

FLOPs served as a direct measure of computational demand across different models, making it possible to compare CL algorithms fairly. However, it doesn’t account for real-world runtime or memory access patterns, which vary based on hardware and software implementations. For instance, two architectures with similar FLOPs might have different inference times on the same hardware due to parallelization or memory bandwidth usage differences.

$\to$ We agree that FLOPs do not perfectly account for real-world runtime and memory bandwidth. So, as suggested, we report the average wall time over three runs for each experiment and summarize the results in Tab. A below. As shown in the table, performance trends are consistent with those observed in Table 16, where FLOPs were used for comparison, although there is a slight difference due to fluctuations in wall time, which are affected by the factors mentioned above.

While the wall time is a practical measure of real-world performance, it can be affected not only by hardware/software implementation but also by interference from system tasks, making it difficult to measure reproducibly. Out of necessity, we choose to use FLOPs instead.

Table A. Comparison of Wall time on CIFAR-100 Gaussian setup, with 2 epochs of training per sample encounter

The adaptive layer freezing proposed in this paper can reduce FLOPs but adds complexity. Calculating the full Fisher Information Matrix (FIM) is computationally intensive because it involves second-order derivatives, which can be prohibitive, especially for large neural networks. The paper must explain why calculating FIM does not introduce additional computation costs.

$\to$ Great point. To avoid the high computational cost in calculating FIM, we use a first-order approximation by considering only the diagonal components instead of computing the full FIM, following [1] (in Line 214 and Eq. 2). Since diagonal components are first-order derivatives, which are naturally obtained in gradient descent, aL-SAR’s additional computation costs are negligible compared to the savings achieved by freezing (only 0.00016% increase in computational cost, as shown in Table 16).

We sincerely appreciate your effort in reviewing our submission. We gently remind the reviewer that we tried our best to address your concerns via our replies and manuscript revision. As the discussion period is nearing its end, we would be glad to hear more from you if there are further concerns.

Thank you for the further clarification. I will raise my score to 8.

In my opinion, the introduction lacks proper background knowledge, and it jumps to the solutions for the existing problems in online CL. I suggest adding at least one paragraph explaining the background and current state of online continual learning and the current high-level challenges that prior work has attempted to solve

$\to$ Thank you for the suggestion! We add a paragraph explaining the background and the current state of online CL at the beginning of the revision as suggested. We highlight why research on online CL has recently garnered interest, discusses various recently proposed methods, and outlines the goals and challenges associated with online CL.

Have you considered reporting forgetting as an additional metric? This is commonly used in CL papers and could provide further insight into the efficacy of your method.

$\to$ Great suggestion! In Sec. A.24 of the revision, we add a comparison of forgetting between our proposed aL-SAR and baseline methods as suggested. For the disjoint setup, which has clear task boundaries, we report traditional forgetting [1]. In contrast, for the Gaussian-scheduled setup—where explicit task boundaries are absent due to a continuous shift in the data distribution modeled by a Gaussian distribution—we report the Knowledge Loss Ratio (KLR) and Knowledge Gain Ratio (KGR) [2], as they do not require explicit task boundaries.

What is the memory selection mechanism? In other words, how memory samples are selected, and how does the memory get updated?

$\to$ We use Greedy Balanced Sampling [3] for memory selection strategy (L831).

Besides the similarity-aware retrieval, is there any other difference between the in-memory and new samples in the training?

$\to$ No, there is no difference. Since we compose mini-batch by retrieving samples only from memory (i.e., memory only training), we do not differentiate between in-memory samples and new samples. However, since new samples may have been used less frequently, they might have a higher probability of being selected.

[1] Chaudhry et al., Riemannian walk for incremental learning: Understanding forgetting and intransigence, ECCV 2018

[2] Koh et al., Online Boundary-Free Continual Learning by Scheduled Data Prior, ICLR 2023

[3] Prabhu et al., GDumb: A Simple Approach that Questions Our Progress in Continual Learning, ECCV 2020