Dear Reviewer PpH5, thank you so much for the review and the constructive suggestions! Here is our response to your questions and concerns.

Clarification around the exact setup and fair comparison with other baselines.

1. Our proposed method addresses the problem of forgetting in Class Incremental Learning (CIL) by incorporating unsupervised learning during the testing phase, a unique approach distinct from Test-Time Adaptation (TTA). In a production setting, where models must adapt as new classes are introduced, test data from previously learned tasks is often abundant but underutilized, serving only for evaluation. We leverage this test data with our method, DoSAPP, to mitigate forgetting through unsupervised learning. While DoSAPP can adapt to test distribution conditions, its primary goal is to reduce forgetting without requiring extensive data storage or replay during supervised training. Unlike TTA, which focuses on addressing input distribution shifts (e.g., data corruption), our method assumes the test data distribution aligns with the training data of learned tasks, enabling effective continual learning.

2. In conventional TTA methods, both adaptation and evaluation are performed on the same test-time data. In contrast, our approach partitions the test data into two distinct subsets: one dedicated to unsupervised learning(Du) and the other reserved for evaluation($D^e$). This partitioning is merely to assess the generalization of the model ensuring a fair assessment of the model’s performance.

3. The comparison between our method and other methods is fair since other methods are also trying to improve CIL performance. We agree that other methods do not utilize test data, however, this is part of the novelty of our approach. In Table 5 in the revised rebuttal submission, we present results combining SPU and SparsCL with vanilla pseudo labeling, along with integrating the most recent TTA method, RMT[a]. We demonstrate that combining typical TTA with CIL gives inferior performance. We further emphasize that typical TTA methods such as test time normalization do not apply directly here as they are designed for input distribution shifts such as corruptions which do not impact the label distributions[b].

[a] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation by Mario et al(CVPR 2023)

[b] Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation by Vianna et al(CoLLAs, 2024)

Lack of baselines

1. With regards to using batch normalization-based methods (test time normalization), we want to also highlight that we use a CLIP ViT model which does not incorporate batch normalization, thus those approaches are not directly applicable. We added more experimental results in Table 5 by integrating RMT[a] the most recent TTA state-of-the-art method, with SPU and SparseCL. It can be observed that our proposed method DoSAPP outperforms all of them. This highlights that TTA methods when fused with a continuous supervised training pipeline cause the model to significantly lose knowledge. As there are long sequences of distinct tasks, it becomes difficult for any TTA method to adapt to these ever-changing source distributions. On the other hand, our method mitigates this issue by the unique design of EMA with dual momentum. We further observe that the TTA method gives inferior performance for almost all datasets in comparison to self-labeling, proving that typical TTA is not suitable for deployment under continuous supervised learning, and expanding tasks. | Method | Cifar100 | Aircraft | Cars | GTSRB | Cub | |---------------------------|----------------|-----------------|--------------------|------------------|--------------| | SPU + Pseudo Labeling| 74.09, 10.43| 27.72, 24.86 | 68.91, 7.34 | 60.17, 6.94 | 61.21, 4.01 | | RMT + SPU | 63.06, 23.28 | 29.33, 15.10 | 62.32, 21.95 | 54.13, 17.56 | 63.87, 6.34 | | RMT + SparseCL | 70.82, 12.25| 27.11, 16.29 | 69.81, 17.22 | 51.98, 11.40 | 60.03, 10.58 | | DoSAPP (from paper) | 79.16, 7.73 | 39.14, 12.55 | 74.87, -0.74 | 72.33, 1.02 | 68.17, 2.15 |

Table-A Accuracy and Forgetting for CIL methods integrated with RMT compared with DoSAPP over different datasets.

[a] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation by Döbler et al(CVPR 2023)

Addressing Questions:

Q-1: Is test time adaptation performed image-wise or batch-wise ?

A-1: Batch wise on $D^u$

Q2: If it is performed batch wise, is it performed using the shuffled test set comprising all tasks at the same time, or is it performed task-by-task ?

A-2 Yes, its batchwise on a shuffled test set $D^u$ comprising of all the previous tasks seen so far, and evaluation is done on a separate partition($D^e$). This can be referred in the Fig 1 and the algorithm block.

Dear Reviewer uBk1, thank you so much for the review and the constructive suggestions! Here is our response to your questions and concerns.

Addressing Weaknesses

Combining TTA methods with other baselines.

We added more experimental results in Table 5 by integrating RMT[a] the most recent TTA state-of-the-art method, with SPU and SparseCL. It can be observed that our proposed method DoSAPP outperforms all of them. This highlights that TTA methods when fused with continuous supervised training pipeline cause the model to significantly lose knowledge. As there are long sequences of distinct tasks, it becomes difficult for any TTA method to adapt to these ever-changing source distributions. On the other hand, our method mitigates this issue by the unique design of EMA with dual momentum. We further observe that the TTA method gives inferior performance for almost all datasets in comparison to self-labeling, proving that typical TTA is not suitable for deployment under continuous supervised learning, and expanding tasks.

Table-A Accuracy and Forgetting for CIL methods integrated with RMT compared with DoSAPP over different datasets.

[a] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation by Döbler et al(CVPR 2023)

Addressing the formatting issues/typos

We have corrected the bold font usage in the table for both metrics.

Clarifying the dual momentum concept

1. In our proposed method, we update only the first MLP layer's parameters (as detailed in lines 258-262 of the revised rebuttal submission). All other parameters remain frozen throughout the training process for both the student and teacher models, ensuring they are not modified during any phase of training.

2. During the supervised phase of the current task, we first estimate the parameter's score”(given by equation 2 in the paper) based on the gradient norm for all candidate parameters in the first MLP layer of each Transformer block. Using this score, we generate a binary mask [Zhang et al., 2023b], which selects the top 10% of those candidate parameters. These selected parameters are the only ones updated during the student model's training.

3. The goal of EMA is to move the teacher smoothly and steadily towards the student model, we observed that applying the standard EMA to all the first MLP layer parameters similarly (both M=0 and M=1) or only to selected parameters (M=1) results in instability in the teacher due to parameters moving at different paces than those of the student. This is where our dual momentum mitigates the logical inconsistency of updates, and provides a smoother and robust average, as given in lines 286-289, and in the section entitled “weighted exponential smoothing with dual momentum”. We further provide an ablation to demonstrate the instability of using single momentum in the table below. It can be clearly observed that any option of single momentum creates a dissonance in training, and dual momentum as proposed in our method, gives stable and better results.

Table B: Average accuracy for different datasets comparing single momentum with our proposed method DoSAPP. Here, m refers to the binary mask computed through gradient norms of candidate parameters.

4. Keeping other parameters frozen(eg those of the transformer) does not pose any danger to the model learning since they represent generalized knowledge in a foundational model like CLIP.

Addressing Presentation Issues

We have addressed all the mentioned issues in the revised rebuttal submission.

Thank you so much for acknowledging our efforts.! Here is our response to your recent questions and concerns.

Combining TTA Methods with Other Baselines

We want to emphasize that although test data is used in TTA, CoTTA, and our method, we address a fundamentally different problem: Can test data be utilized to mitigate forgetting in class-incremental continual learning? Unlike most of the TTA literature ([a,b]), which focuses on handling input distribution shifts (e.g., image corruption), our approach targets scenarios without such shifts. Additionally, many TTA methods relying on batch-level information struggle when label shifts occur ([d, e]) and are not designed to improve results in the absence of input distribution shifts.

In our work, the unsupervised test-time data does not need to be corrupted, as we address a fundamentally different problem. In our setting, the test-time data has a different label distribution compared to the most recent supervised data, often including classes absent from the latest supervised data. This results in a label shift, which is a key distinction that renders many TTA methods inapplicable. Since most TTA methods are designed to handle input distribution shifts and often struggle with label shifts, it is unsurprising that they fail in our scenario, where only label shifts occur without any input distribution shifts.

We also note that our choice of RMT is based on finding a method that does not explicitly target corruptions (unlike, for example, batchnorm adaptation). Although these methods do improve performance in some cases, they underperform as they are simply not designed for this setting.

Pseudo-labeling may seem promising and serves as our first baseline, but it is prone to errors from the main model, which may have forgotten earlier tasks. Solely relying on model predictions amplifies bias toward recent tasks, increasing label confusion. This is where the EMA framework with a teacher-student model becomes crucial. Our simple yet elegant selection of the model that is an expert on a given test data point provides the pseudo-label. Parameter selection and masking are designed to handle large pretrained models efficiently, enabling online updates while minimizing forgetting (as per SPU). Since test-time data may originate from any previous distribution, we utilize the union of masks. Additionally, masking ensures parameters are updated at varying rates. Initial experiments revealed that plain EMA momentum updates led to optimization instability, which we addressed through the introduction of dual momentum for improved stability.

Regarding the interplay between the different components and which technical contributions drive the effectiveness, in Section 5 and Table 3, we fully ablate all components of our method. We are happy to address any specific items that are lacking in the ablations.

We will update our paper with these clarifications.

[a] Continual Test-Time Domain Adaptation by Wang et al(CVPR 2022)

[b] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation by Dobler et al.(CVPR 2023)

[d] Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation by Vianna et al (CoLLAs 2023)

[e] DELTA: DEGRADATION-FREE FULLY TEST-TIME ADAPTATION by Zhao et al. (ICLR 2023)

Clarifying the Dual Momentum Concept

1. As pointed out in [a,b], parameters from the first MLP layer of each transformer block, referred to as candidate parameters, are considered for the update. All other model parameters from other modules for both the student and teacher models remain static and are excluded from discussion.

2. During the supervised training phase of the first task, assume there are 1000 candidate parameters - all from the first MLP block of each transformer layer. Gradient norms are calculated for these parameters, and with sparsity set to 10%, 100 learnable parameters are selected and masked as ($\bf{m}=1$) (set $A$), while the rest are masked as ($\bf{m}$=0). Only these 100 parameters undergo gradient updates in the student model.

3. For EMA, after the first task, under single momentum, all 1000 candidate parameters of the teacher model receive updates. However, only parameters marked (m=1) are effectively updated for the first task. Unchanged parameters (m=0) in the student model remain unchanged in the teacher model as well, only for the first task.

4. Similarly, after the second task, gradient norm calculation is repeated, resulting in a potentially different set of selected parameters (set $B$; $B\neq A$), with possible overlaps. With standard EMA there are two options:

• Updating All Candidate Parameters: Updating all 1,000 candidate parameters in the teacher model fails to account for the varying importance of selected parameters, which should be updated at a faster rate compared to other candidate parameters. Neglecting this distinction results in uniformly slow updates, causing a significant divergence between the teacher and student models. This issue is reflected in the poor performance metrics shown in Table B of the rebuttal experiments (Row 1).

• Updating Only Selected Parameters($m=1$): This approach introduces recency bias. For example, consider a parameter $x_a$ that was selected for task 1 (A) but not for task 2 (B), i.e., $x_a \in A \text{ but } x_a \notin B$ ($x^t_a$ denotes the teacher parameter while $x^s_a$ denotes the corresponding student parameter). When the student is updated on task 1, $x^t_a$ is moving towards $x^s_a$. However, when the supervised session of task 1 ends, $x^t_a$ faces a sudden halt, while other parameters start to move towards the student model e.g., $x^t_b$ starts to move towards the student. These abrupt shifts create an imbalance as $x^t_a$ is suddenly frozen while other parameters continue or start to move. This introduces a dissonance among parameters, thus destabilizing training. To remedy this, our dual momentum ensures that all the candidate teacher model parameters (first layer MLP of each transformer) move towards the student model as updates are performed on the student. However, selected parameters for a given task (e.g., $B$) are updated faster than the rest, ensuring their greater significance is reflected during task $B$.

To recap.

1. Non-selected candidate parameters also receive ema updates but at a slower rate (smaller momentum).
2. Selected parameters ($\bf{m}=1$) are updated at a slightly faster rate.

This ensures gradual and balanced updates ensuring all parameters are moving towards the student in a stable robust manner, maintaining stability and mitigating the drift caused by masking. Our experiments confirm that dual momentum stabilizes training, unlike single momentum or selective updates, leading to teacher model instability.

[a] Overcoming Generic Knowledge Loss with Selective Parameter Update by Zhang et al. CVPR(2024)

[b] Transformer feed-forward layers are key-value memories by Geva et al(EMNLP 2022)

Presentation:

We have made the corrections as suggested. We would like to confirm that we have thoroughly proofread and addressed issues with English, punctuation, and missing spaces. This can be reflected in the most recent revised rebuttal submission.

Dear reviewer 4MGu, thank you for the review and constructive questions/suggestions! Here is our response to your questions and concerns.

Addressing Major Weaknesses

1. Ablation study about the size of test-time data $D^u$

We conducted an ablation with the size of $D^u$, for the CIFAR100 dataset and added it to the Appendix(A.4) in the revised rebuttal submission. In our method, we divided the evaluation data into two halves. One half is for unsupervised learning ($D^u$), and the other half is for evaluation ($D^e$). In the table below, we feed the fraction of $D^u$ for test time learning. $0.25$ means that $25\%$ of the original $D^u$ is fed to the model for unsupervised learning. We notice that when the fraction is below $0.75$, there is an appreciable difference between the performance of our proposed model. However, at $0.75$, the performance is quite close to that of the whole $D^u$.

2. More information about the experiments

1. We have added the error bars in Table 2 for our method DoSAPP. We have not included the error bars for the baselines, for better readability.

2. We conducted a time analysis for our method as requested by the reviewer on a single batch of supervised training phase(on $D^t$) and unsupervised training phase(on $D^u$) for CIFAR100 dataset with a batch size of 64.

   • Training Time per batch: 0.178 ms

   • Unsupervised Training Time per batch: 0.173 ms

   • Inference Time per batch: 0.012 ms

3. Addressing the formatting issues

We thank the reviewer for pointing out the typos. We have fixed them all, and further revised for formatting issues in the revised rebuttal submission. Please let us know if there are any additional concerns.

Addressing Minor Weaknesses

1. Extending our setup to conventional classification backbones, like ImageNet-21k pretrained encoders and FC classifiers

In this work, we focused on foundational models in particular CLIP since it has shown more generalisability than other pretrained models. This can be attributed to the absence of classification heads which are one of the causes of severe catastrophic forgetting in other pretrained architectures with FC classifiers. We follow the experimental standards as given in SPU[a], FinetuningCLIP[b]. Moreover, given the short span of rebuttal, extending this work to pre-trained ImageNet-21k encoders and FC classifiers is currently not possible, and can be considered as future work.

[a] Overcoming Generic Knowledge Loss with Selective Parameter Update by Zhang et al(CVPR 2024)

[b] Finetune like you pretrain: Improved finetuning of zero-shot vision models by Goyal et al(CVPR 2023)

2. Fixing punctuation of equations

We have fixed those issues in the revised rebuttal submission.

Dear Reviewer 2tLK, thank you so much for the review and the constructive suggestions! Here is our response to your questions and concerns.

Fairness of comparison

1. Yes the comparison is fair since we repeated the experiments for all the given baselines in Table-2 with CLIP ViT-B/16 backbone. We have added this clarification in the Table-2’s caption. (line 391 in the revised submission)

2. Since our proposed method is a novel contribution of improving class incremental continual learning(CIL), we compared our method with the most recent and diverse CIL algorithms. We consider the diversity of approaches to the same problem a strength of the benchmarking.

Added the missing ablation of single momentum + union of mask.

We thank the reviewer for pointing this out. We have now added the ablation(3rd row) in Table 5 of the revised rebuttal submission. The new results as compared to DoSAPP are also shown below. For each dataset, average accuracy and forgetting(separated by comma) is shown below,

Explanation of the use of Self Labelling

In Table 2, for self-labeling, a finetuned model is used as correctly observed by the reviewer. To have a more comprehensive comparison, we compare our method with naive integration of most recent CL method(SPU) with pseudo labeling in Table 5. We also want to emphasize that although most methods from the TTA literature are not directly applicable to our scenario, we have now added a SOTA TTA approach RMT[a] to SPU and SparsCL in the revised rebuttal submission(Table 5) and observed that the TTA method actually degrades the performance, as shown in the table below. This highlights that TTA methods which are often designed for input distribution shifts that don't affect the labels do not apply well for our setting where the unsupervised data is coming from previous task data distributions.

Table-A Accuracy and Forgetting for CIL methods integrated with RMT compared with DoSAPP over different datasets.

[a] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation by Döbler et al(CVPR 2023)

Corrected the typos:

We have corrected all the typos, bold font usage in tables, and proofread all the text once more in the revised rebutall submission. Thank you so much for pointing them out.