Adaptive Retention & Correction: Test-Time Training for Continual Learning

We thank reviewer hK7E for the thoughtful and constructive comments. It is evident that the reviewer spent considerable time carefully reading our work and provided valuable insights and suggestions. We really appreciate the detailed feedback and the recognition of the strengths in our approach. Below, we address the specific points raised by the reviewer.

There is a lack of detail on the overall training procedure. How many epochs are used for training?

We apologize for any confusion. In fact, ARC is not involved in the training process. It is only applied during inference, and thus, details such as the number of training epochs are not applicable. During inference, the classifier is adaptively rebalanced for incoming test samples. This process involves only one gradient update per sample, performed in an online manner. In other words, we only use each sample once for classifier tuning.

Is this correction and retention procedure applied after each task? In that case, this would correspond to storing unlabeled memory data.

Yes, this correction and retention procedure is applied after each task. However, we do not store any unlabeled memory data. Instead, our approach relies solely on on-the-fly test samples during inference.

Confusion on the formula of Definition 1

Thank you for pointing out the confusion on $i$ in the max operator. We apologize for the typo in the original definition. The corrected definition is as follows:

$S_i = \max_{s \cdot (i-1) \leq k < s \cdot i} \frac{ \exp{z_k / T^{(t - i)}}}{\sum_{j=1}^{s \cdot i} \exp{z_j / T^{(t - i)}}}$

To clarify this definition, we can compare it to the standard softmax calculation. In the standard softmax, the numerator involves the logits for all classes. However, in our case, we focus specifically on the highest logit for task $i$, which is why the max operator is used in the numerator. Similarly, for the denominator, instead of summing over logits for all classes, we restrict the sum to only the classes revealed to the model up to task $i$, which is also the reason why we have a temperature value $T > 1$ ("accommodate the imbalanced denominator for different tasks" as stated in L344). In this way we can minimize classification bias both between past and current tasks and among past tasks themselves.

This method is connected to Test-Time Adaptation and some discussion in the related work would be appreciated. While TTA methods are rightfully compared in section 4.2, such discussion could be included more thoroughly in the related work.

Thank you for the suggestion! While we briefly discussed TTA methods in Section 2.3, we agree that this section can be expanded to provide a more thorough discussion. In the revised version, we’ll make sure to include a more detailed discussion to better connect our work with TTA methods.

The impact of the temperature is not presented in the paper

Thank you for pointing this out. In the table below we present ablation results on the impact of the temperature value. As shown, incorporating the temperature parameter improves performance consistently:

The notation for the temperature and the number of tasks is the same which is confusing

Thank you for pointing this out! These are supposed to be different notations. We will revise accordingly.

In Figure 1, is the independent classifier trained for doing classification of one task only? Then is the problem a TIL problem? If this is the case, the performances are comparing a TIL problem to a CIL problem, which is unfair since the CIL problem is much harder than the TIL problem.

Yes, you are correct that the independent classifier in Figure 1 is trained under the TIL setting, while the shared classifier is trained under the CIL problem. In fact, the purpose of Figure 1 is to demonstrate why the CIL problem is significantly harder than the TIL problem, i.e., primarily due to classifier bias. This highlights how the way classifiers are constructed can have a much greater impact on overall continual learning performance compared to the backbone architecture, which is a major motivation of our approach. We included this figure to bring attention to a key point that might not be immediately obvious to readers less familiar with continual learning.

Dear reviewer, we would be grateful if you could confirm whether our response has addressed your concerns. Please do not hesitate to let us know whether there is anything else you would like to see clarified or improved before the end of the rebuttal period.

Thank you for your valuable feedback! Regarding the most critical point, we believe there might be a misunderstanding of the continual learning (CL) setting. Specifically, in real-world CL use cases:

1. A model is initially trained on a dataset and deployed in practice.
2. Over time, new requirements emerge or new data becomes available, necessitating updates to the model.
3. The updated model is redeployed and continues to be used in practice.
4. This process repeats as additional data or requirements arise.

ARC specifically operates during Step 3 and does not require storing any past-task data. To illustrate, consider a scenario where we initially train a model to distinguish between cats and dogs. Later, we may want to add a new class, such as birds, to the model (as our approach focuses on class-incremental learning). After training the model on bird data and redeploying it, the model’s practical application is to classify among cats, dogs, and birds. Naturally, during deployment, the model will encounter samples from previous tasks (cats and dogs), allowing ARC to be utilized without storing any past-task data.

Subsequently, we might add another class, such as turtles. After updating the model to include this new class, it is redeployed to classify among four types of animals. Once again, ARC can be applied in a similar manner during deployment, without requiring any storage of past-task data. This cycle reflects typical deployment scenarios, and we believe this is how ARC can be applied after every task in practice.

Furthermore, there is also a metric in CL that refers to the averaged average accuracy: $\bar{A} = \frac{1}{N}A_B$ , where $A_B$ is defined in equation 4 of our paper. This metric explicitly measures cumulative performance by evaluating the model after each task, further validating our design choice of applying our method after each sequential training phase.

Therefore, regarding your concern that our approach implicitly stores past-task memory data, we respectfully disagree. Our method does not store any data from past tasks. Instead, as part of the natural process of being deployed in practice, the model may encounter past-task samples, enabling our method to function without violating the “memory-free” claim.

Nonetheless, we greatly appreciate your suggestion and have conducted additional experiments under the proposed setup, where our method is applied only after the final training task. The results are presented in the table below:

As shown in the results, ARC still provides significant improvements even when applied only after the final training task. We stronly believe that these findings support the effectiveness of our approach.

Details regarding the training procedure

We appreciate your suggestion and will include these details in the appendix to provide readers with a clearer understanding of our training procedure.

In summary, for non-prompt-based methods, we use the SGD optimizer, and for prompt-based methods, we employ the Adam optimizer. During testing, given a batch of test samples, we calculate the loss using Equation 3 from the paper. Based on this loss, we perform a single gradient update. Theoretically, this process is equivalent to training on the test data for just 1 epoch, where each test sample is used exactly once.

We want to emphasize that this design of performing only one gradient update aligns with our goal of simulating realistic scenarios. In standard testing, a model processes a batch of test samples with a single forward pass to produce predictions. In ARC, we adhere to this principle by limiting the process to a single forward pass followed by one gradient update, maintaining the efficiency and practicality of the testing procedure.

We sincerely thank you once again for your thoughtful feedback. If there is anything else that requires clarification, please let us know before the rebuttal period concludes.

Does the ARC algorithm run after training each task? If so, what is the additional time required for training and inference?

Thank you for your question. The ARC algorithm is applied after training on each task. However, we emphasize again that ARC does not interfere with the training process itself, so the additional training time required by ARC is zero.

During inference, due to time limitation, we provide the extra time required for ARC on 4 representative methods (same ones we used for hyperparameter ablation)

As shown, there is a certain amount of additional inference time required for ARC. This is expected, as ARC is specifically designed to address classification bias during inference, which naturally incurs some additional computational cost.

Furthermore, it is worth noting that research in other areas, such as large language models (LLMs), are increasingly exploring methods that enhance performance by utilizing additional inference-time computation. There is a growing consensus that additional inference time is a reasonable trade-off when it leads to significant improvements in model performance. Similarly, we believe the computational cost of our method is well-justified by the substantial performance gains it delivers.

Line 9 in the pseudocode

Thank you for pointing out this typo! We will correct it in the revised version.

The scaling temperature used in $S_i$ and the total number of tasks are both denoted as $T$. Are these referring to the same setting

Thank you again for pointing this out! These are supposed to be different notations. We will revise accordingly.

The hyperparameters used in the reported results are not fully detailed. Were the same hyperparameters applied across different methods and datasets? It seems that $\beta$ will influence the number of test samples used for training, and there is a trade-off between the number of samples and the accuracy of the pseudo-labels. How were the hyperparameters selected?

We conducted experiments (shown in Figure 5) to demonstrate how the choice of hyperparemeters influence our method, and results show that:

• our method is robust to the choice of hyperparameters, a point also acknowledged by Reviewer 1cF7 and hK7E.
• using the same hyperparameters across all datasets and methods would still yield performance gains

However, we choose not to use the same hyperparameter following previous methods. To ensure a fair evaluation, we used the best-performing values of $\gamma$ and $\beta$ for each method and dataset.

In the discussion following Assumption 2, the authors state that a low ratio of $c$ and $\hat{c}$ indicates a sample is more likely misclassified as belonging to a past task. However, Figure 5 suggests that a larger corresponds to better performance, which seems contradictory. Could the authors clarify this point?

Thank you for your comment. We do not believe the results in Figure 5 are contradictory. The results in Figure 5 demonstrate how low the threshold should be for optimal performance. It would only be contradictory if ARC performed better when the threshold was larger than 1, which is not the case.

To further illustrate this, we have conducted additional experiments to evaluate the performance of ARC when $\gamma = 1.1$. The results are as follows:

We thank reviewer EsC3 for the thoughtful and constructive comments. It is evident that the reviewer spent considerable time carefully reading our work and provided valuable insights and suggestions. We really appreciate the detailed feedback and the recognition of the strengths in our approach. Below, we address the specific points raised by the reviewer.

While the motivation to correct misclassified past task samples is sound, the proposed Task-based Softmax Score (TSS) may be problematic. For the earlier tasks, as the temperature increases, the probability distribution flattens, which could hinder the model’s ability to recognize samples from past tasks.

Thank you for pointing this out. To clarify, the denominator of TSS does not sum over logits for all classes (as in regular softmax) but is instead restricted to the classes revealed to the model up to task $i$. For earlier tasks, this results in fewer terms in the denominator, which sharpens the probability distribution rather than flattening it. To counterbalance this sharpening effect, we employ a temperature value $T > 1$ ("accommodate the imbalanced denominator for different tasks" as stated in L344).

To further illustrate the impact of the temperature value, we provide the following ablation study, which demonstrates its effect on average accuracy across different methods. As shown, incorporating the temperature parameter improves performance consistently:

Although the experiments demonstrate the effectiveness of ARC, the paper lacks comparisons with other test-time adaptation (TTA) benchmarks, such as CoTTA.

We would like to point out that we have compared ARC with other TTA benchmarks, as shown in Table 3, where TENT is included as a representative TTA method. This has also been acknowledged by Reviewer hK7E.

Regarding CoTTA, we tried to include it in our experiments. However, our findings reveal that CoTTA is not compatible with prompt-based methods. Directly applying CoTTA to these methods leads to a decrease in performance, as shown below:

For non-prompt-based methods, while CoTTA does provide some improvements, the gains are less significant than those achieved by ARC. This is evident in the following comparison:

Therefore, we did not report the comparison with CoTTA.

Additionally, the introduction of extra data for classifier training (referred to as retention) raises concerns about increased training costs that should be addressed.

We would like to clarify that ARC does not introduce any additional training costs. The retention mechanism is applied solely during inference, where the classifier is adaptively rebalanced for incoming test samples. This process involves only one gradient update per sample, performed in an online manner. As a result, the computational overhead introduced by ARC is confined to the inference stage (details in the answer below), without impacting the training process.

The description of the experimental setup in Figure 1 is inadequate. For instance, is the class-incremental learning setup applied in this toy example? How are the two types of classifiers trained and tested? Which task’s accuracy is measured in the figure?

We apologize for the confusion. To clarify:

• The independent classifier in Figure 1 is trained and tested under the TIL setting, while the shared classifier is trained and tested under the CIL setting.
• $A_i$ in the figure refers to the average accuracy of task $i$, as defined in Equation 4 of the paper.

In fact, the purpose of Figure 1 is to demonstrate why the CIL problem is significantly harder than the TIL problem, i.e., primarily due to classifier bias. This highlights how the way classifiers are constructed can have a much greater impact on overall continual learning performance compared to the backbone architecture, which is a major motivation of our approach. We included this figure to bring attention to a key point that might not be immediately obvious to readers less familiar with continual learning.

Dear reviewer, we would be grateful if you could confirm whether our response has addressed your concerns. Please do not hesitate to let us know whether there is anything else you would like to see clarified or improved before the end of the rebuttal period.

We thank reviewer ALn4 for the thoughtful and constructive comments. It is evident that the reviewer spent considerable time carefully reading our work and provided valuable insights and suggestions. We really appreciate the detailed feedback and the recognition of the strengths in our approach. Below, we address the specific points raised by the reviewer.

Need to report the training time during inference, since it might add overhead when deployed to the real-world.

Thanks for the advice! Due to time limitation, we provide the extra time required for ARC on 4 representative methods (same ones we used for hyperparameter ablation):

As shown, there is a certain amount of additional inference time required for ARC. This is expected, as ARC is specifically designed to address classification bias during inference, which naturally incurs some additional computational cost.

Furthermore, it is worth noting that research in other areas, such as large language models (LLMs), are increasingly exploring methods that improve performance by utilizing additional inference-time computation. There is a growing consensus that additional inference time is a reasonable trade-off when it leads to significant improvements in model performance. Similarly, we believe the computational cost of our method is well-justified by the substantial performance gains it delivers.

What if the incoming sample from the past task is not allow to use during inference? For example, it is deployed to the embbedding device that need real-time inference, will additional training add much overhead? Or is it possible to not update at every step? It will be interesting to provide such results.

Thank you for raising this question. We would like to emphasize again that our method does not store any test samples in memory. Instead, it operates using on-the-fly test samples and performs a single gradient update as each sample is processed by the model. So whether or not past task sample is allowed to use is irrelevant to our method.

Regarding the computational overhead, as noted in our response above, the additional time required for inference varies across methods. In the worst-case scenario with L2P, our method results in a frame-per-second (FPS) drop from approximately 60 to 43 on an NVIDIA 4090 GPU using the ViT-B/16 backbone, which still meets the criteria for real-time inference.

Reducing the frequency of updates is indeed a promising approach to further minimize the additional inference time. For instance, this could involve quantifying how "biased" the classifier is based solely on a batch of test samples before deciding whether an update is necessary. However, exploring this idea in detail falls outside the scope of this rebuttal. We appreciate your suggestion and consider it an interesting direction for future work.

Can this approach extend to other computer vision task such as detection and segmentation? It would be great if the methods can also fix the problem in such scenario by re-balancing the classifcation head in object detection.

Yes, we believe that the idea of ARC can be applied to any computer vision task where classifier bias is a challenge. In fact, there are already works exploring similar directions, such as [1], which addresses bias in continual semantic segmentation. However, applying ARC to tasks like detection or segmentation would require certain modifications to the implementation details to account for the unique characteristics of these tasks. For example, in object detection, ARC would need to re-balance not only the classification head but also adapt to region proposals or bounding box predictions to mitigate potential biases.

[1] RBC: Rectifying the Biased Context in Continual Semantic Segmentation

At Table 1, it seems strange that for CodaPrompt, the position of the difference for Split CIFAR-100 Inc5 is oppositve. The difference sign should be 5.8 (+0.1) instead of 5.7 (-0.1)

Thank you for pointing this out! We agree that your suggested revision would indeed improve the clarity of the table. We will revise accordingly.

Dear reviewer, we would be grateful if you could confirm whether our response has addressed your concerns. Please do not hesitate to let us know whether there is anything else you would like to see clarified or improved before the end of the rebuttal period.

We thank reviewer 1cF7 for the thoughtful and constructive comments. It is evident that the reviewer spent considerable time carefully reading our work and provided valuable insights and suggestions. We really appreciate the detailed feedback and the recognition of the strengths in our approach. Below, we address the specific points raised by the reviewer.

Quantify the computational cost of the self-retention procedure

Thanks for the advice! Due to time limitation, we provide the extra time required for ARC on 4 representative methods (same ones we used for hyperparameter ablation) in the table below:

As shown, there is a certain amount of additional inference time required for ARC. This is expected, as ARC is specifically designed to address classification bias during inference, which naturally incurs some additional computational cost.

Furthermore, it is worth noting that research in other areas, such as large language models (LLMs), are increasingly exploring methods that improve performance by utilizing additional inference-time computation. There is a growing consensus that additional inference time is a reasonable trade-off when it leads to significant improvements in model performance. Similarly, we believe the computational cost of our method is well-justified by the substantial performance gains it delivers.

Did you run the hyperparameter search for the adaptive self-retention procedure on the same dataset that was used for evaluation? Better to ensure hyperparameter tuning is conducted on a separate dataset from the evaluation dataset.

Yes, the hyperparameter search is ran on the same dataset, following previous methods. Furthermore, our experiments in Fig 5 shows that our method is robust to the choice of hyperparameters, and that even using the same hyperparameter across all datasets and methods would still yield performance gains. Therefore, to ensure a fair comparison, we used the best-performing values of $\gamma$ and $\beta$ for each method and dataset.

Have you tried applying your method to other continual learning algorithms, like BEEF, EASE, MEMO, and FOSTER?

Thank you for your question. We actually have applied ARC to MEMO, and the results are reported in Table 1 of the paper. Additionally, ARC has been applied to a total of eight methods, covering a diverse range of approaches, including prompt-based and non-prompt-based methods as well as rehearsal-based and non-rehearsal-based methods. We believe this already demonstrates the versatility and generalizability of our method.

Due to time constraints, it may be challenging to provide results for additional methods, such as BEEF, EASE, and FOSTER, within the scope of this submission. However, we consider this an interesting direction for future exploration.

In line 88, you formulate the two observations as empirical findings. However, in Section 3.3, they are called assumptions. Do you have any empirical results that show to what extent these actually hold?

Yes, the empirical validation of assumptions 1 and 2 is included in Table 4 and 5 of the paper.

Dear AC,

We thank all reviewers and AC for the insightful review and the constructive feedback provided. We greatly appreciate the positive recognition of our work, particularly the effectiveness of leveraging test samples to address catastrophic forgetting for memory-free continual learning. Based on the feedback, we are considering refining the title of our paper to better reflect its contribution, specifically 'Adaptive Retention & Correction: Test-Time Training for Continual Learning.

However, we are not sure how we can update the title on the OpenReview page. Could you let us know how to do that?

Best, Authors