Controllable Continual Test-Time Adaptation

Thank you for your valuable comments and kind words to our work. At the same time, we sincerely apologize for any inconvenience caused by some grammatical and notational errors in your reading. The revised article has been re-uploaded. Below we address specific questions.

Q1. Why the symmetric cross-entropy loss is used rather than KL or reverse KL in eq. 10?

A1. SCE loss is based on previous work RMT, where they found that SCE maintains a balance in the gradient for high and low confidence predictions, which benefits the optimization problem.

Q2. What happens if a batch contains samples from multiple domains? I wonder how much performance degradation would result from that scenario.

A2. The experimental setup you described is different from the CTTA (Continual Test-Time Adaptation) that we are studying; it is another setup called WTTA (Test-time Adaptation in Dynamic Wild World), which was first proposed by SAR$^1$. Due to the differences in experimental details, the performance of the two setups cannot be directly compared.

Q3. Figure 3 is quite intuitive. Can you show similar figures for different target domains over time?

A3. Of course, we have provided the results for an additional 4 domains (noise, blur, weather, digital) in the tsne_big.pdf in the Supplementary Material, please take a look.

Q4. The performance of RMT recorded in this paper is lower than the results reported in the original RMT paper, with a noticeable difference. What is the reason for this?

A4. Because the warm-up operation in RMT uses a large amount of source domain data, we believe this violates the setup of CTTA. Therefore, the data reported in our paper is after removing this operation. (If we also included this operation, the results would improve, but clearly, this is unreasonable.)

Q5. It still inherits the weaknesses of previous works that use pseudo labels. Depending on the accuracy of the pseudo labels, there is a possibility that the class shift may be inaccurately estimated.

A5. This is indeed an important issue, which has been mentioned in the summary section of our paper (line 288). At the same time, we have made significant efforts to address it (line 145). We adopted the method from EATA by setting an entropy threshold to filter out samples with low entropy, which indicates higher uncertainty, and used the certain good samples to construct prototypes.

Q6. The idea of using a domain prototype was originally proposed in the UDA task, so it’s not a new concept. However, it has not been actively utilized in TTA because it’s challenging to obtain an accurate domain prototype given the nature of TTA, which receives online mini-batches as input. What is the mini-batch size used in this experiment? Mini-batch size is considered a significant issue in TTA.

A6. In terms of setting the batch size, we follow the previous methods CoTTA and RMT. Specifically, the batch sizes for cifar10-c and cifar100-c are set to 200, while for imagenet-c it is set to 64. At the same time, we also recognize that mini-batch size is considered a significant issue, especially having a considerable impact on the acquisition of class prototypes. Therefore, we conducted tests on different batch sizes for imagenet-c, and the results are as follows:

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.

[1] Niu, Shuaicheng, et al. "Towards stable test-time adaptation in dynamic wild world." arXiv preprint arXiv:2302.12400 (2023).

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1. Could you clarify how "random order" is defined and implemented in the experiments mentioned in section 4.8?

A1. In the CTTA experiment, there are 15 target domains, and everyone tests them in the same order (i.e., the order in Table 123). This may lead to the method overfitting to this order, so we randomly selected 10 different domain variation orders for testing, and the final results are averaged.

Q2. Does the availability of source data impact the performance of your method? Testing with varying amounts of source data could provide insights into its practical utility when labeled source data is limited.

A2. Note that during the test time, we only used the class prototypes extracted from the source domain data and did not use all the source domain data.

Q3. The computation symbol "cov[]" used in formula (1) is unclear. Could you provide a definition or explanation?

A3. In our article, "cov[]" means covariance.

Q4. Can you elaborate on the relationship between formulas (7) and (8)? What is the underlying intuition linking these formulas?

A4. In brief, formula (8) is the objective of our loss, while formula (7) is the implementation of our loss. We hope that the model's output is insensitive to changes in the domain, meaning that the model output should not change when activations are slightly added or removed along the domain bias direction, as stated in formula (8). We can achieve this goal by suppressing the directional derivative, as shown in formula (7).

Q5. Considering practical constraints where not all classes may appear in each batch (e.g., batch sizes less than 1000), how might this affect the calculation of domain shift direction and the applicability of formula (9)?

A5. We will only perform calculations on the categories that exist within the batch.

Q6. Typo in lines 71 and 73. DSCL $\to$ CSCL?

A6. Yes, this is a typo, and we will correct it.

Q7. Marginal Improvements

A7. The improvement in classification accuracy of the report is relatively small (Table 123), which may be related to the single order of the domain. Most of the previous work on CTTA adopted a single order, which may not reasonably assess the performance of various methods. Therefore, we supplemented with new experiments (section 4.8). We randomly selected 10 different domain variation orders for testing, and the final results are averaged. It can be seen that in this case, our improvement is significant (1.6%, 3.1%, and 2.4% on CIFAR-10C, CIFAR100-C, and ImageNet-C, respectively).

Q8. Inconsistency in Compared Methods

A8. ViDA was only included in the ImageNet-C experiments and not in the CIFAR experiments because we directly used the data provided in the ViDA paper. However, the backbone used in the ViDA paper for the cifar10-c and cifar100-c datasets is ViT, while we used ResNet, so there is no comparability. The same applies to other methods.

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.

Thank you for the quick response.

Q1: Still the number of baseline methods in Table 7 is much less than Table 123 if you want to compare C-CoTTA in diverse settings. The absence of direct results in their paper is not a valid reason to exclude it from comparison.

A1: In order to better compare performance, we conducted experiments with the comparison methods on the CIFAR10-C, CIFAR100-C, and ImageNet-C datasets, using 10 different random corruption sequences. The backbone is the same for the same dataset. The table is as follows: (Note that the two methods, ViDA and CMAE, both used ViT for experiments in the original paper on cifar10-c and cifar100-c. Here, for comparison, we use the same ResNet backbone as other methods.)

Q2: how does the performance of the proposed method change when class prototypes are extracted from less source data.

A2: We conducted observations through experiments, taking the CIFAR100-C dataset as an example, where each category has 500 images. Previously, we used all 500 images to extract category prototypes. Now, we are testing the scenarios of using 1 image, 5 images, 10 images, 50 images, 100 images, and 500 images, to observe the impact of the number of source domain data on the method, as shown in the table below: （Note that the results for batch sizes ranging from 1 to 100 are obtained by running the experiment 5 times and taking the average, as the source domain data selected may vary each time.）

Q3: will only computing with existing classes affect the overall performance across all classes.

A3: Since the batch size in the experimental setup is smaller than the total number of categories, we cannot determine how the performance would be when optimizing with all categories.

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.

Thank you for the response. I appreciate the clarification provided, but some of my concerns remain unresolved. Specifically:

1. Unclear Contributions: The contributions of the paper are still not clear. If the plug-and-play feature is a key contribution, it needs to be validated with various existing CTTA methods to demonstrate its generalizability.
2. Fairness of Comparison with Baselines: The proposed method assumes stronger conditions compared to some baselines. A thorough discussion of the specific conditions under which C-CoTTA is compared to other methods would be helpful. [W5]
3. Metric Validity and Relevance: My concern regarding the validity and relevance of the metrics remains unanswered. [W3]

Given these unresolved issues, I will maintain my current rating.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1. How are the hyperpaprameters lambda_1, lambda_2 tuned? Is any validation corruption used?

A1. $\lambda_1$ and $\lambda_2$ are derived from adjusting based on 4 different domains other than the 15 domains used in the experiments. For example, the CIFAR-10-C dataset actually has 19 types of corruption (different corruptions represent different domains), and our parameters were adjusted based on 4 corruptions (Speckle Noise, Gaussian Blur, Spatter, Saturate). In the formal experiments, we selected 15 completely different domains (Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation, Pixelate, JPEG).

Q2. Can you report numbers for the two settings when i. lambda_1 = 0, and ii. lambda_2 = 0 to analyze the contribution of different losses.

A2. This is actually our ablation experiment (Table 4), where CSCL represents the case of $\lambda_1 = 0$ and DSCL represents the case of $\lambda_2 = 0$.

Q3. Is the paper not just an application of CAV for continual TTA?

A3. No, CAV is just a tool we use in our method to represent class shift and domain shift; it plays a supportive role and is not the core of our method. The starting point of our method is to suppress the opposing shifts between categories in the representation space. We discovered through t-SNE feature dimensionality reduction that during the domain adaptation process, opposing shifts occur between categories, leading to blurred classification boundaries, and the situation worsens with continuous transformations of the domain. Therefore, we explicitly control the direction of category shifts to prevent them from occurring in opposition, ensuring a clear classification boundary. (This is what "Controllable" means in our method). In this context, CAV is used to represent the direction of class shift.

Q4. Is it non-trivial to apply CAV for settings such as TTA? Are the DSCL and CSCL losses not applicable to domain adaptation problems in general? If not, why does it make sense for TTA?

A4. First of all, it can be confirmed that the methods in our article, including CAV, DSCL, and CSCL, can potentially be applied to TTA (Test-Time Adaptation) and even more broadly to domain adaptation. However, we did not conduct relevant tests because our initial research focus was on CTTA (Continuous Test-Time Adaptation). Considering that in CTTA, the domain continuously changes, the classification boundary blurriness caused by category shifts is likely to be more severe than in regular TTA. Therefore, our methods may be more suitable for CTTA. Of course, prompted by the reviewers, we also hope to examine the effectiveness of our methods in TTA and domain adaptation. We attempted to apply our methods in the TTA scenario (i.e., recovering the model when the domain changes), as shown in the table below, and it can be seen that our methods still perform very well.

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.

Thank you for the quick response. This time I understand your core concerns.

1. The DSCL loss represented by Equation 7 indeed suppresses domain shifts, reducing the model's sensitivity to domain shifts. However, our focus on "Controllable" mainly revolves around the CSCL loss, which explicitly controls the direction of category shifts, preventing the blurring of classification boundaries caused by opposing category shifts. This loss is the biggest difference from previous work and can be considered the main loss of our paper.

2. First, our hyperparameter tuning was only performed on the validation set, specifically to find the values of $\lambda_1$ and $\lambda_2$ that yield the best classification performance on the validation set, as shown in Appendix C. Methods like RMT and SATA also performed similar tuning. The reason for the small performance differences may be related to the single order of the domain. Most previous work on CTTA adopted a single order, which may not reasonably assess the performance of various methods. Therefore, we supplemented our study with new experiments (section 4.8). We randomly selected 10 different domain variation orders for testing, and the final results are averaged. It can be seen that in this case, our improvement is significant (1.6%, 3.1%, and 2.4% on CIFAR10-C, CIFAR100-C, and ImageNet-C, respectively). The table is shown in the figure below (which is Table 7 in the paper).

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1. How does this approach differ meaningfully from existing methods in CTTA

A1. In general, our approach is not fundamentally about maximizing inter-class distance and minimizing inter-domain distance in the representation space; in other words, this is not our starting point, but it does achieve these goals. The starting point of our method is to suppress the opposing shifts between categories in the representation space. We discovered through t-SNE feature dimensionality reduction that during the domain adaptation process, opposing shifts occur between categories, leading to blurred classification boundaries, and the situation worsens with continuous transformations of the domain. Therefore, we explicitly control the direction of category shifts to prevent them from occurring in opposition, ensuring a clear classification boundary. (This is what "Controllable" means in our method). Next, to demonstrate that we have achieved this, we plotted t-SNE graphs while comparing the inter-class distances of different methods (because theoretically, if categories do not shift in opposition but rather move away from each other, the inter-class distance should increase; thus, inter-class distance serves as an auxiliary tool to validate our method rather than being the starting point).

Q2. While the results slightly outperform previous approaches on average, I have concerns about whether hyperparameter tuning was conducted in a way that avoids overfitting to the evaluation benchmark.

A2. We have a total of three hyperparameters, among which the entropy threshold $E_0$ is directly taken from the parameters of EATA without any adjustment process, while $\lambda_1$ and $\lambda_2$ are derived from adjusting based on 4 different domains other than the 15 domains used in the experiments. For example, the CIFAR-10-C dataset actually has 19 types of corruption (different corruptions represent different domains), and our parameters were adjusted based on 4 corruptions (Speckle Noise, Gaussian Blur, Spatter, Saturate). In the formal experiments, we selected 15 completely different domains (Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation, Pixelate, JPEG).

Thank you for your interest in our work and your suggestions. Here is our answer to your confusion

1. We have always believed that CSCL loss is crucial for the TTA task because it can effectively control the shift in categories, preventing the blurring of classification boundaries caused by opposing shifts between categories, which leads to a decline in performance. As shown in Figure 3 of the article, both CoTTA and SATA methods exhibit blurred classification boundaries. For the CTTA task, continuous domain transformations result in increasingly blurred classification boundaries and accumulating errors, making CSCL even more necessary. We tested the scenario of cycling through 15 domains 10 times, and the experimental results are shown in Table 6 and Figure 5. It can be observed that our method improves classification performance as the domains are repeatedly encountered, ultimately achieving an average error rate that is significantly lower than that of other methods.

2. After our confirmation, there are no hyperparameters in CoTTA and SATA, and no methods for hyperparameter tuning are provided in RMT and EcoTTA. They only indicate the sensitivity of model performance to hyperparameters on the test set. However, the final reported results all use the best-performing hyperparameters, which is actually unreasonable.

If there are any parts of my response that still confuse you, please point them out, and I will reply promptly.