Variational Continual Test-Time Adaptation

Weakness of Noverty

1. CIFAR10 dataset is not enough

Response: Many CTTA methods focus on classification problems because validating classification on foundational and well-recognized datasets can reduce potential confounding factors and more accurately assess the effectiveness of the proposed methods. Many applications can be viewed as classification problems, such as semantic segmentation. We did not only conduct experiments on CIFAR10. In Tab. 2&3, we validated on CIFAR100 and ImageNet, which are generally considered more complex than CIFAR10. Many existing CTTA works have been conducted on basic classification tasks such as PETAL and RoTTA, which demonstrates that our experiments are sufficient. Given the very short rebuttal period, it is challenging to immediately validate the method's effectiveness in specific real-world scenarios, but we find your suggestion valuable and will incorporate it in the future.

2. BNN initialization

Response:

(1) The variational warm-up (VWU) strategy aims to obtain a suitable variational distribution on the source data, and is based on a pre-trained CNN. Of course we can train a BNN from scratch, but the training is difficult. VWU strategy is more like a pre-trained trick instead of BNN initialization.

(2) The initialization used by VCL is "the prior $N (0, I)$ and ... a very small initial variance (10^−6)", which is a direct initialization. However, VWU is different from the initialization method. The significant difference is that we require a complete pre-trained CNN, and the advantage of VWU can be seen in Appendix F. In Tab. 10, we have compared with common initialization and find that VWU is more convenient。

(3) [1] also aims to convert a pre-trained model into a BNN but depends to complicated model design. Instead, we prefer a simple reparameterization trick to solve this problem, which is to quickly offer a pretrained source BNN model for the following TTA algorithm.

(4) We have compared VWU with random initialization in Tab. 10. We further compare VWU with VCL initialization in Table 8 in the attached PDF and also find similar performance.

[1] Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models. CVPR. 2024.

Weakness of Clarity

1. Why does the teacher use EMA?

Response: In the mean-teacher (MT) structure, a student model is trained on unlabeled data and receives guidance from a teacher model. The teacher model is an exponential moving average (EMA) of the student model, provides stable pseudo-labels for the unlabeled data. EMA smooths the updates to the teacher model, reducing fluctuations. This stability helps prevent the student model from being misled by error accumulation, promoting more robust learning.

2. Why is data augmentation an important component?

Response: A large pool of augmentations and averaging them would give more robust evaluation. In Eq.10, we use the mean entropy derived from data augmentations to represent the confidence of the two prior models, and mix up the two priors with a modulating factor. Each addition item is a softmax with one type of augmentation. The softmax is to given the confidence of the source and teacher models. Eq.13 is an improved version for teacher log-likelihood in Eq.9, which picks up the augmented logits with a larger confidence than the raw data with $\epsilon$ margin. We have conducted the ablation study on augmentation in Appendix E.2, Tab.9. Tab.9 shows that increasing the number of augmentations can enhance effectiveness.

Weakness of Notation

1. "scale-mixture prior"?

Response: The title "Mixture-of-Gaussian prior" is a simplification of "a scale mixture of two Gaussian densities as the prior" as description in the mentioned paper. Thanks to the suggestion, we will add a description and citation in the section to distinguish our method from the Gaussian Mixture Model method to avoid confusion.

2. p0 (Fig 2) and p1 (Eq 11)

Response: This is a typo error, we will revise this.

Weakness of Experiment

1. No standard errors

Response: We have provide the standard error comparison in Appendix H, Table 11. In Table 11, we conduct 10 different orders, and show the average performance of all compared methods. The proposed VCoTTA outperforms other methods on the standard error of three dataset, which shows the effectiveness of the prior calibration in CTTA.

2. Why use BS to evaluate uncertainty

Response: BS is a well-recognized uncertainty estimation method, quantifying the MSE between predicted probabilities and actual outputs. [1] states that "one of the first metrics ... widely used ... is Brier score (BS)". BS has been widely used in many uncertainty estimation such as [1-4]. We also evaluate uncertainty using ECE metric, and the results can be seen in Table 9 in the attached PDF, and we can find similar results as BS.

[1] Better Uncertainty Calibration via Proper Scores for Classification and Beyond. NeurIPS 2022.
[2] A probabilistic framework for lifelong test-time adaptation. CVPR 2023.
[3] Calibration of neural networks using splines. ICLR 2020.   
[4] Intra order-preserving functions for calibration of multi-class neural networks. NeurIPS 2020.

Question 1: Line 169: what is a “serious data augmentation”?

Response: This should be "a series of data augmentation". We will revise this.

Question 2: Line 178: what is a confidence function?

Response: Max value of softmax output.

Question 3: Eq 13: what is x′?

Response: x is the raw test data while x' is its augmented version.

Question 4: Alg 1, line 2, initialize the variances to be some small?

Response: Yes, in VWU, we initialize the varizances with log(1+e^-4)

Question 5: how is $\bar{p}_1$ defined?

Response: The teacher model is initialized to be the same as the student model and will be different from the student after updating.

Weakness 1: Further analysis on efficiency

Response: Our approach incorporates a Variational Warm-Up (VWU) strategy during pretraining and utilizes VCoTTA for test-time adaptation. We conduct additional cost analyses under various settings, including different batch sizes and model sizes. The detailed cost results are provided in Tables 4 and 5 of the attached PDF for both the warm-up and test phases. The results indicate that while the VWU strategy becomes more efficient with an increase in batch size, this also leads to higher memory consumption. Similarly, during the test phase, batch size has a similar effect, and the efficiency is significantly influenced by the model size too. It is important to note that CTTA comprises two primary components: testing and adaptation. The use of Bayesian methods does not impact testing efficiency, as variance is not actively engaged during inference.

Weakness 2: Typographical and grammatical errors

Response: Thank you for your valuable suggestion. We will thoroughly review the paper for any typographical and grammatical errors and make revisions to improve clarity and readability. Additionally, we will include more references to substantiate the accuracy of our claims.

Weakness 3: Revision suggestion on the heading of Sec. 5.7.

Response: Thank you for your suggestion. Since Section 5.7 presents experiments on continual domains with gradually changing shifts, we will update the heading to "Comparisons under Gradually Changing Domain Shifts".

Weakness 4: Mean-teacher statement clarification

Resonse:

(1) "where the teacher model...." lacks specificity: The mean-teacher (MT) structure is a method in semi-supervised and unsupervised learning where a student model, trained on both labeled (if semi-supervised) or unlabeled data, receives guidance from a teacher model. The teacher model, which is an exponential moving average (EMA) of the student model’s weights, provides stable pseudo-labels for the unlabeled data. This setup encourages the student model to produce consistent predictions and effectively leverage large-scale unlabeled data, improving generalization and overall performance by stabilizing training and enhancing learning from broader data distributions.

(2) Which model (teacher or student) benefits from this guidance and in what manner: In the mean-teacher structure, the student model benefits from the guidance provided by the teacher model. The teacher model is an EMA of the student model's weights. These pseudo-labels guide the student model during training by providing consistent targets for the unlabeled data, which helps the student model to generalize better and improve its performance. This process ensures that the student model learns more robust features from the unlabeled data, enhancing its ability to make accurate predictions on new, unseen data.

(3) "where the teacher model guides the unlabeled data" seems incorrect: We will revise the explanation of the mean-teacher structure in our paper and provide more reference to support our statement. Thank you for your suggestion!

Weakness 1: Why superior to SOTA

Response: Our method outperforms the SOTA approaches because it leverages the BNN ability to estimate model uncertainty, which reduces error accumulation from continual unknown domains during the testing phase. We find that the unreliable priors may affect the performance of BNNs in the CTTA task. To address this, we utilize variational inference to compute the Evidence Lower Bound (ELBO), and propose to improve the calculation of the Entropy and KL terms. For the Entropy term, we propose using a Mean-Teacher (MT) structure to transform the original conditional entropy into cross-entropy, taking advantage of MT's delayed update characteristic. For the KL term, we introduce a Gaussian mixture prior enhancement method that directly reduces the impact of unreliable priors. Additionally, the variational weight uncertainty strategy enables the model to have some before-test uncertainty estimation capability. These modules allow the proposed Bayesian method to mitigate the influence of unreliable priors in CTTA tasks, leading to better performance. All of these modules are explained in detail in the paper, and ablation experiments are provided. If there are any unclear aspects, please feel free to ask further questions.

Weakness 2: Motivations of our design

Response: The key components and workflow as been shown in the above response. We further explain Eq. 10 and Eq. 13 as follows. In Eq 10, we use the mean entropy derived from a given series of data augmentation to represent the confidence of the two prior models, and mix up the two priors with a modulating factor. Each addition item is a simple softmax with one type of augmentation, and $\mathcal{I}$ denotes the augmentation types. The softmax is to given the confidence of the source model and $1-\alpha$ means the confidence of the teacher model. Eq. 13 is an improved version for teacher log-likelihood in Eq. 9. Eq. 13 picks up the augmented logits with a larger confidence than the raw data with $\epsilon$ margin. Eq. 13 can be regarded as a filter, meaning that for each sample, the reliable teacher is represented by the average of its augmentations with $\epsilon$ more confidence than the raw data's.

Weakness 3: Comparison with CoTTA

Response:

(1) BNN for CTTA task: First, BNNs offer several advantages over traditional CNNs, especially in scenarios where uncertainty estimation, robustness to overfitting, and the ability to incorporate prior knowledge are important. BNNs provide several features that are beneficial for test-time adaptation, including uncertainty estimation, robustness to overfitting, the incorporation of prior knowledge, adaptive complexity, and enhanced interpretability. These advantages enable BNNs to adapt more effectively when encountering new, unseen, or uncertain data during test-time, making them well-suited for dynamic and evolving environments. However, directly using existing BNN for CTTA is ineffective because of the unreliable prior, and our goal is to reduce the influence of unreliable prior. See more detail in the response of Weakness 1. Some works using BNN for traditional TTA task can be found in [1-3].

(2) CoTTA vs. Ours: In comparison with CoTTA, we have the following difference：

1. CoTTA is based on CNN while our method is based on BNN, and the advantage of BNN for CTTA task can be seen above.
2. CoTTA focuses on the error accumulation and catastrophic forgetting when using CNN. Our method solves the unreliable prior issue under BNN structure, which may lead to error accumulation and catastrophic forgetting.

(3) About the reviewer's understanding: The objective the reviewer mentioned is the ELBO (Eq. 6) of Variational Inference (VI) in CTTA, which is derived from the VI assumption. VI assumes that there exists a variational distribution $q(\theta)$ that approximates the true posterior $p(\theta|\mathcal{U})$ . The approximation process can be represented by a KL divergence optimization (Eq. 5). However, it is difficult to directly optimize the KL divergence, the ELBO is an alternative for optimization. For prediction, the BNN can be reduced to a CNN because the variance is not used

[1] Extrapolative continuous-time bayesian neural network for fast training-free test-time adaptation. NeurIPS, 2022.
[2] Bayesian adaptation for covariate shift. NeurIPS, 2021.
[3] Task-agnostic continual learning using online variational bayes with fixed-point updates. Neural Computation, 2021.

Question 1: Long adaptation period

Response: We have evaluated on long-adaptation period in our Appendix I, Fig. 6. In Fig. 6, we evaluate on 10 loops of a same corruption order, which means we have 15 domain shift repeat 10 times, yielding 150 domain shifts, we believe that 150 domain shifts is a long adaptation period. The results show that most compared methods obtain performance improvement in the first several loops, but suffer from performance drop in the following loops. This means that the model drift can be even useful in early loops, but the drift becomes hard because of the unreliable prior. The results also indicate that our method outperforms others in this long-term adaptation situation and has only small performance drops.

Question 2: Motivation and ablation study on Eq. 13

Response: The motivation of Eq.13 can be seen in the response of Weakness 2. We have conducted the ablation study in Appendix E.2, Table 9. Table 9 shows that increasing the number of augmentations can enhance effectiveness, but this hyperparameter ceases to have a significant impact after reaching 32.

Question 3: Mode averaging

Response: We guess the reviewer means "model averaging", i.e., set $\alpha=0.5$ for Eq. 11. Yes, this has been verified in Table 5 in the submitted manuscript. We may misunderstand the question, and the reviewer can give further description.

Question 4: One gradient step

Response: Yes, your are right.

Thank you for your comments, we response to the difference of low-level distinctions between CoTTA and VCoTTA as follows.

1. CoTTA Contributions and Equations: First, let's review the three main contributions of CoTTA and then explain the equation differences between our approach and CoTTA in regard to the 3 contributions and 8 equations. Note that Eq. 1, 2 and 5 are the updates for student and teacher models.

   • Weight-Averaged Pseudo-Labels: This contribution is in fact the Mean-Teacher (MT) structure use in CTTA. We also adopt the MT structure like CoTTA’s design. The equation differences are:
     • CoTTA Eq. 1&5: Student update. The reviewer has mentioned this, and our further explanation is provided below (Response 2).
     • CoTTA Eq. 2: Teacher update. Both CoTTA and VCoTTA use EMA for updating, but VCoTTA is based on BNNs (Our Eq. 16), where the BNN is updated in Gaussian distribution.
   • Augmentation-Averaged Pseudo-Labels (CoTTA Eq. 3&4): CoTTA chooses to use either augmented or non-augmented teacher pseudo-labels based on whether the source model's confidence in the sample exceeds a certain threshold. In our method, we do not make such a judgement. For the augmentation, we first leverage data augmentation to calculate the mixing coefficient between the teacher prior and the source prior (Our Eq. 10), then enhance the teacher prior (Our Eq. 13) as we mentioned in the rebuttal.
   • Stochastic Restoration (CoTTA Eq. 6&7&8): CoTTA randomly restores part of student model parameters to the corresponding parameters in the source model with a certain probability to preserve the knowledge from the source. Our method does not employ this stochastic restoration strategy. Our mixture of priors (Our Eq. 11,12,15) may have similar form as CoTTA Eq. 8, but we do not restore student model using source model randomly but mix the teacher prior and source prior with an adaptive factor $\alpha$.

2. Difference in Student loss: The reviewer has pointed out the student loss difference in low-level. Compared to CoTTA, the main difference in student loss is the KL term at the end of our Eq.15. The KL term can be considered a regularization constraint on the model parameters. $q_t$ , $q_0$ and $\bar{q}_t$ represent the parameter distributions of the student model (to be optimized), source model, and augmented teacher model, respectively. In our method, all three distributions are Gaussian. The calculation of the KL term for Gaussian distributions can be referenced in Eq. 25 of our appendix, which can be computed directly in closed form. The meaning of the KL term is to provide constraints on the student model update when faced with unreliable priors, using both the source model and teacher model. These two constraints are controlled by the weights calculated in our Eq. 10.

Thank you to the reviewer for pointing out that we need to improve our method presentation. We will enhance the clarity of our descriptions in the paper to make the meaning of the formulas easier for readers to understand.

Weakness 1: Bayesian approach for CTTA and why superior to SOTA

Response:

(1) Bayesian approach for CTTA

Bayesian networks have already been applied in the field of TTA task. For example, [1] develops a continuous-time Bayesian neural networks to process non-stationary streaming data in real-time. [2] provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters based on Bayesian method at test-time. BayTTA [3] designs a Bayesian model for medicine TTA task. Actually, Bayesian networks can also be applied to online learning. [4] proposes a Bayesian-inference based recommendation system for online social networks. [5] is based on Bayesian networks, and is assumed to transition smoothly in the joint space of numerical parameters and graphical topology, allowing for robust online network learning.

In CTTA task, in addition to efficiency, stable testing and adaptation are also important. A CTTA model may suffer from error accumulation due to uncertainties arising from multiple domain shifts over extended periods. Bayesian methods can mitigate this issue by estimating uncertainty. CTTA involves two main parts: testing and adaptation. The use of Bayesian methods does not impact testing efficiency, as variance is not actively engaged during inference.

[1] Huang H, et al. Extrapolative continuous-time bayesian neural network for fast training-free test-time adaptation. NeurIPS, 2022.
[2] Zhou A, et al. Bayesian adaptation for covariate shift. NeurIPS, 2021.  
[3] Sherkatghanad Z, et al. BayTTA: Uncertainty-aware medical image classification with optimized test-time augmentation using Bayesian model averaging[J]. arXiv preprint arXiv:2406.17640, 2024.  
[4] Yang X,  et al. Bayesian-inference-based recommendation in online social networks. IEEE TPDS, 2012.  
[5] Wang Z, et al. Time varying dynamic Bayesian network for nonstationary events modeling and online inference. IEEE TSP, 2010. 

(2) Why superior to SOTA

Our method outperforms the SOTA approaches because it leverages the BNN ability to estimate model uncertainty, which reduces error accumulation from continual unknown domains during the testing phase. We find that the unreliable priors may affect the performance of BNNs in the CTTA task. To address this problem, we utilize variational inference to compute the Evidence Lower Bound (ELBO), and propose to improve the calculation of the Entropy and KL terms. For the Entropy term, we propose using a Mean-Teacher (MT) structure to transform the original conditional entropy into cross-entropy, taking advantage of MT's delayed update characteristic. For the KL term, we introduce a Gaussian mixture prior enhancement method that directly reduces the impact of unreliable priors. Additionally, the variational weight uncertainty strategy enables the model to have some before-test uncertainty estimation capability. These modules allow the proposed Bayesian method to mitigate the influence of unreliable priors in CTTA tasks, leading to better performance. All of these modules are explained in detail in the paper, and ablation experiments are provided. If there are any unclear aspects, please feel free to ask further questions.

Weakness 2: Bayesian approach for CTTA and why superior to SOTA

Response:

In our paper, we have evaluate several highly related hyperparameters including confidence margin in Table 8, augmentation number in Table 9, warm-up epochs in Fig. 4, warm-up data scale in Fig. 5. We further provide some other hyperparameters in the attached tables including learning rate, batch size and the softmax temperature.

(1) Learning rate: See Table 1 in the attached PDF. (2) Batch size: See Table 2 in the attached PDF. (3) Softmax temperature: See Table 3 in the attached PDF.

Question 1: Space and time cost of offline and online phase

Response: We provide more cost experiments under different setting including different batch size of different model size. The detail cost values can be seen in Table 4 and 5 in the attached PDF.

Question 2: Reseting to source model

**Response: In CTTA task , model does not know when the domain shift happen. Thus, resetting model to source model is not feasible in CTTA task. We conduct this experiment on CoTTA and our method, the results can be seen in Table 7 in the attached PDF. We find that the resetting performance is less effective than the continual setting on both CoTTA and our method. The results show that the existing datasets may have shared knowledge across domains.

Question 3: Best non-adaptive $\alpha$

Response: We further evaluate on more $\alpha$, and the results can be seen in Table 6 in the attached PDF. The best non-adaptive $\alpha$ is 0.7. The results show that setting a fixed $\alpha$ is not effective enough for CTTA task. This underscores the significance of striking an adaptive balance between the two prior models in an unsupervised environment. The trade-off implies the need to discern when the source model’s knowledge is more applicable and when the teacher model’s shifting knowledge takes precedence.

Thank you for the additional results. These resolved some additional concerns I had about the paper. I am raising my score from 5 to 6.