DOTA: Distributional Test-Time Adaptation of Vision-Language Models

Weakness 1. Why updating the feature distribution for different categories works better?

The proposed online data estimation method is based on the method in the existing paper[1]. Theoretically, for any sequence, the average regret of the proposed method converges to zero in the limit. This means that the proposed method can finally obtain an accurate estimation of the test data distribution, leading to better classification performance.

In addition, we visualized the improvement in accuracy during the test process, and we can see that as the number of test samples increases, the model performance continues to improve.

Weakness 2. Difference and the advantages of the proposed method compared with T3A.

T3A is similar to TDA [2], an important baseline and reference in this paper, as both store representative test samples to guide the classification of subsequent ones. However, this approach represents an inefficient use of test samples. Specifically, T3A and TDA naively maintain a limited cache of representative samples during testing, dynamically updating the cache with higher-confidence samples. This strategy results in test-time forgetting, as older cached samples are discarded when new confident samples are added.

In contrast, our proposed method does not rely on preserving representative features. Instead, it continuously estimates the distribution of test samples, allowing the model to adapt dynamically to the test environment. This design enables the model to maintain long-term adaptability to changes in the environment, avoiding the pitfalls of test-time forgetting. The related claims are also experimentally verified in Figure 3.

Weakness 3. Is it suitable for traditional TTA or Domain Generalization tasks?

The proposed method is a general continuous estimation method. However, in non-visual language models, appropriate adjustments need to be made to adapt to changes in application scenarios. For example, the initialization of the mean feature vectors of different categories needs to be adjusted. Due to the limitations of the length and focus of the article, we will leave the extension to other scenarios for future work.

The proposed method is also applicable to Domain Generalization tasks. We report the results of some Domain Generalization tasks datasets (variant datasets of ImageNet) in the paper, which shows that the proposed method can achieve good performance.

Weakness 4. More details and explanation about the $f_k(x)$ in Eq.(3).

Thank you for your suggestion. We will include the corresponding explanation in future revisions. Equation 3 is derived from the classic Gaussian Discriminant Analysis algorithm, which constructs the classifier by estimating the data distribution for different categories. A detailed derivation process and explanation will be added in future versions to ensure clarity. Here we offer a shortened explanation.

Explanation of $f_k(\mathbf{x})$:

The function $f_k(\mathbf{x})$, often referred to as the discriminant function, measures how well a data point $\mathbf{x}$ fits the distribution of class $k$. It combines two key components of Gaussian Discriminant Analysis:

1. Mahalanobis Distance:

   $$-\frac{1}{2} (\mathbf{x} - \mu_k)^T \Sigma_k^{-1} (\mathbf{x} - \mu_k)$$

   This term calculates the squared Mahalanobis distance between $\mathbf{x}$ and the class mean $\mu_k$, scaled by the covariance matrix $\Sigma_k$. It measures how far $\mathbf{x}$ is from the mean of class $k$, taking into account the variance and correlation of features within that class.

2. Normalization Term:

   $$-\frac{1}{2} \log |\Sigma_k|$$

   This term accounts for the determinant of the covariance matrix $\Sigma_k$, which represents the spread (volume) of the Gaussian distribution for class $k$. It ensures that classes with larger variances are appropriately normalized.

Intuition:

• A larger $f_k(\mathbf{x})$ value indicates a higher likelihood that $\mathbf{x}$ belongs to class $k$.
• $f_k(\mathbf{x})$ combines both the distance from the mean (how similar $\mathbf{x}$ is to the class center) and the shape of the distribution (to adjust for variances).

In the context of classification, $f_k(\mathbf{x})$ is used in the softmax function to compute the posterior probability $P(y = k | \mathbf{x})$, which determines the most likely class for $\mathbf{x}$:

Weakness 5. The uncertainty estimation method is simple.

Thank you for your suggestion regarding uncertain sample selection. Based on your feedback, we extended our experiments to include entropy and variance-based methods in addition to our initial confidence-based baseline. We evaluated these methods on multiple datasets and tested their accuracy on the selected uncertain samples. The results are summarized below:

Experimental Results:

From the experimental results, the confidence-based method has the best uncertainty sample selection ability. The selected samples are more likely to be misclassified by the model.

To strengthen our evaluation, we added a more competitive baseline for test-time adaptation (TTA) based on human feedback. Since we are the first to define this specific problem, there are few directly related methods available for comparison. To address this, we compared our approach to a concurrent submission (ATPT [3]) to ICLR.

The comparison results, using 5% human feedback, are summarized in the table below. The proposed method significantly outperforms ATPT.

Moreover, we are also changing the previous comparison method to introduce human feedback and provide a stronger baseline. Specifically, we added an uncertain sample cache on the TDA to maintain the mean of uncertain samples. The experimental results are shown below. The experimental results indicate that the proposed method achieves superior performance. However, TDA may fail in certain cases due to its inefficient utilization of human feedback samples. Specifically, merely storing uncertain samples and their labels in the cache does not lead to performance improvement, possibly because the features of uncertain samples may contain data noise.

Weakness 6. Ablation study for adaptive fusion probability.

We conducted corresponding experiments and analysis. The table illustrates the performance improvements achieved with different $\eta$ and $\rho$ values compared to a standard zero-shot classifier. When $\rho$ is fixed, performance decreases as $\eta$ increases, indicating that higher $\eta$ values negatively impact performance. Conversely, when $\eta$ is fixed, performance gradually declines as $\rho$ increases, reflecting a consistent downward trend. However, the model's performance remains relatively robust to changes in these hyperparameters. The experiments were carried out using a ViT-B/16 backbone on the ImageNet validation set.

It can be observed that as $\rho$ increases (indicating the diminishing effect of dynamic fusion), the performance of Dota consistently decreases.

[1] On-line estimation with the multivariate gaussian distribution

[2] Efficient test-time adaptation of vision-language model

[3] Active test time prompt learning in vision-language models

Comparison with T3A

T3A is a classical test-time adaptation method. However, if T3A does not discard old samples, the repository grows larger as the number of test samples increases, leading to higher storage and computation costs. Specifically, its storage and computational complexity are $O(n)$, where $n$ is the number of test samples. In contrast, our method only maintains the mean and covariance matrix without requiring additional storage space. Thus, both its storage and computational complexity are $O(1)$.

On the negative impact of low-confidence samples:

1. In fact, our initial version of the proposed method updated model parameters using only high-confidence samples, following the strategy used in T3A and TDA methods. However, the performance improvement was limited compared with TDA. The fundamental reason is that low-confidence predictions also contain information that is beneficial for model training. The experimental results are shown in the following table

2. Theoretically, the objective function of the proposed method can also be equivalent to certain prior test-time adaptation approaches, such as entropy minimization. Specifically, previous entropy minimization methods can be viewed as minimizing the cross-entropy loss between the current predicted class probability and the predicted class probability.

3. This process can also be interpreted as a single iteration of the Expectation-Maximization (EM) algorithm \citep{moon1996expectation}. Importantly, just like traditional EM algorithms, our method uses the predicted probabilities of the old classifier to train the new classifier. Specifically, obtaining the zero-shot classification probability corresponds to the expectation step, while estimating $\{{\mu_k, \Sigma_k}\}$ based on the zero-shot predicted probability corresponds to the maximization step, adhering to the principle of maximum likelihood estimation.

4. Moreover, the estimation process can also be intuitively understood as reweighting, where the zero-shot predicted probabilities (usually well calibrated[1]) are used as weights to adjust the contributions of samples to different classes, thereby mitigating the impact of the potential inaccuracies in the zero-shot predicted probabilities.

5. Finally, in practice, on ImageNet test set, each class contains only 50 samples, which may not be sufficient for the model to select a large enough number of high-confidence test samples for distributional test-time adaptation. Therefore, it is crucial to make effective use of the information from low-confidence samples. Meanwhile, due to the inherent biases of CLIP, it may naturally exhibit low confidence for certain categories, making it difficult to adapt to these categories when using only high-confidence samples.

However, we also very acknowledge your perspective that, in some cases, using only high-confidence samples can also be a good choice, and it serves as a complementary approach to the method we proposed.

[1] Revisiting the calibration of modern neural networks.

Dear reviewer, we have update the manuscript with our latest discussion and promise to improve it further.

Weakness 1. Detailed case study focused on a particular domain or a challenging dataset.

We did not use other more challenging datasets initially to maintain a comparison with previous TTA works. Based on your suggestion, we selected three challenging dataset in the medical domain for evaluation. Below, we provide the details of the experimental setup, datasets, and results:

1. Experimental Setup:

• Backbone: PLIP[1]
• Baseline: We used PLIP's predictions as the baseline.

2. Datasets:

We evaluated our method on the following medical image datasets, where labels were converted into descriptive sentences (e.g., converting "tumor" into "H&E image of a tumor") for evaluation. The model was tested on the test dataset without additional fine-tuning.

1. Kather colon dataset (9 different tissue types).
2. PanNuke dataset (benign vs. malignant).
3. WSSS4LUAD dataset (tumor vs. normal).

3. Unconfident Sample Selection Method:

We adopted confidence-based selection for identifying low-confidence samples for human feedback.

4. Experimental Results:

The following table summarizes the accuracy results. Note that "Human Feedback" indicates the accuracy achieved with the default evaluation setting, while "Human Feedback*" reflects the accuracy when all samples with ground truth labels are considered correct, as per the reviewer's suggestion.

5. Observations:

The proposed method (DOTA) with human feedback consistently improves accuracy compared to the baseline (PLIP) across datasets, especially when using the suggested evaluation setting (Human Feedback*).

Weakness 2. Hyperparameter analysis.

To better validate the sensitivity of our model to hyperparameters, we conducted systematic experiments and analyses with the following details:

1. On the Selection of Hyperparameter $\sigma$. We tested the values of $\sigma$ in the range $[0.0001, 0.001, 0.002, 0.004, 0.008, 0.02]$ while keeping other parameters fixed. The experimental results show that the model demonstrates strong robustness to different $\sigma$ values. The accuracy (acc) for each value of $\sigma$ is as follows:

   $\sigma$	0.0001	0.001	0.002	0.004	0.008	0.02
   Acc	70.58	70.63	70.68	70.64	70.56	70.36

   • These results indicate that the impact of $sigma$ on the model's performance is minimal, showcasing its robustness to this parameter.

2. On the Selection of Hyperparameters $\eta$ and $\rho$: Further experiments were conducted by fixing $\sigma$ and testing $\rho$ values in $[0.005, 0.01, 0.02, 0.03]$, while adjusting $\eta$ in $[0.2, 0.3, 0.4, 0.5]$. The results are summarized in the table below:

   $\eta \backslash \rho$	0.005	0.01	0.02	0.03
   0.2	70.68	70.66	70.51	70.43
   0.3	70.66	70.51	70.28	70.16
   0.4	70.66	70.48	70.19	70.03
   0.5	70.66	70.44	70.08	69.91

These results further demonstrate the robustness of our model to the selection of $\eta$ and $\rho$, validating the stability of our proposed method. The above experiments were conducted using the ViT-B/16 backbone on the ImageNet validation set.

Weakness 3. visualizations and analyzation of uncertain samples.

Thank you for your suggestion to provide additional visualizations illustrating test-time adaptation with human feedback. To address this, we conducted an analysis on ImageNet to investigate the distribution of human feedback across different classes and to determine whether the feedback is concentrated in specific categories. We included samples with 5% and 15% human feedback for this analysis.

From the results, it can be observed that when the human feedback rate is set to 5%, approximately half of the categories receive at least one ground truth label from human feedback. However, the performance improvement due to human feedback remains relatively consistent even when only a limited number of samples per category are labeled.

Below is the distribution of human feedback under a 5% feedback setting:

[0, 300]: 300 classes receive no feedback. [1, 203]: 203 classes receive exactly one ground truth label. ['2–6', 390]: 390 classes receive between 2 and 6 ground truth labels. ['7–11', 92]: 92 classes receive between 7 and 11 ground truth labels. ['12–16', 15]: 15 classes receive between 12 and 16 ground truth labels.

This analysis highlights two key observations:

• The proposed method achieves stable performance improvements even with an uneven distribution of human feedback across categories.
• The more challenging samples tend to correlate with class imbalance.

Question 1. Availability of code.

We will definitely open source our code, and in fact our code is already ready.

Question 2. Why is DOTA's improvement on ResNet small?

The potential reason lies in the difference in image representation dimensions obtained by ResNet and ViT-B/16. Specifically, the representation dimension of ResNet is 1024, whereas for ViT-B/16, it is 512. In our method, this disparity results in a significant change in the number of parameters required for estimating the test data distribution. Without any simplification, the number of distribution parameters that need to be estimated for each category is $D(D+1)$, where $D$ is the representation dimension. Consequently, the number of parameters for ResNet is approximately four times that of ViT-B/16. This difference may explain the slight improvement in model performance when using the ResNet backbone.

[1] Pathology Language and Image Pre-Training

We thanks for your valuable suggestions and we will try to address your concerns as follows.

Weaknesses 1. Factors analysis associated with the experimental protocol.

Subweakness a. Hyperparameter analysis.

To better validate the sensitivity of our model to hyperparameters, we conducted systematic experiments and analyses with the following details:

1. On the Selection of Hyperparameter $\sigma$. We tested the values of $\sigma$ in the range $[0.0001, 0.001, 0.002, 0.004, 0.008, 0.02]$ while keeping other parameters fixed. The experimental results show that the model demonstrates strong robustness to different $\sigma$ values. The accuracy (acc) for each value of $\sigma$ is as follows:

   $\sigma$	0.0001	0.001	0.002	0.004	0.008	0.02
   Acc	70.58	70.63	70.68	70.64	70.56	70.36

   • These results indicate that the impact of $sigma$ on the model's performance is minimal, showcasing its robustness to this parameter.

2. On the Selection of Hyperparameters $\eta$ and $\rho$: Further experiments were conducted by fixing $\sigma$ and testing $\rho$ values in $[0.005, 0.01, 0.02, 0.03]$, while adjusting $\eta$ in $[0.2, 0.3, 0.4, 0.5]$. The results are summarized in the table below:

   $\eta \backslash \rho$	0.005	0.01	0.02	0.03
   0.2	70.68	70.66	70.51	70.43
   0.3	70.66	70.51	70.28	70.16
   0.4	70.66	70.48	70.19	70.03
   0.5	70.66	70.44	70.08	69.91

These results further demonstrate the robustness of our model to the selection of $\eta$ and $\rho$, validating the stability of our proposed method. The above experiments were conducted using the ViT-B/16 backbone on the ImageNet validation set.

Subweakness b. Results across multiple data ordering.

We conducted experiments on five test data in different ordering on multiple datasets. The experimental results are shown in the following table. From the experimental results, we can see that the order of test data has little effect on the prediction performance.

Weakness 2. More baselines.

We added the following baselines accepted at NeurIPS 2024 most recently. Among the compared methods, BoostAdapter and HisTPT both have the ability of continuous learning. Compared with these methods, the proposed method is still competitive. When DOTA with human feedback is performed, the proposed method is able to achieve even more leading performance.

Weakness 3. Adapting to an evolving stream.

We combined the ImageNet and ImageNet-V2 datasets, first testing the performance on ImageNet and then on ImageNet-V2. This setup evaluates the model's ability to adapt to a continuously changing data stream. The experimental results are shown below.

For example, when testing ImageNet-V2 separately, the performance is 64.41, but after integration with ImageNet, the performance slightly improves to 65.09. Investigating test-time adaptation under changing data stream distributions is an interesting direction; however, it may be beyond the scope of this study.

Weaknesses 4. Test-time forgetting of TDA.

1. In principle, when TDA adapts during testing, it will continuously discard the information stored in its cache, which will also cause it to continuously forget information.
2. Moreover, we conduct experiments on the ImageNet dataset. When testing on the ImageNet dataset, we record the performance of the most recent 5,000 test samples and compare them with the original zero-shot classifier performance, recording the relationship between the improvement in model performance and the number of test samples seen. The experimental results are shown in the following table. From the experimental results, we can see that the performance of TDA increases first and then decreases (due to its forgetting during testing), while the proposed method does not.

Weakness 5. TTA with human feedback

Subweakness a. Stronger baseline

Thanks for your suggestion, we added a stronger baseline for TTA based on human feedback. Please note that since we are the first to define this problem, there are few related methods. Therefore, we compared it with a paper (ATPT[4]) submitted to ICLR at the same time. The experimental results demonstrate that our approach significantly outperforms the proposed method.

Moreover, we are also changing the previous comparison method to introduce human feedback and provide a stronger baseline. Specifically, we added an uncertain sample cache on the TDA to maintain the mean of uncertain samples. The experimental results are shown below. The experimental results indicate that the proposed method achieves superior performance. However, TDA may fail in certain cases due to its inefficient utilization of human feedback samples. Specifically, merely storing uncertain samples and their labels in the cache does not lead to performance improvement, possibly because the features of uncertain samples may contain data noise.

Subweakness b. Effectiveness evaluation of selecting uncertain samples based on confidence.

To emphasize the necessity of human feedback, we introduce an additional evaluation metric (ACC*): for samples with human feedback, the labels updated after human feedback are used as the predicted labels to evaluate the accuracy of all test samples. The experimental results are as follows. It can be seen from the experimental results that When confidence-based uncertain sample selection is used, the model performance can be significantly improved.

Subweakness c. Regarding confidence calibration.

The paper proposes a confidence-based uncertainty sample selection baseline. It assumes that CLIP is relatively well calibrated. Corresponding analysis has been conducted in some existing works, for example, the Expected Calibration Error of CLIP on the ImageNet dataset is 1.51%[5]. More discussion will be added to the paper.

Subweakness d. The accuracy of the random baseline in Table 7.

We have added the experimental results. The reason we do not report a random sampling baseline is that when the amount of test sample data is large enough, the accuracy of random sampling should be close to the accuracy of the zero-shot classifier.

Minor weakness 1. Experiments on other models.

1. Experimental Setup:

• Backbone: PLIP[1]
• Baseline: We used PLIP's predictions as the baseline.

2. Datasets:

We evaluated our method on the following medical image datasets, where labels were converted into descriptive sentences (e.g., converting "tumor" into "H&E image of a tumor") for evaluation. The model was tested on the test dataset without additional fine-tuning.

1. Kather colon dataset (9 different tissue types).
2. PanNuke dataset (benign vs. malignant).
3. WSSS4LUAD dataset (tumor vs. normal).

3. Unconfident Sample Selection Method:

We adopted confidence-based selection for identifying low-confidence samples for human feedback.

4. Experimental Results:

The following table summarizes the accuracy results. Note that "Human Feedback" indicates the accuracy achieved with the default evaluation setting, while "Human Feedback*" reflects the accuracy when all samples with ground truth labels are considered correct, as per the reviewer's suggestion.

Minor weakness 2. We added the experimental results of TDA in Table 4.

We ran the TDA code in our local environment and reported the performance on the entire test data and the last 50% of the samples. The experimental results are shown below. From the experimental results, we can see that the performance of the last 50% of samples of our method is almost higher than that of all test samples, which shows that the performance of our method is improving. However, TDA is different, and its performance has declined on the caltech101, cars, and pets datasets.

[1] Frustratingly Easy Test-Time Adaptation of Vision-Language Models

[2] BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping

[3] Historical Test-time Prompt Tuning for Vision Foundation Models

[4] Active test time prompt learning in vision-language models

[5] Open-Vocabulary Calibration for Fine-tuned CLIP

[6] Pathology Language and Image Pre-Training

We thanks for your valuable suggestions and we will try to address your concerns as follows. We are eager to engage in a more detailed discussion with you.

Weakness 1. How to conduct Distributional Test Time Adaptation (DOTA).

As shown in Fig. 1 in the paper, DOTA dynamically updates distributions of different classes during testing. Specifically, DOTA always maintains the distribution information of different classes (i.e., mean and covariance matrix) during testing, and updates its distribution information based on its representation information after obtaining new samples (Eq. 6 in the manuscript).

The setting of DOTA. Benefiting from the test-time adaptation (TTA) framework, which leverages data distribution estimation without altering the original model parameters or prompts, and utilizing a vectorized distribution estimation strategy, the proposed method is inherently independent of test batch size. This flexibility arises because the vectorized approach allows the model to effectively adapt to the data distribution across multiple instances simultaneously, enabling it to work seamlessly in both single-image TTA settings (batch size = 1) and multi-image TTA settings (batch size > 1). In practice, to be consistent with the comparison method, we set the batch size to 1.

Weakness 2. Samples for distribution estimation.

1. When there is no human feedback, all test samples are used to update the data distribution.
2. In practice, however, the potential parameter estimation errors caused by this data update method are controllable, thanks to the implicit weighting process in the update process (higher confidence samples have greater weights, and vice versa).
3. In principle, the current data distribution update process can be seen as a traditional EM algorithm. The EM algorithm follows the principle of maximum likelihood estimation, i.e., estimating model parameters by selecting parameter values that make the observed data most likely to occur.
4. To further avoid the negative impact of low confidence samples, we propose human-feedback-based TTA. This approach updates the test data distribution parameters by detecting low confidence samples and collecting human feedback.

Weaknesses 3 & 4. How to conduct TTA with human feedback.

1. Evaluation. To ensure that the proposed method is comparable with other methods in performance (using the same number of test samples), we evaluate accuracy during testing by using the predicted labels from before collecting human feedback, even for samples where human feedback is collected.
2. More evaluation metrics. To emphasize the necessity of human feedback, we introduce an additional evaluation metric (ACC*): for samples with human feedback, the labels updated after human feedback are used as the predicted labels to evaluate the accuracy of all test samples. The experimental results are as follows. It can be seen from the experimental results that when human feedback is used, the model performance can be significantly improved. Also note that since we are not always able to accurately identify misclassified samples (another difficult task), the accuracy improvement may be smaller than the proportion of human feedback.

3. Our method is a baseline version of TTA based on human feedback (as far as we know, we are the first to propose this paper that is very meaningful in real-world deployment), how to better collect human feedback, and based on human feedback further improving performance is a goal that needs attention in the future.

Weakness 5. Stronger baselines for TTA

We enhance the baseline from following three perspectives.

1. We incorporate more sample selection strategies. The experimental results are shown in the table below. From the experimental results, it can be seen that the confidence-based selection can achieve better performance.

2.We include another comparison method (ATPT)[1], which, like ours, is currently under review for ICLR 2025. The experimental results demonstrate that our approach significantly outperforms the proposed method.

3. Moreover, we are also changing the previous comparison method to introduce human feedback and provide a stronger baseline. Specifically, we added an uncertain sample cache on the TDA to maintain the mean of uncertain samples. The experimental results are shown below. The experimental results indicate that the proposed method achieves superior performance. However, TDA may fail in certain cases due to its inefficient utilization of human feedback samples. Specifically, merely storing uncertain samples and their labels in the cache does not lead to performance improvement, possibly because the features of uncertain samples may contain data noise.

[1] Active test time prompt learning in vision-language models

Weakness 6. Amount of human feedback.

1. Firstly , based on your suggestions, we added experiments with human feedback ratios of 1% and 2%, and the results are shown in the following table. As can be seen from the table, the model can still achieve absolute performance improvement with only 1% feedback.

The number of feedback instances is influenced not only by the feedback ratio but also by the number of test samples. In practice, for most datasets, a 5% feedback ratio typically requires only about 200 feedback instances, averaging around 2 feedback instances per category. However, when the feedback ratio is too small, each category may receive as little as 0.1 feedback instances. For example, in the case of ImageNet, a 1% feedback ratio results in approximately 0.1 feedback instances per category. How to improve model performance when there are very few feedback times is another promising direction in the future. But it may be beyond the scope of this paper.

Weakness 7. Choice of hyperparameters

During the experiment, to ensure a fair comparison with other methods, we adopted the same settings, specifically adjusting the model hyperparameters using the validation set. At the same time, we also reconducted the experiment, and the proposed method is basically quite robust to hyperparameters. This allows us to achieve quite good performance on the test set without tuning parameters. The experimental results are shown below.

To better validate the sensitivity of our model to hyperparameters, we conducted systematic experiments and analyses with the following details:

1. On the Selection of Hyperparameter $\sigma$. We tested the values of $\sigma$ in the range $[0.0001, 0.001, 0.002, 0.004, 0.008, 0.02]$ while keeping other parameters fixed. The experimental results show that the model demonstrates strong robustness to different $\sigma$ values. The accuracy (acc) for each value of $\sigma$ is as follows:

   $\sigma$	0.0001	0.001	0.002	0.004	0.008	0.02
   Acc	70.58	70.63	70.68	70.64	70.56	70.36

   • These results indicate that the impact of $sigma$ on the model's performance is minimal, showcasing its robustness to this parameter.

2. On the Selection of Hyperparameters $\eta$ and $\rho$: Further experiments were conducted by fixing $\sigma$ and testing $\rho$ values in $[0.005, 0.01, 0.02, 0.03]$, while adjusting $\eta$ in $[0.2, 0.3, 0.4, 0.5]$. The results are summarized in the table below:

   $\eta \backslash \rho$	0.005	0.01	0.02	0.03
   0.2	70.68	70.66	70.51	70.43
   0.3	70.66	70.51	70.28	70.16
   0.4	70.66	70.48	70.19	70.03
   0.5	70.66	70.44	70.08	69.91

These results further demonstrate the robustness of our model to the selection of $\eta$ and $\rho$, validating the stability of our proposed method. The above experiments were conducted using the ViT-B/16 backbone on the ImageNet validation set.

Finally, since we can collect human feedback during testing, it may also be an interesting direction to perform real-time parameter adjustment with the help of human feedback labels in the future.

Q1: Test Distribution Estimation

The proposed method is a general framework, allowing the batch size during test-time adaptation to be arbitrary. In Eq.4 and Eq.6, the iterative updating equation are expressed in the form of tensor operations to maintain their generality. However, to ensure a fair comparison with baseline methods, we set the batch size to 1 in practice. Specifically, the proposed method estimates the data distribution of all samples in the test set in an online manner, i.e., $\mu$ and $\sigma$. There is no need to store the representations of test samples; instead, only their statistical features ($\mu$ and $\sigma$) are maintained. When a new sample is obtained, the stored statistical parameters ($\mu$ and $\sigma$) are dynamically iterated and updated based on the information from the current sample and previous information. $n$ is the number of the test sample, indicating which test sample it is.

Instead of estimating a distribution for each sample, we estimate the data distribution of different categories for all test samples in an online manner, that is, given a sample, the estimated distribution is updated based on the information of the current sample.

Q2. Selection of uncertain samples.

Subquestion a. Why does confidence perform better than similarity in practice?

Compared to similarity, confidence is more discriminative for uncertain samples. The reason is that confidence implicitly takes into account the similarities of multiple different classes, whereas the maximum similarity only considers the similarity to a single class. For example, when an image is close to both class 1 and class 2, the similarities to both classes might be high. However, after applying softmax, the confidence will be distributed between class 1 and class 2, making it easier for us to identify this uncertain sample.

Subquestion b. Confidence estimated from zero shot classifier?

The confidence estimated is from zero shot classifier. The key factor is that in practice, the zero-shot classifier is a relatively well-calibrated classifier. For example, on the ImageNet dataset, the error between its average confidence and classification accuracy is approximately (Expected Calibration Error, ECE) 1.51%, that is, the error between the confidence it gives and the true accuracy is less than 1.51%. Existing research has also shown that the pre-trained CLIP model is well-calibrated in the zero-shot setting[1].

[1] Revisiting the Calibration of Modern Neural Networks

Because there may be some misunderstandings, we want to clarify any potential misunderstandings regarding our Distributional Test-Time Adaptation (DOTA) process. Below is a detailed explanation of the workflow:

1. Global Distribution Estimation:
   Unlike estimating a separate distribution for each individual sample, our approach estimates the overall test data distribution across different classes during testing. This is achieved in an online manner, considering all test samples in a streaming manner.

2. Class-wise Distribution Maintenance:
   For each class, we maintain distribution information in the form of a mean vector and a covariance matrix. As new samples are processed, these distribution parameters are updated iteratively by incorporating the information from previous estimated distribution and new samples.

3. Efficient Iterative Updates:
   The equation used to update the class-wise distribution is fully vectorized, making it computationally efficient and independent of the number of processed samples. This ensures scalability and feasibility for various testing scenarios (single image TTA and multi image TTA).

4. Bayesian Inference for Class Probabilities:
   With the updated mean vectors and covariance matrices for each class, we leverage Bayes' theorem to compute the class probabilities of each test sample. This enables robust and adaptive predictions during the testing phase.

5. Versatility of the Proposed Approach:
   Our proposed method is versatile, functioning seamlessly in both single-image and multi-image scenarios. Its vectorized distribution estimation strategy, coupled with the maintenance of class-specific parameters, ensures robust and efficient adaptation to diverse test-time data distributions.

We hope this explanation clarifies any doubts regarding our DOTA process and highlights its practical advantages. Thank you for your thoughtful feedback and the opportunity to further elucidate our approach.

If you have other confusions due to our unclear explanation, please do not hesitate to send us a message.

Dear reviewer,

Thank you for your constructive comments.

We evaluate the impact of incorporating human feedback on model performance using a larger dataset (over 1 million test samples). Specifically, we introduce more human feedback during the early stages of model testing (the first 50,000 samples), but stop introducing feedback or updating the model in the later stages of testing. The experimental results are as follows. It can be seen that when the model is adapted during testing with human feedback, the more test samples the model has in the future, the greater the benefits it brings, and the lower the cost of human feedback.

Assuming the number of test samples increases by 10 times, and the model performance remains stable ( due to the data are independent and identically distributed), we can hypothesize that the corresponding performance could be as follows:

These results suggest that incorporating human feedback, even in small feedback rate, is crucial for improving model performance and achieving higher levels of accuracy. We also acknowledge that there may be some trade-offs between the amount of human feedback introduced and the performance of the model, and that this may not be applicable in all scenarios.

Thank you very much for your thoughtful response.

We would be happy to provide an example to further explain why incorporating human feedback can be effective.

Consider the situation where the model is already deployed. Without human feedback, the model's performance will remain static and will not improve.

However, once human feedback is collected, the model’s performance can be further improved in terms of the standard ACC metric (standard acc means that when evaluating performance, without the help of human feedback labels). In other words, after deployment, even if we stop collecting human labels at a certain stage, the model's performance of DOTA will still be better than it was before the feedback loop began. This could be particularly important for certain datasets, such as the EuroSAT dataset, where introducing 5% human feedback resulted in an increase in the model's standard accuracy from 57% to 65%.

At the same time, as the model’s performance improves, we can gradually reduce the amount of human feedback required by using a higher feedback ratio in the early stages of deployment and a lower ratio in the later stages. When we stop collecting human feedback, the ratio of human feedback will continue to decrease as the number of test samples increases.

We hope the above discussion addresses your concerns. If you have any further questions, we would be happy to continue the conversation.

Finally, if you consider this an important task and scenario, it might be worthwhile to allow this paper to be accepted so that it can reach a wider audience and contribute to further improving performance of TTA with human feedback.

Dear reviewer, we have update the manuscript with our latest discussion and promise to improve it further.

Dear authors,

Thank you for the clarifications and the additional experiments performed for lower feedback rate. From a bigger perspective, I still have conflicting thoughts on the effectiveness of having human feedback during TTA. I do believe it is a problem worth exploring. Yet, even with all the experiments you have presented, I don't see it being effective enough in terms of the effort (having a human labeler) to reward (accuracy gain) ratio, in general. Only in the case of Eurosat, it seems to be useful, the results are shown with 5%. I still stand by the belief that 5% and 15% is too much labels to ask for. The paper focuses on this rate of feedback in most experiments. I do acknowledge the authors efforts in doing experiments for lower feedback rate. This only becomes part of the analysis done but not the main experiments nor is expected to in this process.

After all this I am choosing to be positively inclined by increasing the score, primarily because it is a problem that is worth exploring and acknowledging the efforts in this direction.

Some suggestions or perspectives, maybe beyond the purview of this work alone. From a practical perspective, what would be a good study on TTA with human feedback:

1. The focus of the work should be on trying to achieve significant gains with minimal feedback (1-2%). From a practical perspective, if 1% data is labeled, an overall gain in accuracy (ACC*) of about 3-4% would justify the labeling cost. (None of the results in this work show performance improvement in this order. I cannot comment on what is possible to achieve based on these observations. I am only intending to say what is practical to expect from an algorithm here).
2. A more practical metric is ACC*. Well, if one has asked the label, why should the prediction be used to calculate the performance. This also inherently measures the sample selection method also. Selecting harder or inaccurate samples for labeling would result in higher acc* compared to accuracy without human feedback.
3. How and when are the samples selected, especially in TTA where one does not have access to all data at once? This plays a major role and I believe it is crucial to design this process well.

I appreciate the authors for putting forth this problem setting. But it is still far from a practical approach for TTA with human feedback.

Dear authors,

Thank you for the response and clarifications. This work does provide some new insights for TTA in vision-language models. Although there are some similarities with existing works like T3A/TDA, as they dynamically update the weights of the classifier/adapter and Dota updates the feature distribution for classification. I would like to increase my score to 6.