1. Empirical evidence supporting saussianity assumption

Thank you for requesting empirical evidence to support the Gaussianity assumption of test-time embedding distributions in Dota，we believe that elucidating the rationale for this assumption will enhance the research community’s understanding of our method. We address this by providing theoretical and experimental support to validate the reliability of our Gaussian assumption.

Theoretical Support for Gaussianity

Our assumption that CLIP embeddings follow a multivariate Gaussian distribution is grounded in prior work. Chupin and Dahyot (2024)[1] show that a single Gaussian model achieves high accuracy in CLIP embedding spaces (e.g., CIFAR100: 89.0%), closely matching Gaussian Mixture Models (G=2: 89.7%), as CLIP’s contrastive learning promotes intra-class embedding clustering, approximating Gaussian distributions.

Experimental Validation

To validate this assumption, we conducted experiments on the ImageNet validation set (1000 classes), extracting 50 standardized CLIP embeddings per class. We employed: (1) Shapiro-Wilk tests to assess marginal normality across 512 dimension，as proposed by Shapiro and Wilk (1965) for evaluating normality in small to moderate sample sizes[2]; (2) single Gaussian fitting to compute log-likelihoods with class-specific means $\mu_c$ and shared covariance $\Sigma$， following the methodology described by Duda et al. (2001) for multivariate Gaussian modeling[3]. Results show a Shapiro-Wilk pass rate of 88.91% ± 3.98% (p > 0.05), confirming strong marginal Gaussianity, and a mean log-likelihood of -1206.84 ± 93.54, indicating a reasonable fit to $\mathcal{N}(\mu_c, \Sigma)$, within the typical range (-500 to -1500). These findings support Dota’s Gaussianity assumption. We will include this experiment in the revised manuscript to strengthen our analysis.

[1] Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces

[2] An Analysis of Variance Test for Normality (Complete Samples)

[3] Duda, R. O., Hart, P. E., & Stork, D. G. Pattern Classification (2nd ed.).

2. More baselines

Thank you for noting the absence of Bayesian and continual learning-based test-time adaptation (TTA) methods, such as SHOT and CoTTA, in our comparisons. To strengthen the evaluation of Dota’s performance, we supplemented experiments with O-TPT and CoTTA baselines, with results sourced from the O-TPT paper (ViT-B/16) and our adapted CoTTA tests, respectively. Additionally, we clarify the rationale for not including SHOT as a baseline.

Experimental Setup for CoTTA

To align CoTTA with our experimental setup (ViT-B/16 backbone, no data augmentation, online streaming updates), we modified CoTTA by:

(1) reducing batch size from 64 to 1;

(2) removing image augmentation;

(3) replacing ResNet50 with ViT-B/16 to match CLIP’s architecture.

Supplementary Results

Results indicate:

1. O-TPT optimizes text feature orthogonality to reduce expected calibration error (ECE), primarily addresses overconfidence in model predictions, resulting in modest accuracy improvements compared to baselines.
2. CoTTA does not significantly outperform baseline models, likely due to its parameter updates during training, which may hinder full utilization of CLIP’s original classification capabilities, compounded by architectural changes from ResNet to CLIP.

The reason why SHOT not included in the comparisons

SHOT was excluded as a baseline because it requires access to the entire target domain data for batch training, conflicting with TTA’s requirement for online streaming updates.

3. Cases in which zero-shot predictions are noisy or poorly calibrated

Thank you for your insightful comments regarding Dota’s performance when initial zero-shot predictions are noisy or poorly calibrated. To address this, we discuss the cases of noisy and poorly calibrated zero-shot predictions separately below.

Noisy Zero-Shot Predictions

Dota’s design leverages distribution estimation to enhance the predictive capability of the base model, even under inaccurate zero-shot predictions. Experimental results in Table 1 demonstrate that, on datasets where CLIP’s zero-shot performance is weak, Dota consistently improves accuracy, such as Aircraft (26.25% vs. 23.22%) and EuroSAT (62.78% vs. 50.42%), highlighting the robustness of our distribution-based approach.

Poorly Calibrated Zero-Shot Predictions

The zero-shot classifier (CLIP) used in our model has been shown to be well-calibrated by prior studies [4]. CLIP’s strong calibration effectively reduces the influence of potentially inaccurate zero-shot predictions, as samples with lower confidence contribute less to the final distribution estimation. In practical testing, even in scenarios where CLIP’s calibration is suboptimal, calibration techniques such as Temperature Scaling or FLYP can be applied first to calibrate the predictions, followed by updating Dota’s model parameters using the calibrated results.

[4]Revisiting the calibration of modern neural networks

4. Comparison with prototype-based continual learning methods under evolving distributions

Thank you for requesting a comparison between Dota and prototype-based continual learning methods in handling evolving distributions. TDA can be considered a prototype-based continual learning method as it maintains a dynamic cache of representative test-sample features (keys) and pseudo labels (values), akin to prototypes, to adapt to evolving distributions. In this experiment , we adopt TDA as a baseline for prototype-based continual learning methods.

Experimental Setup

To evaluate Dota’s superiority, we conducted an experiment on ImageNet-C (severity=5, datasets: brightness, fog, frost, snow) using a ViT-B/16 backbone. The setup involves sequential testing (brightness → fog → frost → snow) without resetting model parameters. Results are shown below:

Experiment Results Analysis: Dota consistently outperforms TDA across all datasets, achieving a higher average accuracy (42.85% vs. 41.63%) and greater stability during distribution shifts, demonstrating superior robustness in handling evolving distributions compared to prototype-based methods. We will include this analysis in the revised manuscript to further clarify Dota’s advantages. Additional details are available upon request.

5. Visualization of evolving class distributions in test-time adaptation

We sincerely appreciate the reviewer for suggesting the visualization of estimated class distributions, which can help readers gain a deeper understanding of our method’s implementation process.

We conducted experiments on the ImageNet validation set to evaluate the mean cosine similarity between class embeddings modeled by DOTA at intervals of 5000 samples. The experimental results are presented in the table below:

Analysis of Experimental Results:

1. After updating with 5000 samples, the cosine similarity of DOTA’s class embeddings drops to a relatively low value (0.3025), indicating that our method can effectively model each class accurately with few samples, quickly adapting to downstream tasks.
2. As the number of updated samples increases, the cosine similarity continues to decrease steadily, demonstrating that our method can sustain learning in scenarios with abundant samples, further improving the modeling distribution quality for each class.

6. Considerations for the weighting mechanism of zero-shot and test-Time adapted classifiers

We sincerely acknowledge the reviewer for highlighting the lack of a principled basis in our fusion strategy weight control. Our dynamic fusion weight method, inspired by practical experience, demonstrates heuristic yet effective performance, as evidenced by the experimental results in Tables 1–3 of the paper. Dota employs an adaptive fusion strategy (Section 3.3) that dynamically combines predictions from both classifiers based on the number of test samples, balancing the stability of zero-shot predictions with the accuracy of test-time adaptation.

The current weighting strategy can be enhanced by incorporating real-time uncertainty estimation of online samples to dynamically adjust the fusion weights. Such sample-level adaptive weighting could better balance the capabilities of zero-shot and test-time classifiers, leading to improved performance.

Question 1. Mismatch Between Shapiro-Wilk Test and Multivariate Normality Assumption

We appreciate the opportunity to engage in this discussion and thank you for your insightful feedback. In response, we offer a comprehensive justification for our assumption that CLIP embeddings for a given class follow a multivariate Gaussian distribution from the following four perspectives: a discussion about marginal normality, two independent lines of theoretical support, and further empirical validation.

Perspective 1: Disscusion the Normality Assumption: Marginal vs. Multivariate

We agree with the reviewer on the critical distinction between marginal and multivariate normality. Our use of the per-dimension Shapiro-Wilk test served as a necessary falsification test. Because multivariate normality requires each dimension to be marginally normal, demonstrating that our embeddings pass this test establishes a critical precondition, providing initial evidence that our assumption is statistically plausible.

Perspective 2: Theoretical Support from CLIP's Formulation

The CLIP learning process is equivalent to an optimal linear projection[1]. A foundational property of Gaussian distributions is their closure under linear transformation. Therefore, if the input features for a class are modeled as a Gaussian (as shown in [1]), the resulting embeddings will necessarily follow a Gaussian distribution.

Perspective 3: Theoretical Support from Central Limit Theorem

An independent and more general justification comes from the Central Limit Theorem (CLT). A high-dimensional embedding is an aggregation of numerous lower- and mid-level features (e.g., edges, textures). The CLT posits that the sum of many weakly correlated random variables will tend toward a normal distribution. Consequently, it is reasonable to expect that the feature for a single class, being a composite of countless such signals, will naturally approach a multivariate Gaussian distribution.

Perspective 4: Further Empirical Validation and the Principle of Utility

To validate our core hypothesis, we evaluated the multivariate normality of CLIP features on the ImageNet dataset. Following the methodology proposed by Mardia et al. [2], we conducted the test by calculating the squared Mahalanobis distance, $(x - \mu)^T \Sigma^{-1} (x - \mu)$.

Key Finding: For the 512-dimensional features, the experimental mean squared Mahalanobis distance was $396.02$. Considering the complexity of the data and the fitting process, this value is relatively close to the theoretical expectation of $512$. The result demonstrates that the distribution of these class features is largely consistent with the theoretical multivariate Gaussian model.

More discussion

However, we acknowledge that for high-dimensional data, a perfect statistical model is impractical. Therefore, we adopt a simplifying assumption. The guiding principle is practical utility, not absolute correctness. We choose a multivariate Gaussian distribution because it is:

1. Theoretically Sound: It is a rational choice supported by theories like the Central Limit Theorem.

2. Proven in Practice: Models built on this assumption achieve strong downstream performance, which in turn validates its usefulness.

This embodies the principle that all models are wrong, but some are useful. Therefore, while imperfect, the Gaussian assumption is the most rational and effective choice for this task.

Question 2. Response to Comment on CLIP Calibration Metrics

We provide comparative data on the Expected Calibration Error (ECE) metric to substantiate CLIP’s superior calibration performance. While the referenced paper [3] visually illustrates CLIP’s robust calibration without tabular comparisons, we draw on data from [4] and [5] to present the following comparative table of ECE metrics on ImageNet dataset for different models:

1. In the table, “TS” refers to Temperature Scaling, and “VS” refers to Vector Scaling. These are calibration strategies applied to DenseNet and ResNet to reduce Expected Calibration Error (ECE), whereas CLIP adopts no such strategies.
2. Experimental data were obtained on the ImageNet dataset.

Results: The table demonstrates that CLIP’s ECE is significantly lower than that of DenseNet and ResNet. Even after applying scaling strategies to reduce ECE for the baseline models, CLIP’s ECE remains lower, confirming its superior calibration performance.

[1] Understanding Contrastive Learning via Gaussian Mixture Models

[2] Multivariate Normality Tests Using Mahalanobis Distance.

[3] Revisiting the calibration of modern neural networks

[4] Open-Vocabulary Calibration for Fine-tuned CLIP

[5] On Calibration of Modern Neural Networks

Dear Reviewer,

Thank you for your valuable feedback. In our author response, we have provided a detailed response to your concerns and questions.

We hope our response addresses your concerns and would be happy to discuss any follow-up questions you may have.

Thank you again for your time and effort!

1. Statistical significance for results in tables 1–3

We sincerely acknowledge the reviewer for suggesting the inclusion of statistical significance results, such as standard deviation or confidence intervals, to strengthen the evaluation of our results in Tables 1–3. To address your suggestion, we conducted additional experiments with five random seeds (1, 2, 3, 4, 5) across the cross-domain generalization (Tables 1), NDS (Table 2), and medical scenarios (Table 3). The supplementary results include baseline performance, original results (seed=1), mean accuracy across seeds, sample standard deviation, and 95% confidence intervals for each dataset.

Table 1

Table 2

Table 3

Experiment Results Analysis: The standard deviation remains below 0.25 for most datasets, demonstrating Dota’s robust and stable performance across different random seeds. This stability arises because Dota avoids random initialization or backpropagation-based optimization, with seed variations only affecting the data stream order, which minimally impacts performance. We will incorporate these statistical results in the revised manuscript to enhance the reliability of our findings.

2. Recommendations for hyperparameter tuning in practice

In response to the reviewer’s request for clarification on the setting and tuning of online estimation hyperparameters ($\omega$, $\sigma^2$, $\eta$, $\rho$), we address this by detailing the initialization and tuning process for these hyperparameters, emphasizing their practical implementation and generalizability.

Hyperparameter Initialization

The hyperparameter $\omega$, used to initialize the distribution of each class in Dota, is set to $10^{-4}$ to ensure uniform and small initial values across classes, minimizing interference with subsequent EM algorithm updates. In practice, this value requires no further adjustment.

Tuning Strategy for $\sigma^2$, $\eta$, and $\rho$

We recommend first conducting preliminary tuning for $\sigma^2$, followed by a grid search to optimize $\eta$ and $\rho$. Given Dota’s strong generalization across datasets, hyperparameters tuned on one dataset, such as ImageNet validation, can be directly applied to similar datasets like ImageNet-C, ImageNet-A, and eta. Owing to Dota’s lightweight and efficient design, this tuning process incurs low computational cost and is straightforward to implement.

3. Limitations of proposition 3.1 for a single EM iteration

Thank you for highlighting the limitation of Proposition 3.1, which focuses on a single EM iteration using soft zero-shot outputs without analyzing repeated updates or deviations from Gaussianity. To address this, we provide experimental results for the Dota model under single, double, and triple EM iterations.

Analysis of Experimental Results

Single Iteration

Advantages: Compared to multiple iterations, a single iteration results in a more lightweight and faster model, making it suitable for rapid adaptation to downstream scenarios.

Disadvantages: This may lead to slight performance degradation.

Multiple Iterations

Advantages: Compared to a single iteration, multiple iterations enable Dota to better fit each class embedding distribution, improving classification performance.

Disadvantages: Multiple iterations require repeated updates to model parameters, with each additional iteration increasing computational time by approximately 10%. Moreover, as the number of iterations increases, performance improvements become marginal.

4. Limitations of the gaussian assumption in embedding distributions

We appreciate the reviewer for highlighting the need for a deeper discussion of the Gaussian assumption’s limitations. We address this by elaborating on the theoretical justification and empirical validation of our Gaussian assumption, while acknowledging potential limitations in certain datasets.

Theoretical Justification

Our assumption that CLIP embeddings follow a multivariate Gaussian distribution per class is supported by prior work. Chupin and Dahyot (2024) [1] demonstrate that a single Gaussian model achieves high accuracy in CLIP embedding spaces (e.g., CIFAR100: 89.0%), closely matching Gaussian Mixture Models (G=2: 89.7%), indicating that a single Gaussian effectively captures class feature clustering, driven by CLIP’s contrastive learning optimizing embeddings for intra-class proximity.

Ablation Study on Covariance Updates

Our ablation study (Table 7) further validates this assumption: omitting covariance updates significantly reduces performance (e.g., EuroSAT: 55.06% vs. 62.78%), demonstrating the effectiveness and importance of Gaussian modeling.

Limitations of the Gaussian Assumption

Although prior studies validate that CLIP embedding spaces for different classes can often be modeled by a single multivariate Gaussian distribution, performance may still be suboptimal on certain datasets. The single Gaussian assumption may not fully capture multimodal distributions, potentially contributing to suboptimal performance in datasets like Aircraft and Caltech101.

[1] Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces

Dear Reviewer,

Thank you for your valuable feedback. In our author response, we have provided a detailed response to your concerns and questions.

We hope our response addresses your concerns and would be happy to discuss any follow-up questions you may have.

Thank you again for your time and effort!

1. Considerations regarding the assumption of uniform class priors ($P(y=k) = 1/K$)

We sincerely acknowledge the reviewer for highlighting the limitation of assuming uniform class priors ($P(y=k) = 1/K$) in real-world test-time adaptation (TTA) scenarios with potential class imbalance. To address this concern, we have (1). clarified the prevalence and rationale of this assumption,(2). demonstrated Dota’s compatibility with non-uniform class priors, and (3). provided empirical evidence showcasing Dota’s robustness in handling imbalanced class data streams.

Prevalence and Rationale of Uniform Class Priors

Assuming uniform class priors is a common and reasonable practice in TTA, particularly when prior information is unavailable. Several studies explicitly adopt this assumption as a starting point, achieving effective model initialization in the absence of test-time prior information [1-5].

DOTA'S Compatibility with Non-Uniform Priors

Dota seamlessly integrates with non-uniform class priors when available in real-world scenarios. By incorporating class-specific weights into the probability computation, our method adapts to given prior distributions, demonstrating its flexibility and compatibility.

Robustness to Class-Imbalanced Data Streams

To evaluate Dota’s performance under class imbalance, we conducted experiments on the ImageNet validation dataset, simulating imbalanced data streams using a Dirichlet distribution with varying concentration parameters ($\alpha$) and dividing the dataset into 5 and 10 time slices. The results are shown in Tables 1 and 2 below:

Table 1

Table 2

Results show stable performance (average accuracy ~70.39–70.83%), with later slices often achieving higher accuracy. This indicates that class imbalance has a limited impact on overall performance.

We acknowledge that uniform priors may not fully capture real-world distributions. In the revised manuscript, we will expand Section 3.2 to discuss this limitation and explore adaptive prior estimation as a future direction to enhance robustness further.

[1] Bayesian Test-Time Adaptation for Vision-Language Models

[2] Improving CNN Classifiers by Estimating Test-Time Priors

[3] Continual Test-Time Domain Adaptation

[4] Posterior Adaptation With New Priors

[5] The Hitchhiker’s Guide to Prior-Shift Adaptation

2.Concerns regarding the rationale for the fusion of Eq. 7 (($\cos(x, w_k)/\tau + \lambda f_k(x)$)

Thank you for your valuable feedback regarding the rationale for the additive fusion strategy in Eq. 7 ($\cos(x, w_k)/\tau + \lambda f_k(x)$). To address this concern, we have elucidated the heuristic yet effective design of our logit-level fusion, underscored its widespread use in the test-time adaptation (TTA) literature, and outlined future directions to strengthen the theoretical and empirical foundations of the approach.

Rationale for Logit-Level Fusion: Heuristic yet Effective

The additive fusion in Eq. 7 ($\cos(x, w_k)/\tau + \lambda f_k(x)$) is designed to balance the robust prior knowledge encoded in the pre-trained vision-language model (e.g., CLIP) with dynamic adaptation to test-time distribution shifts. This logit-level combination is computationally efficient, aligns with the softmax-based classification framework, and leverages the complementary strengths of both classifiers, as the zero-shot component provides generalization while the test-time component refines predictions based on test data. We acknowledge that this fusion strategy is largely heuristic. However, this dynamic and incremental approach ensures minimal interference with model outputs in early stages while effectively refining predictions as Dota’s per-class modeling becomes more accurate in later stages, proving highly effective in practical scenarios.

Prevalence of Additive Fusion in TTA

Several studies collectively demonstrate that additive fusion at the logit level is a widely adopted and effective strategy in TTA [6-9]. This indicates that, despite its heuristic nature, our fusion strategy aligns with approaches commonly used in the field, particularly for vision-language models (e.g., CLIP), as it efficiently leverages pre-trained knowledge while incorporating test-time information.

Future Directions: Uncertainty and Ensemble Learning

The current logit-level fusion can be enhanced by incorporating real-time uncertainty estimation of online samples to dynamically adjust the fusion weights. Such sample-level adaptive weighting could better balance the capabilities of zero-shot and test-time classifiers, leading to improved performance. Additionally, leveraging ensemble learning principles, we can exploit the scalability of additive fusion to integrate multiple diverse classifiers, thereby combining their strengths for more robust outcomes.

[6] Efficient Test-Time Adaptation of Vision-Language Models

[7] Tip-Adapter: Training-Free Adaptation of CLIP for Few-Shot Classification

[8] Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations

[9] Bayesian Test-Time Adaptation for Vision-Language Models

3. Analysis of strategies for mitigating catastrophic forgetting in continual learning

We appreciate the reviewer for highlighting the need for a clearer explanation of how Dota mitigates catastrophic forgetting. To address this concern, we: (1) first provide a theoretical analysis of Dota’s distribution updates to elucidate its mechanism, and (2) subsequently present experimental evidence that demonstrates its superior performance compared to cache-based methods like TDA.

Theoretical Analysis

Dota employs the EM algorithm to iteratively update the mean and covariance of class-conditional distributions, integrating statistical characteristics from early test samples into global parameters (Section 3.2). Unlike cache-based methods like TDA, which discard previously stored sample information during updates, Dota’s cumulative parameter updates preserve comprehensive statistical information, effectively reducing catastrophic forgetting.

Experimental Evidence

As shown in Figure 3 of the paper, Dota outperforms cache-based TDA in scenarios prone to catastrophic forgetting. The experimental results demonstrate that Dota progressively enhances model performance as the number of test samples increases. In contrast, TDA exhibits an initial improvement followed by a decline, as its cache update process discards representative samples, leading to performance degradation.

Response to Question about catastrophic forgetting.

We begin by revisiting the classic definition of catastrophic forgetting and exploring its manifestation in the specific scenario of TTA. Building on this, we design new experiments to demonstrate that existing cache-based methods, such as TDA, indeed suffer from forgetting previously acquired knowledge from historical samples.

A Closer Look at Catastrophic Forgetting

Catastrophic forgetting, by its classic definition, occurs when a model loses previously learned "old knowledge" while acquiring "new knowledge."[2] Traditionally, "old" and "new" knowledge correspond to discrete "tasks" or data distributions with clear boundaries.

Sample-level Catastrophic forgetting

However, during test-time, the model may processes a continuous stream of data with a gradually shifting distribution, causing these task boundaries to blur. Consequently, the unit of forgetting transitions from the macro "task" level to the micro "sample" level.

This issue is particularly pronounced in cache-based methods like TDA. Their fixed-size cache enforces a "one-in, one-out" policy. This means to adapt to a new sample (e.g., the 101st), the model must discard the data from an old sample (e.g., the 1st). This act is the essence of sample-level forgetting, leading to a permanent loss of knowledge associated with the discarded sample.

Further Experimental Analysis and Key Findings

To rigorously validate the issue of catastrophic forgetting in existing methods, we further designed two experiments based on Retrospective Testing and reinterpreted Figure 3. The core idea is to have the model, after a period of continuous adaptation, revisit and re-evaluate its performance on historical test samples. Our experiments revealed two levels of forgetting:

1. Forgetting without Significant Distribution Change: To assess this phenomenon, we designed a two-stage experiment. This process first involves adapting a model on ImageNet and recording its performance (Stage 1), after which we freeze the model's parameters and re-evaluate it on the early 25,000 test samples in the dataset (Stage 2).We hypothesize that if no forgetting occurs, this initial adaptation stage should directly lead to an increase in the model's performance. Conversely, a performance decrease between stage 1 and stage 2 would suggest that the model has experienced forgetting. Experimental results show that TDA suffers from obvious performance degradation, while our method does not.

2. Domain-Level Forgetting: We challenged the model by adapting it first to an original domain (ImageNet), then to a new, shifted domain (ImageNet-C), and finally re-evaluating its performance on the original domain. TDA's performance failed to improve upon returning to ImageNet, indicating it had forgotten the original domain's features. DOTA, however, showed improvement, demonstrating it retained this knowledge.

3. We reinterpreted Figure 3 and added additional experiments. We found that catastrophic forgetting degrades a model's generalization capabilities by eroding previously learned information. More strikingly, we found that under certain conditions, continued adaptation is actively counterproductive, yielding worse results than if adaption had stopped. This is because as we continue to adapt, we forget more than we learn.

The table below illustrates this clearly. The standard TDA model's performance peaked after 20,000 samples and then degraded. A model whose training was frozen at that point (TDA frozen) consistently outperformed the continuously trained model thereafter. Our proposed method, DOTA, effectively mitigates this degradation and delivers robustly superior performance.

[2] Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

We are glad to receive your response and thank you for the opportunity for further discussion.

1. Introducing SAR as a baseline

Regarding the 'SAR' method mentioned in your reply, we have proceeded with the assumption that it refers to the one described in previous paper[1]. Our discussion below is based on this premise. Please correct us if we are mistaken.

While SAR (Sharpness-Aware and Reliable Entropy Minimization) is effective for improving the stability of standard classification models. Adapting the SAR directly to CLIP is a non-trivial modification. The primary difficulties are as follows:

1. Architectural Incompatibility with CLIP's Dual-Encoder

Original SAR's optimization requires a complete gradient calculated across all of a model's parameters. However, CLIP's dual-encoder architecture splits parameters between two independent modules: an image encoder ($f_{\theta_I}$) and a text encoder ($f_{\theta_T}$). Applying SAR to just the image encoder would yield only a partial gradient ($\nabla_{\theta_I}L$), ignoring the text encoder's influence.

2. Fine-tuning CLIP's image encoder via SAR may affect its native zero-shot classification capability.

The fine-tuning process modifies only the image encoder, while the text encoder remains static. This disrupts the alignment between image and text features in their shared embedding space. As a result, the image features are mapped to a new distribution that no longer aligns with the original text embeddings. This may prevent the model from fully leveraging the performance of the SAR method within the CLIP architecture.

3. Experimental results

More Baselines: Although SAR is not fully compatible with CLIP, we have included its experimental results for a more complete analysis. The results confirm that its performance is indeed suboptimal. For further comparison, we also present the results of TDA, which serve to highlight the effectiveness of our own method.

Results Analysis: Compared to the SAR method, we observe that DOTA exhibits comparable robustness to imbalanced data streams, while demonstrating superior classification performance. In comparison with the TDA method, DOTA exhibits stronger robustness to imbalanced data streams (with a performance difference of only 0.29% between α=0.1 and α=1, whereas TDA shows a difference of up to 1.04%). Furthermore, we believe that appropriately modifying and applying methods like SAR to pre-trained models such as CLIP is an interesting direction, which we leave for future work.

[1] Towards Stable Test-Time Adaptation in Dynamic Wild World

Dear Reviewer,

Thank you for your valuable feedback. In our author response, we have provided a detailed response to your concerns and questions.

We hope our response addresses your concerns and would be happy to discuss any follow-up questions you may have.

Thank you again for your time and effort!