We thank the reviewer for taking the time to review our work. We appreciate your positive feedback on the quality and clarity of our paper. Below, we provide detailed responses to your valuable comments.

Q1: Impact of single and batch adaptation for accumulation

Thank you for your insightful question. We agree that adapting on a batch introduces more information that accelerate the accumulation of feature statistics. Specifically, when adapting on a single image, the accumulation of feature statistics is indeed slower, as each test step incorporates only one sample. In contrast, batch adaptation allows the model to leverage multiple samples simultaneously, enabling faster accumulation. For fair comparison, we follow the setting in previous works that compare in single adaptation (batch size = 1).

W2 & Q2: Performance on corrupted datasets

We thank the reviewer for this excellent suggestion to broaden our empirical evaluation. Our initial experiments focused on the challenging OOD and cross-domain benchmarks standard in recent VLM-TTA literature. However, we agree that evaluating robustness against common corruptions provides a more complete and compelling picture of our method's capabilities.

Following this valuable advice, we have conducted additional experiments on CIFAR-10-C, CIFAR-100-C, ImageNet-C, and VisDA-C, using the same CLIP ViT-B/16 backbone. We compare our method, SCA, against the strong state-of-the-art baseline, DPE.

As shown in table, SCA is effective against both domain shifts and common corruptions, broadening its applicability.

Q3: Ablation study on other datasets

Thank you for this suggestion. In our paper, our ablation studies were primarily conducted on the cross-domain benchmark for faster validation. Here, we provide the results of ablation study on large scale ImageNet:

As shown in table, we have several observations: (1) our full method (71.75%) consistently outperforms all variants of the feature caching baseline (w/o statistics accumulation). This demonstrates that our approach of accumulating knowledge from all samples into compact statistics is fundamentally more effective than maintaining a limited-size cache of discrete features, even on large-scale datasets. (2) Dynamic soft label assignment plays a great role in performance gain. This underscores the critical importance of this component, especially on a challenging, large-scale dataset with 1000 classes where the risk of generating incorrect hard pseudo-labels is extremely high. This confirms that our uncertainty-aware labeling is essential for robust adaptation.

W1: Comparison with WATT and TENT

We prvoid results of WATT and TENT on cross-domain benchmark using CLIP ViT-B/16. We will prvoide results of WATT and TENT using different backbone and datasets in our final version.

W3: Ambiguous notation

Thank you for pointing this out. We will fix it in the final version for clarity.

Thank you for your thoughtful review. If there are any concerns or if further clarification is needed, we are more than happy to provide additional details or revise accordingly.

We would like to thank the reviewer for taking the time to review our work. We appreciate your recognition of the originality of our paper. According to your valuable comments, we provide detailed feedback.

W1: Motivation

Thank you for this constructive feedback. Firstly, we briefly introduce the mechanism of existing cache-based methods.

Existing cache-based methods, like TDA, maintain a fixed-size feature cache for each class during test time. The process works as follows:

1. Cache Update Mechanism: For each incoming test sample, the model first generates a pseudo-label (predicted class) and a corresponding confidence score (typically measured by prediction entropy). It then attempts to update the cache associated with the predicted class:

• The model locates the specific cache for the predicted class.
• If the cache for the predicted class is not full: The sample's feature is added to it.
• If the cache for the predicted class is full: The new sample's confidence is compared to the confidence of the features already stored in it. If the new sample is more confident than the least confident feature in the cache for predicted class, its feature replaces that one. Otherwise, the new sample is discarded.

2. Prediction: The final prediction for a new sample is a combination of the model's original zero-shot prediction and a new prediction derived from comparing the sample's feature similarity to the features stored across all class-specific caches.

Then we provoide a analysis to address your specific points:

What types of features are forgotten during adaptation. To be precise, the forgetting issue we address in this paper refers to a correct but relatively low-confidence (high-entropy) feature being overwritten by an incorrect but high-confidence (low-entropy) one. For example, on the DTD dataset, a canonical "striped" texture feature might be overwritten by a misclassified "lined" texture that the model confidently mistakes for "striped."

How it Affects Performance: the model's predictions rely on comparing new samples to these cached features. When a correct cached feature is replaced by an incorrect one, the model's ability to correctly match samples is impaired, leading to misclassifications and a performance drop, as shown in Figure 2(a). Our SCA method, by accumulating statistics from all samples rather than maintaining a fixed-size "winner-takes-all" cache, is immune to this specific type of forgetting.

The Role of Low and High-entropy samples:

• Correct low-entropy (high-confidence) samples serve as strong positive examples. They reinforce the quality of class representations in the cache, leading to more accurate future predictions.
• Incorrect low-entropy samples may replace correct ones in the cache, causing forgetting. Because of their high confidence, these incorrect samples can hold cache slots for a long time, blocking later higher-entropy but correct samples.
• High-entropy (low-confidence) samples typically fail to enter the cache, which leads to the loss of potentially useful, though uncertain, information. Our SCA enables these potential samples to contribute to the accumulated statistics in a controlled way through dynamic soft label assignment.

In the original version, we included more details on the forgetting and blocking issues in the Supplementary Material (Appendix A) due to page limits. In the final version, we will move this content into the main paper and add more visualizations, such as the performance curves of our method in Figure 2(a), to make the motivation clearer.

W2: Rationality of our accumulation scheme

Thank you for this excellent question, which allows us to elaborate on the theoretical motivation behind our approach.

1. Theoretical Justification for the Least Squares Formulation:

Our core idea is to move beyond simple feature matching and instead learn a linear classifier ($\boldsymbol {W} _{t}$) directly in the feature space, which adapts over time. The least squares formulation

$\min_{\boldsymbol{W} _t} \|\| \boldsymbol{X} _{1:t} \boldsymbol{W} _{t} - \boldsymbol{Y} _{1:t} \|\| _\mathrm{F}^2 + \gamma \|\|\boldsymbol{W} _{t} \|\| _\mathrm{F}^2$

is the classic way to achieve this for classification. The closed-form soultion is

$\boldsymbol{W} _t=(\boldsymbol{X} _{1:t}^\top\boldsymbol{X} _{1:t}+\gamma\boldsymbol{I})^{-1}\boldsymbol{X} _{1:t}^\top\boldsymbol{Y} _{1:t}.$

As this solution shows, to compute the optimal classifier $\boldsymbol {W} _t$, we do not need to store the entire, ever-growing feature matrix $\boldsymbol{X} _{1:t}\in \mathbb{R}^{n _{1:t} \times d}$ ($n _{1:t}$ denotes the number of features extracted from all test samples up to time $t$, $d$ denotes the feature dimension). We only need store two fixed size statistics :

• $\boldsymbol{G} _{1:t}=\boldsymbol{X} _{1:t}^\top \boldsymbol{X} _{1:t}=\boldsymbol{G} _{1:t-1}+\boldsymbol{X} _t^\top\boldsymbol{X}_t$ (the Gram matrix): This captures the second-order statistics of the features. It encodes the covariance structure between all feature dimensions.
• $\boldsymbol{C} _{1:t}=\boldsymbol{X} _{1:t}^\top \boldsymbol{Y} _{1:t}=\boldsymbol{C} _{1:t-1}+\boldsymbol{X} _t^\top\boldsymbol{Y} _t$: This captures the first-order statistics relating features to their class labels.

Therefore, formulating the problem as least squares naturally leads to a highly compact and efficient online learning scheme. We are not arbitrarily "compacting features." Instead, we are storing the essential sufficient statistics necessary to train a robust linear classifier using all the data encountered so far. This provides a strong theoretical grounding for our statistics caching approach.

2. Comparison with Class Prototype Methods:

Class prototype methods only store and update the mean feature vector for each class. This is a first-order statistics approach. It implicitly assumes that classes form simple, spherical clusters in the feature space and only captures the center of each class. It's conceptually similar to a nearest-neighbor classifier.

Our SCA additionally uses second-order statistics. This means our model understands not just the center of each class's data cloud, but also its shape, orientation, and variance. This allows the learned classifier to create much more sophisticated decision boundaries. We also provide the results of class prototypes methods on cross-domain benchmark using ViT-B/16:

As shown in the table above, our method significantly outperforms Class Prototype methods.

W3: Robustness of weighting strategies for prediction fusion

Thanks for your suggetsion. We provide the average accuracy of our SCA on cross-domain benchmark using different weighting strategies, including softmax probability [1] and mahalanobis distance [2] :

As shown in the table, our SCA method is compatible with various weighting strategies and show robust performance. We selected entropy due to its simplicity and widespread adoption in the literature.

[1] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks, ICLR'17
[2] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS'18.

W4: Prompt Initialization

We follow the existing cache-based methods that use prompt ensembling strategy to initialize the prompt. This practice is widely adopted in the literature to ensure a robust and fixed zero-shot classifier, which serves as the foundation for these feature-space adaptation methods.

For prompt-tuning based methods, the use of a single prompt for methods like TPT and DiffTPT is also the standard protocol, as their focus is on demonstrating adaptation of the prompt itself.

Additionally, we present results using high-quality prompts obtained through training as the enhanced prompt initialization for prompt-tuning based TTA [1]. Specifically, we use the prompts trained by MaPLe [2] as the initialization, a widely adopted approach in prompt-tuning based TTA. As shown in the table below, the performance of prompt-tuning based methods is significantly lower than that of cache-based methods.

[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS'22
[2] Maple: Multi-modal prompt learning, CVPR'23

W5: Feature visualizations

Thank you for this suggestion. Existing cache-based methods store the features of specific samples, allowing for the use of visualization techniques, such as t-SNE, to directly assess cache quality. In contrast, our SCA method does not store the features themselves but rather feature statistics that summarize information from all past samples, making such visualization methods less applicable. In the final version, we will include performance curve visualizations to compare the impact of the features or feature statistics stored by different methods on adaptation.

Thank you for your thoughtful review. If there are any concerns or if further clarification is needed, we are more than happy to provide additional details accordingly.

We sincerely thank the reviewer for their constructive and insightful feedback. The comments help us to significantly improve the clarity and rigor of our paper. We address the concerns below.

W1 (part 1): On the significance of the forgetting and blocking issues

Thank you for raising this important point about the significance of forgetting and blocking issues. We clarify that these are indeed practically significant because they stem from a fundamental algorithmic vulnerability.

This vulnerability is inherent in any cache-based method that combines a fixed-size cache with a confidence-based replacement strategy. For any given dataset, a population of both low-entropy, incorrect samples and high-entropy, correct samples typicall always exist. Every time a low-entropy, incorrect sample arrives, there is a chance it will displace a correct feature from the cache, resulting in forgetting. And when a highly confident incorrect sample arrives early, blocking issues occurs.

In a real-world online setting, we cannot control or guarantee the arrival order of these samples. This makes the system's performance highly susceptible to "unlucky" sequences that trigger failure modes. Much like a security flaw in software, this design represents a persistent threat. While it may not be exploited in every single instance, its potential to cause performance degradation when triggered.

W1 (part 2): On the synergy between statistics accumulation and dynamic soft labeling

Thank you for this insightful question. We respectfully argue that our dynamic soft labeling technique is not a simple "plug-and-play" module that can be easily combined with DPE for similar gains. DPE, like other cache-based methods, maintains a cache of discrete, raw feature vectors in distinct, class-specific slots. This discrete storage design is ill-suited for a soft pseudo-label like {'cat': 0.7, 'dog': 0.3}, as it raises following challenges:

• How should the feature be stored? Should it be placed in the 'cat' cache with a weight of 0.7 and the 'dog' cache with a weight of 0.3? This would require storing an additional confidence weight with each feature, complicating the cache structure.
• How would this affect cache occupancy? If one sample now occupies slots in multiple class caches, does it count towards the size limit $M$ for each of them? This could lead to a single, highly uncertain sample disproportionately blocking space for many other more confident samples.
• How would the replacement mechanism work? When considering replacing a feature in the 'cat' cache, should its confidence be its standalone prediction confidence, or should it be discounted by its soft label weight (0.7)? If a new, confident 'cat' sample arrives, should it be allowed to replace the 'cat' portion of our soft-labeled sample? Doing so would break the integrity of the original soft label, leaving a partial entry in the 'dog' cache that no longer reflects the full uncertainty of the original prediction.

In contrast, in our SCA framework, dynamic soft label assignment is highly compatible with statistics accumulation. More specifically, they form a synergistic relationship:

• Statistics accumulation unlocks the full potential of soft labeling: The update rule for our first-order statistics, $\boldsymbol{C} _{1:t-1}+\boldsymbol{X} _t^\top\boldsymbol{Y} _t$ naturally handles a soft probability vector $\boldsymbol{Y} _t$. The contribution of each feature to the class correlations is weighted by its soft probability, which is mathematically elegant and effective.
• Soft labeling enables robust statistics accumulation: Accumulating knowledge from every single sample is powerful but risky. If we were to use hard labels, errors would quickly accumulate and corrupt our statistics. Our dynamic soft labeling strategy acts as an uncertainty-aware regularizer to make our statistics accumulation strategy robust.

In conclusion, our dynamic soft labeling is not a simple plug-and-play module. It is tightly integrated with our statistics accumulation framework, with each part supporting the other. This synergy is key to our strong performance.

W2: Negative aspects of accumulation in multiple domain settings

Thank you for your insightful comment. Consistent with existing CLIP TTA methods, our approach primarily focuses on the single-domain setting, where test data come from a single dataset. Extending CLIP TTA methods to the multi-domain setting, where test samples from different domains are mixed, is a valuable and important direction. However, to the best of our knowledge, there are currently no TTA methods for VLMs that explicitly claim to address this scenario.

Indeed, in the multi-domain setting, accumulation in our scheme may introduce negative effects. As a potential solution, we note that feature representations from different domains often exhibit some degree of difference. This property can be used to identify a sample's domain and accumulate feature statistics separately for each domain. We consider this a promising direction for future work and will add a discussion of this limitation and potential extensions in the final version.

W3: Hyperparameters selection

We select hyperparameters based on the validation set of DTD, a relatively challenging dataset (with CLIP ViT zero-shot accuracy of 44.27). Additionally, DTD has a small number of test samples, which allows for efficient validation. That's why DTD appears frequently across multiple tables.

The main text omits the hyperparameter analysis of the ridge coefficient $\gamma$. Here, we present results on a cross-domain benchmark using ViT-B/16 by varying $\gamma$ from 0 to 1e6. As shown in the table below, setting $\gamma$ to zero leads to numerical instability due to a singular Gram matrix. Within the range from 1e2 to 1e6, performance remains stable, indicating that any value within this interval is appropriate.

L1: limitation section

Thanks for your suggestion. We will move it from suplementary material to the main text in the final version.

We greatly appreciate your insightful review and are open to discussing any remaining issues. Please feel free to contact us if additional changes or clarifications are needed.

Thank you for this insightful comment. Here, we provide the detailed response.

W1 (part 1)

1.Forgetting issues

To better illustrate the forgetting issues, we conduct an additional set of control experiments.

Firstly, we identify a small set of "forgetting trigger samples." For each dataset, we select 10 random classes and find the single, lowest-entropy misclassified sample for each class. To ensure these 10 trigger samples are admitted by the cache, we filter out any correctly classified samples from the entire test set that have an even lower entropy.

From the remaining pool of test samples, we randomly draw 500 samples to serve as our fixed "final test samples." All other remaining samples are designated as "initial samples," used to allow the cache to accumulate sufficient class-specific knowledge. Next, we consider the following two sequences to update the model's cache:

• Sequence A: [initial samples]
• Sequence B: [initial samples, forgetting trigger samples]

The forgetting trigger samples, with their low entropy, are designed to infiltrate the cache built from the initial samples, causing it to forget previously learned information. In other words, the cache updated with Sequence A represents the state before forgetting, while the cache updated with Sequence B represents the state after forgetting. To quantify the impact of forgetting, we measured the performance difference between Sequence A and Sequence B on the final test samples. The performance drop is calculated as (Accuracy of A - Accuracy of B). The results are shown in the table below:

We have several observations: (1) The SOTA baseline DPE is highly vulnerable to forgetting. (2) SCA is inherently robust to forgetting, as its accumulation design prevents error samples from fully overwriting the stored knowledge. (3) Remarkably, SCA even leverages these "forgetting triggers" to improve performance (a negative drop), turning a vulnerability into a strength.

2.Blocking issues

We present additional results about the blocking issues through simulating "bad cases": we randomly selected 10 classes and identified the single lowest-entropy misclassified sample for each class, placing them at the beginning of the test sample sequence. As shown in the table below, the impact of blocking issues on SCA's performance is much smaller that DPE. This is because SCA's statistics-based accumulation is not limited by fixed cache slots, preventing initial high-confidence errors from occupying resources and hindering learning from subsequent samples.

Finally, we want to clarify the practical relevance of these controlled experiments. These forgetting and blocking issues are not artifacts of our experimental design. On the contrary, they are constantly occurring at a micro-level in any test sequence. However, the negative impact of bad cache updates (forgetting and blocking issues) may be obscured by the effects of subsequent good updates. This makes it hard to directly observe or quantify the harm of these vulnerabilities in a standard evaluation. The purpose of our controlled experiments is to isolate these specific variables. By altering the sequence of just a small fraction of the data, we create a scenario where the harm from forgetting and blocking can be directly and intuitively measured.

We will update the corresponding content (e.g., Figure 2) in the original paper based on the existing content and include additional visualizations in the final version.

We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper well-motivated. According to your valuable comments, we provide detailed feedback.

W1 & Q2 & Q6: Performance on OOD benchmark

Thank you for your insightful question regarding the performance comparison with the SOTA baseline DPE [1] on the OOD benchmark using a ResNet-50 backbone. Here we provide the explanation:

Our proposed SCA is an entirely training-free approach. It operates by accumulating feature statistics (first and second-order moments) from all past samples and constructs a classifier via a closed-form solution (as detailed in Eq. 6, 7, and 8). The performance of this design is therefore directly and heavily dependent on the intrinsic quality and linear separability of the features provided by the backbone network.

The OOD benchmark is a particularly demanding scenario; its large number of classes and fine-grained stylistic shifts place extreme requirements on the discriminative power of the features. That's why performance drop occurs between Cross-Domain benchmark and OOD benchmark (Q6: Performance drop between Table 1 and Table 2).

While ResNet is a powerful CNN architecture, its features may be less semantically rich or linearly separable compared to those from ViT backbones. In this context, DPE's additional training phase can effectively compensate for minor imperfections in the ResNet-50 feature space, allowing it to find a slightly better decision boundary and thus achieve a marginally higher accuracy. Our purely statistical SCA is more constrained by the inherent quality of these features.

[1] Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models, NeurIPS'24

W2 & Q5: Ablation on ImageNet

Thank you for this suggestion. In our paper, our ablation studies were primarily conducted on the cross-domain benchmark for faster validation. Here, we provide the results of ablation study on ImageNet:

As shown in table, we have several observations: (1) our full method (71.75%) consistently outperforms all variants of the feature caching baseline (w/o statistics accumulation). This demonstrates that our approach of accumulating knowledge from all samples into compact statistics is fundamentally more effective than maintaining a limited-size cache of discrete features, even on large-scale datasets. (2) Dynamic soft label assignment plays a great role in performance gain. This underscores the critical importance of this component, especially on a challenging, large-scale dataset with 1000 classes where the risk of generating incorrect hard pseudo-labels is extremely high. This confirms that our uncertainty-aware labeling is essential for robust adaptation.

W3 & Q3: Efficiency comparison (whether data augmentation cost is included)

Thank you for this important question regarding the efficiency comparison.

The numbers reported in Table 3 exclude data augmentation, as none of the methods use it on the cross-domain benchmark. This is a standard evaluation protocol followed by prior work in this area, including the baselines we compare against like DPE and TDA. Therefore, the reported runtimes reflect the full processing time for each method under this no-augmentation setting.

Q1: Varying $|\Omega|$ and its impact on pseudo-label sharpness

Thanks for your insightful question. The core logic behind this claim is tied to how the candidate set $\Omega$ is formed based on the initial CLIP prediction probabilities ($p_\mathrm{text}$) and the threshold $\tau(1-p_\mathrm{text}^{(1)})$. Here $p_\mathrm{text}^{(1)}$ is the value of the top-ranked class in $p_\mathrm{text}$.

• For a confident sample: The initial prediction $p_\mathrm{text}$ may exhibit a very high probability for one class ($p_\mathrm{text}^{(1)}$). According to Eq. (9), the candidate set $\Omega$ will likely contain only a small number of classes (small $|\Omega|$) that exceed the threshold $\tau(1-p_\mathrm{text}^{(1)})$. Upon renormalization in Eq. (10), the resulting pseudo-label $\hat{Y}_t$ becomes a near one-hot vector, making it sharp.
• For an uncertain sample: The initial prediction $p_\mathrm{text}$ shows probabilities spread across multiple classes, none of which is individually large. To meet the cumulative probability threshold $\tau(1 - p_\mathrm{text}^{(1)})$, Eq. (9) includes multiple classes in the candidate set $\Omega$ (resulting in a large $|\Omega|$). As a result, the final pseudo-label $\hat{Y}_t$ distributes its probability mass across these candidates, making it soft.

Compared to using hard labels, which completely ignore uncertainty information, our dynamic threshold provides a way to adapt the label's sharpness based on the model's own confidence.

Q4: Hyperparameters selection

We select hyperparameters based on the validation set of DTD, a relatively challenging dataset (with CLIP ViT zero-shot accuracy of 44.27). Additionally, DTD has a small number of test samples, which allows for efficient validation. That's why DTD appears frequently across multiple tables.

Q7: Suggestion on highlighting "whether need training" in Table 3

Thanks for your insightful suggestion, which helps make our paper clearer. In the final version, we will add a "Requires Training" column to Table 3 to highlight that our method, unlike DPE, is training-free.

We deeply appreciate your efforts and would be happy to further clarify or discuss any remaining concerns. Please feel free to let us know if additional modifications or explanations are required.