We thank the reviewer for their thorough analysis and constructive feedback.

---

W1: Discussion on the assumption of Gaussian distributions with shared covariance.

We acknowledge that assuming a unimodal Gaussian distribution with a shared covariance may not capture the full complexity of all hypothetical, multi-modal data structures. For instance, the performance on the ImageNet-C corruption benchmark is significantly lower compared to clean ImageNet. However, as we discuss in Appendix A.4 (Limitations and Discussion), this assumption was a deliberate design choice made to prioritize practicality and efficiency, and it is supported by strong empirical evidence.

Furthermore, as we show later in Q3 of our rebuttal, a more complex model does not inherently yield better results. We found that adopting class-specific separate covariance matrices actually degrades performance, while significantly increasing memory usage and inference time. This suggests that our simpler assumption strikes a crucial and effective trade-off between model fidelity, robustness, and computational efficiency. This balance is precisely what enables ADAPT to be, as the reviewer insightfully highlighted, "a compelling solution for real-time or resource-constrained scenarios, a significant advantage over many existing TTA methods."

---

W2: More comparisons with prior works.

We extend the comparisons to include most of the works mentioned by the reviewer, excluding DPCore [1] and FOA [4], which require source data and thus violate ADAPT's source-free protocol. Instead, recent SOTA training-free methods such as AWT [6] and RA-TTA [7] are included. A more detailed comparison and discussion will be provided in the final version.

References:

[1] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML 2025)

[2] Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS 2021)

[3] DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025)

[4] Test-Time Model Adaptation with Only Forward Passes (ICML 2024)

[5] Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025)

[6] AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024)

[7] RA-TTA: Retrieval-Augmented Test-Time Adaptation for Vision-Language Models (ICLR 2025)

---

Q1: Clarification on the Oracle ADAPT.

As noted in Section 4.3, Oracle ADAPT uses the same setup as transductive ADAPT, but with one key difference: it leverages test set ground-truth labels to estimate the target distribution. While this is not feasible in real-world TTA scenarios, this setting serves as an upper bound for performance. In contrast, our standard ADAPT is fully unsupervised and label-free.

As shown in Table 4 of our main paper, our standard method achieves results within 5% of this oracle, showing that ADAPT can closely approximate ideal adaptation performance without supervision. This highlights the strength and practicality of our approach in real-world test-time adaptation settings.

---

Q2: Efficiency analysis.

We already provided efficiency analysis in Table 6 of our main paper. Compared with optimization-based methods (e.g., TPT, DiffTPT, and TPS) and training-free methods like TDA, ADAPT achieves the largest performance gains on ImageNet with just 1h 11m and 0.93GB in the online scenario.

Notably, our transductive ADAPT builds the knowledge bank once and predicts all samples at once, requiring only 0.73 minutes and 3.37GB. These results highlight ADAPT as a practical solution balancing accuracy and efficiency.

---

Q3: Comparison with separate covariance matrices.

We conducted an additional experiment on ImageNet by replacing our shared covariance matrix with class-specific covariance matrices, while keeping all other settings identical.

As shown in the table below, this variant leads to a noticeable drop in accuracy, along with substantial increases in both computation time and memory usage. We attribute this performance degradation to data sparsity. When only a few high-confidence samples are available for each class, estimating a full covariance matrix becomes unreliable. This can lead to overfitting and poor generalization.

In contrast, using a shared covariance matrix allows the model to pool information across all classes, yielding a more stable and robust estimate. This design is especially beneficial in online or data-scarce scenarios, where efficiency and generalization are crucial.

These results further validate our design choice. Our shared covariance not only enhances performance but also supports ADAPT's efficiency, low latency, and applicability across diverse settings, making it a practical solution for real-world deployment.

---

Q4: Why exclude the current test sample from class mean updates in the online but include all samples in the transductive settings?

We already analyzed and discussed this in the last paragraph of Section 3.2 (Remark) of our main paper, with further details provided in Appendix A.4 (Remark). We summarize the key points as follows.

In the online setting, we exclude the current test sample from class mean updates to enable a one-pass, iteration-free solution. High-confidence samples are automatically stored in the knowledge bank and used for future updates without additional processing. So, this design also avoids injecting noise from low- or mid-confidence predictions. As shown in Table 8 of our main paper, this strategy achieves comparable accuracy to the iterative version (with the current test sample) while achieving a 4× speedup on ImageNet in the online scenario.

In contrast, the transductive setting assumes access to the entire test set beforehand, allowing all samples to contribute to a more accurate estimate of the class distribution. To leverage global consistency without iterative optimization procedures such as Majorization-Minimization, we replace hard latent assignments with soft labels.

As demonstrated in Table 8 of our main paper, this choice leads to an effective balance between accuracy and efficiency.

---

Q5: On the robustness to high inter-class similarity and low-confidence scenarios.

Thank you for raising this question regarding challenging cases with high inter-class similarity and limited confident samples.

As demonstrated in the fine-grained categorization results in Table 4 of the main paper, where distinguishing between classes is particularly difficult, ADAPT achieves state-of-the-art results, highlighting its effectiveness even when classes are highly similar.

To further assess the impact of limited high-confidence samples, we vary the number of available samples per class (denoted as N) in this constrained setting. The results in the table below show that our method remains robust. Specifically, Online ADAPT performs reliably with as few as 4–6 confident samples per class, and Transductive ADAPT maintains competitive accuracy even when provided with only two confident samples per class.

We acknowledge, however, that extremely low confidence across all classes presents a challenge, especially for the efficiency of our ADAPT in the online scenario. As shown in Table 9 of our main paper, if the test stream begins with many low-confidence predictions, it takes longer to collect reliable samples for the memory bank, which may slightly delay adaptation and degrade performance.

---

Q6: Analysis of parameter $\beta$.

The parameter $\beta$ in Eq. 5 serves to map the scalar $\alpha$ into the range [0, 1], which is then applied in computing the class mean in Eq. 9 and Eq. 12. The value of $\alpha$ controls the balance between the CLIP prior and the empirical mean of confident test features. We already provided a detailed analysis of α in Section 4.3 and Figure 2 (right) of the main paper.

W2: Detailed comparisons with prior works.

Thank you for this follow-up question. We are glad to clarify in detail the key distinctions from prior work.

(1) Distinction from T3A [4]

ADAPT and T3A differ fundamentally in their modeling paradigm, which directly impacts their robustness and applicability.

Methodology (Distribution-aware vs. Prototype-based):

• T3A is a non-parametric method using pre-trained classifier weights as prototype anchors. It adjusts these prototypes by computing centroids from a support set formed from a batch of test data. This centroid-based approach is sensitive to outliers, especially for classes with very few confident samples.
• In contrast, ADAPT is a parametric, distribution-aware method. It models the class-conditional feature distribution ($\mu_k$ , $\Sigma$) from high-confidence historical samples. By representing each class with a global distribution rather than a simple centroid, the influence of any single outlier is naturally diminished.

Applicability (Online(bs=1) vs. Online(bs>1) ):

• T3A's reliance on batch-based centroid calculation makes it unsuitable for strict online scenarios (bs=1) where samples arrive sequentially.
• ADAPT is explicitly designed to support both transductive and online (bs≥1) settings.

(2) Distinctions from DPCore [3] and DynaPrompt [5]

While these methods also use similar knowledge bank mechanisms, they are fundamentally prompt-tuning methods. In contrast, ADAPT is entirely training-free and prompt-tuning-free.

Stored Elements (Learnable Prompts vs. Raw Samples):

• The "banks" in DPCore and DynaPrompt store and continuously update a set of learnable prompt vectors, introducing new parameters that must be optimized.
• ADAPT's knowledge bank is simpler: it stores the raw feature vectors of high-confidence historical test samples, requiring no training or new parameters.

Update Strategy:

• DPCore employs a batch-based, distance-weighted update, requiring access to a new batch of test data. This is incompatible with strict online (bs=1) settings.
• DynaPrompt relies on computationally expensive data augmentation for its prompt selection and uses prompt-based hard thresholds to manage its buffer.
• In contrast, our ADAPT (Eq. 9) is batch-free, augmentation-free, and threshold-free. It dynamically updates its knowledge bank based on the zero-shot prediction confidence of each sample, making it uniquely suited for true online adaptation.

In summary, ADAPT distinguishes itself by being a training-free, prompt-tuning-free, and distribution-aware method. Its unique mechanisms lead to high efficiency and broad applicability in both online (bs≥1) and transductive settings. To clearly and intuitively illustrate the key distinctions between ADAPT and prior work, we provide a comparative summary in the table below.

---

We hope this clarification resolves your concerns. If you have any further questions, we'd be glad to discuss them in more detail.

We sincerely appreciate the reviewer's thoughtful follow-up and their consideration of our previous explanations. We are pleased to provide our responses to the follow-up points below.

W1: On the Robustness to the Uniform Class Prior Assumption in Non-Stationary.

(1) Assumption on the Uniform Class Prior:

While a simplification, the uniform class prior is a common assumption in source-free TTA [1,2,3] where the true test distribution is unknown. Importantly, ADAPT is designed to mitigate this issue via its fixed-size per-class knowledge bank. This structure ensures class-conditional likelihoods are estimated from a balanced set of historical samples, providing inherent robustness to class imbalance. Our Q5 analysis further confirms this, showing reliable performance even with very few samples per class (e.g., 2-4).

References:

[1] A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024)

[2] Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024)

[3] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models (NeurIPS 2023)

(2) Empirical Validation in Imbalanced and Non-Stationary Scenarios:

We acknowledge that the performance of our source-free, training-free, and prompt-tuning-free approach can be challenged when the test distribution undergoes continuous, abrupt shifts composed of low-confidence or distribution-shifted samples, as this makes the accurate estimation of a rapidly changing distribution inherently difficult.

To position our work within the context of the literature the reviewer provided, we wish to highlight a key distinction between two families of TTA protocols. Our ADAPT aligns with one family [4, 5, 7], while the other [1, 2, 3] is built upon different experimental frameworks and evaluation goals. For this reason, a direct comparison with our work would not be equitable. Therefore, to explore ADAPT's behavior under the challenging conditions the reviewer mentioned, we have designed an experiment that simulates an imbalanced and non-stationary scenario specifically within our TTA protocol. While not a direct comparison to [1, 2, 3], we believe this analysis provides valuable insight into ADAPT's robustness.

Experimental Setup: We partitioned 1,000 ImageNet classes into three non-overlapping sets (Group A, B, C) to randomly construct a 12,500-sample test stream for each phase. Imbalance was created by drawing 80% samples from a single "dominant" group in each phase, with the remaining 20% from the other groups. Non-stationarity was induced by abruptly shifting the dominant group across four sequential phases: Phase 1 (A is dominant) → Phase 2 (B is dominant) → Phase 3 (C is dominant) → Phase 4 (A is dominant again).

Evaluation and Analysis: We compared the zero-shot CLIP with ADAPT. The results, summarized in the table, highlight ADAPT's resilience across all phases:

• Rapid Initial Adaptation: Quickly adapts in Phase 1 with a significant +2.33 accuracy gain.
• Resilience to Abrupt Shifts: When the distribution suddenly changes in Phase 2, ADAPT not only adapts but improves, widening the gain to +2.89. This showcases its ability to rapidly learn from new distributions without being destabilized by the shift.
• Cumulative Improvement: In Phase 3, ADAPT achieves its peak gain of +6.01. This suggests that as its knowledge bank is exposed to more diverse data, the model's adaptive capability becomes even more potent.
• No Catastrophic Forgetting: Most critically, when the stream reverts to Group A in Phase 4, ADAPT's performance remains high, achieving a robust gain of +3.76. This proves it can handle recurring distributions without catastrophically forgetting knowledge learned in earlier phases.

In summary, this targeted experiment demonstrates that ADAPT remains robust even under imbalanced and non-stationary test streams. Its per-class knowledge bank enables consistent performance across shifting distributions, alleviating the limitations imposed by the uniform prior assumption. These findings strengthen the practical viability of ADAPT and support its application to real-world, dynamic environments.

Thank you for the detailed follow-up. The new experiments and clarifications have been very helpful in addressing my concerns.

I believe my remaining confusion for both W1 and W2 stems from the specific definition of "online setting" used in your paper. Typically, in the TTA literature, the online setting is defined by a batch of data arriving at each time step, where the batch size is not strictly constrained (and "transductive setting" is the traditional "offline setting"). Your paper uses a more restrictive definition where the batch size is strictly 1, which could be referred to as "online setting (bs=1)." As the batch size is 1, the temporal correlation is different from the settings in [1-3].

This is an important distinction for your comparison table. The table currently suggests T3A and DPCore are not applicable in an "online setting," which could confuse readers, as those papers state their methods do work online. My understanding is that they are applicable to conventional online settings (e.g., with a batch size of 16) but may not be designed for the bs=1 scenario. Clarifying this in the table—for example, by specifying the setting as "Online (bs=1)"—is important for positioning your work accurately. The table is very informative and will be a valuable addition to the paper with this clarification if included in the final version.

I have one remaining question regarding W1: You mention a key distinction between "two families of TTA protocols" to explain why a direct comparison with some literature is not equitable. Could you please elaborate on what defines these two families and provide representative citations for each?

We sincerely thank you for this final suggestion. We completely agree that adding an additional row for "Applicability (Online, bs>1)" is a great way to highlight that ADAPT's unique strength lies in its broad applicability and effectiveness even under the more restrictive single-instance online (bs=1) protocol.

We have already incorporated this change into the comparison table provided in our earlier response. The updated table is also shown below for added clarity, highlighting that DynaPrompt fails in the conventional online setting (bs>1) due to its instance-based model reset. We will also add this information and discussion to the final version of our paper, as it helps to further emphasize what was recognized as a key strength: "the framework's flexibility to be a major plus; its ability to operate in both online and transductive settings without requiring source data makes it broadly applicable."

Thank you again for your constructive feedback throughout this process, which has significantly helped strengthen our paper.

We thank the reviewer for their thoughtful comments and feedback.

---

W1 & Q1: Clarification on the superiority of ADAPT over memory-based methods under comparable prompt conditions.

We would like to clarify that the superior performance of ADAPT stems not from advanced prompt engineering, but from its core methodology: distribution-aware adaptation.

(1) Methodological Advantage: As the reviewer correctly noted, prior memory-based methods like TDA rely on a non-parametric, instance-wise k-NN approach, making their predictions highly sensitive to individual noisy samples in online scenarios. In contrast, ADAPT employs a parametric, GDA-based strategy to explicitly model the class-conditional distributions. This modeling not only reduces the influence of individual outliers but also leads to more stable and robust decision boundaries, as illustrated in Figure 3 of our main paper.

(2) Performance Beyond Prompt Engineering: To directly address this concern, we evaluated ADAPT against TDA using a wide array of identical prompting strategies. The results summarized in the table below highlight that our ADAPT consistently and significantly outperforms TDA across all tasks and prompting settings. This confirms that the gains stem from the method itself, not from stronger textual prompts.

(3) Robustness to Prior Removal: As further proof, Figure 2 (right) in our main paper shows that ADAPT maintains SOTA performance even when the prompt-based CLIP prior is completely removed ($\alpha=1.0$). This highlights that ADAPT’s strength lies in its data-driven, distributional modeling, not in external textual guidance.

---

Q2: Comparison with AWT.

While ADAPT uses the same handcrafted prompts as AWT, their core adaptation strategies differ fundamentally. AWT aligns multiple augmented views of each test image with textual prompts via Optimal Transport, making its performance highly dependent on visual augmentations.

To illustrate this, we compare ADAPT with both original AWT and a variant (AWT*) without visual augmentations. As shown in the table below, AWT suffers a significant performance drop without augmentation, confirming this reliance. In contrast, ADAPT achieves consistently strong performance without requiring such augmentations.

This highlights ADAPT's ability to perform source-free, training-free adaptation without relying on computationally expensive visual augmentations. This makes ADAPT a more streamlined and versatile solution, which is readily applicable to both online and transductive settings.

We sincerely thank the reviewer for this thoughtful follow-up. While the raw margins (e.g., +0.6% on Task 1, +0.3% on Task 2) may appear small at first glance, they are in fact substantial given the saturation of these benchmarks and the nature of the shifts. Moreover, ADAPT’s design offers advantages in robustness, generality, and efficiency-flexibility that we believe outweigh the perceived limitations.

---

1. Significance in a Saturating Benchmark

Performance on the Task 1 and Task 2 benchmarks is approaching saturation, where the gap between top-tier methods is typically less than 0.5%, as shown in the table below. In this context:

• ADAPT achieves consistent, positive gains across all prompts, for example, +0.58% on Task 1 (online) with CLIP templates, outperforming all listed recent SOTA works. Similarly, on the extremely challenging Task 2 (ImageNet-C, severity 5), even a +0.3% gain reflects improved robustness under the harshest corruption.
• In the transductive setting, ADAPT’s advantage widens further (+1.30% on Task 1, +1.91% on Task 2), demonstrating its full potential when more test data is available. | Method | Venue | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. | Gains | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | TDA (CLIP Template) | CVPR 2024 | 69.23 | 63.19 | 65.06 | 80.67 | 51.51 | 65.11 | 65.93 | | | DMN-ZS | CVPR 2024 | 72.25 | 58.28 | 65.17 | 78.55 | 53.20 | 63.80 | 65.49 | -0.44 | | AWT | NeurIPS 2024 | 71.32 | 60.33 | 65.15 | 80.64 | 51.60 | 64.43 | 65.81 | -0.12 | | DPE | NeurIPS 2024 | 71.91 | 59.63 | 65.44 | 80.40 | 52.26 | 64.43 | 65.93 | +0.00 | | ZERO | NeurIPS 2024 | 71.17 | 62.75 | 65.23 | 80.75 | 50.59 | 64.83 | 66.10 | +0.17 | | BCA | CVPR 2025 | 70.22 | 61.14 | 64.90 | 80.72 | 50.87 | 64.41 | 65.57 | -0.36 | | ECALP | ICLR 2025 | 71.26 | 58.52 | 65.72 | 80.77 | 54.66 | 64.92 | 66.19 | +0.26 | | ADAPT (Online, CLIP Template) | | 70.37 | 63.72 | 65.15 | 80.95 | 52.34 | 65.54 | 66.51 | +0.58 | | ADAPT (Trans., CLIP Template) | | 71.17 | 64.72 | 66.18 | 81.28 | 52.78 | 66.24 | 67.23 | +1.30 |

---

2. Core Technical Insights

Beyond numerical improvements, ADAPT’s improvements come from a shift from instance-based to distribution-aware adaptation:

• TDA’s Limitation: Purely instance-based, TDA relies on nearest-neighbor voting and is sensitive to outliers and noisy samples, leading to unstable decision boundaries under distribution shift (see Figure 3 in the main paper).
• ADAPT’s Advantage: By modeling class-conditional distributions (progressive mean estimation and shared covariance), ADAPT stabilizes decision boundaries, naturally down-weights outliers, and adapts more gracefully as shifts worsen.
• Hyperparameter Robustness: Due to its instance-based nature, TDA is highly sensitive to data distribution shifts and requires careful hyperparameter tuning. As a result, its performance heavily depends on multiple dataset-specific thresholds. Using shared hyperparameters from ImageNet (noted as TDA*) can lead to a significant performance drop of 1.45%. In contrast, ADAPT uses the same set of hyperparameters across all datasets without per-dataset tuning, making it much easier to deploy.
• Full-Data Utilization: Unlike TDA, ADAPT can exploit the transductive setting to refine its estimates over the entire test distribution, unlocking further gains in accuracy and efficiency. | Method | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. | Gains | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | TDA (Online) | 69.23 | 63.19 | 65.06 | 80.67 | 51.51 | 65.11 | 65.93 | | | TDA* (Online) | 69.23 | 63.19 | 62.55 | 79.15 | 48.28 | 63.29 | 64.48 | -1.45 | | ADAPT (Online) | 70.37 | 63.72 | 65.15 | 80.95 | 52.34 | 65.54 | 66.51 | +0.58 | | TDA (Trans.) | 69.25 | 63.00 | 65.13 | 80.43 | 51.55 | 65.03 | 65.87 | -0.06| | ADAPT (Trans.) | 71.17 | 64.72 | 66.18 | 81.28 | 52.78 | 66.24 | 67.23 | +1.30 |

---

3. Efficiency–Flexibility Advantage

• Online Mode (High Accuracy): On ImageNet, ADAPT’s online mode takes 1h 11m vs. TDA’s 50m, only +21 min for a +1.4% accuracy gain.
• Transductive Mode (High Efficiency): ADAPT can process the full test set in 0.73 min, ~70× faster than TDA’s online mode, while maintaining strong performance.
• Unique Flexibility: TDA’s purely instance-based online design cannot benefit from transductive processing, whereas ADAPT can operate in either high-accuracy online or high-efficiency transductive mode, offering a practical deployment advantage.

---

In summary, ADAPT delivers SOTA performance on saturated benchmarks, improves robustness in severe shifts, requires no dataset-specific tuning, and offers a flexible accuracy–efficiency trade-off that TDA cannot match. These characteristics are core strengths, not limitations.

We thank the reviewer for the valuable feedback.

---

W1: Assumption of Gaussian distributions with a shared covariance.

Thank you for this insightful question. We agree that the unimodal Gaussian assumption introduces a simplification and may be limiting in hypothetical scenarios involving highly complex feature distributions. However, as demonstrated in the main paper, ADAPT consistently achieves SOTA performance across diverse and challenging settings, including domains with severe visual corruption (Table 3) and fine-grained classification tasks (Table 4). These empirical results suggest that, despite the assumption, our method remains robust and effective in real-world applications.

To further investigate this design choice, we conducted an ablation study where we replaced the shared covariance matrix with class-wise covariance matrices, keeping all other settings fixed. This led to a notable drop in accuracy, along with increased memory usage and inference time. We attribute this to data sparsity: in many TTA settings, only a few high-confidence samples are available per class, making class-wise covariance estimation unreliable and prone to overfitting.

In contrast, our shared covariance pools statistics across classes, producing a more stable and generalizable estimate, which is crucial in online or data-scarce environments. This further supports our design choice: the shared covariance assumption is not merely for reducing computational overhead but is crucial for improving stability and generalization. As the reviewer also noted, a key strength of ADAPT lies in its broad applicability, particularly in practical TTA scenarios where low latency and limited resources are critical.

---

W2: Hyperparameter analysis.

We already provided a detailed hyperparameter analysis in Section 4.3 of our main paper, particularly in Figure 2 and Table 7. For clarity, we summarize our response as follows:

(1) Confidence Threshold: Our method does not use any manually defined confidence threshold. Instead, we adopt a fixed-size memory bank that is dynamically updated; new samples with higher confidence scores replace those with the lowest scores. This adaptive mechanism eliminates the need for hand-crafted thresholds and ensures that only the most reliable samples are retained.

(2) Bank Size (L): ADAPT exhibits strong robustness to this parameter. As shown in Figure 2 (left), performance in the transductive setting remains stable across a wide range of values for L, while in the online setting, accuracy plateaus once L exceeds 10. We thus use a single, fixed configuration across all datasets (L=16 for online, L=6 for transductive), with no per-dataset tuning required.

(3) CLIP Prior Means: ADAPT is not sensitive to the initialization of the CLIP prior means. Table 7 verifies the robustness of our method under various mean settings. Furthermore, Figure 2 (right) shows that even when the CLIP prior mean is entirely removed (α=1.0), ADAPT still achieves state-of-the-art performance.

In summary, ADAPT does not rely on dataset-specific hyperparameter tuning or static priors. Instead, it leverages dynamic, data-driven mechanisms to adaptively maintain a reliable knowledge bank. This design choice ensures strong performance, high robustness, and ease of deployment in real-world settings.

---

Q1: What is the effect of significant noise in the early high-confidence predictions on long-tail adaptation?

To assess the impact of early-stage prediction noise under long-tailed distributions, we conducted a controlled experiment with the following setup:

(1) Long-tail dataset construction: We constructed a long-tailed subset from ImageNet-1K by sampling classes according to a power-law distribution, resulting in:

• Head class: 61 classes (≥40 shots, 2,719 images)
• Medium class: 338 classes (10–40 shots, 7,396 images)
• Tail class: 601 classes (≤10 shots, 2,416 images)

(2) Worst-case noise injection: To simulate an extreme scenario, we deliberately injected noise during the early 30% phase of adaptation. Specifically, we corrupted 30% of the high-confidence predictions by randomly reassigning them to incorrect classes. This models a challenging case where early memory is contaminated with misleading information.

(3) Evaluation and Findings: We compared the baseline CLIP model with ADAPT in this adverse setting. The results, summarized in the table below, yield three key observations:

• While both methods degrade under heavy noise, ADAPT consistently achieves higher final accuracy (64.27% vs. 63.25%), demonstrating superior overall robustness.

• For the extremely few-shot tail classes, both methods are highly vulnerable to the strong initial misguidance, exhibiting a comparable performance drop. This is an expected outcome, as a 30% initial corruption can heavily skew parameter estimates when very few samples are available.

• ADAPT demonstrates self-correction capability: As adaptation progresses, the accumulation of correctly identified samples (especially from head/medium classes) allows the model to progressively refine its distribution estimates, effectively mitigating the influence of the initial noisy predictions. This confirms that ADAPT is a more reliable solution for practical, noisy, long-tailed scenarios.

---

Q2: Clarification on the required priors and historical samples for ADAPT.

ADAPT operates in a training-free and label-free manner. The only information it uses is derived from (1) a small set of high-confidence samples from the test stream itself and (2) optional class-mean priors from CLIP.

(1) Historical Samples: ADAPT maintains a fixed-size memory bank containing previously seen test samples with high-confidence predictions. These are not labeled samples, but are selected based on prediction confidence during inference. The memory bank is continuously updated, replacing the lowest-confidence entries with more reliable samples, and is used to guide the adaptation of incoming samples. This strategy eliminates the need for manual thresholds, making ADAPT a fully self-supervised approach.

(2) CLIP Prior (Optional): We optionally initialize the class-wise means using CLIP’s textual priors, but this prior is not essential. As shown in Table 7 and Figure 2 of the main paper, even when these CLIP priors are removed (α = 1.0), ADAPT still achieves strong performance. This highlights the robustness of our approach to the choice of prior.

In essence, ADAPT is a self-contained method that adapts directly to the test stream, eliminating the need for external labeled data or strong, fixed priors.

Dear Reviewer,

Thank you very much for your thoughtful follow-up and positive feedback. We're glad that our responses have fully addressed your concerns, and we truly appreciate your decision to recommend acceptance of our work.

Sincerely,

The Authors

We thank the reviewer for the constructive comments. We provide our feedback as follows.

W1: Performance and robustness under significant shifts.

We agree that robustness against unreliable predictions under severe domain shift is essential. To further assess our method under significant shifts with synthetic corruptions, we benchmarked ADAPT against TDA, a state-of-the-art memory-based method, across five levels of increasing shift severity. As shown in the table below, while all methods degrade as the shift intensifies (an expected trend in an unsupervised setting), our ADAPT consistently outperforms TDA across all levels.

Notably, ADAPT's performance advantage is even more significant under severe shifts (e.g., Level 4 and Level 5), where the risk of memory corruption is greatest. This provides strong evidence that our proposed mechanism is more resilient to noisy pseudo-labels, validating its superior robustness for practical, severe distribution shifts.

---

W2: How having classes with very few high-confidence samples affects the results.

We analyze how the number of high-confidence samples per class (denoted as N, ranging from 2 to 10) influences performance. As shown in the table, while our online method is somewhat sensitive to extremely low sample counts (e.g., 2 per class), it recovers quickly and achieves robust performance with only 4 to 6 samples per class.

Notably, our transductive ADAPT maintains high and stable performance even with just 2 high-confidence samples per class. We attribute this to its ability to leverage the global structure of the full test batch, which regularizes parameter estimates and ensures robustness under data scarcity.

---

W3: On the convergence and divergence bounds of estimated parameters.

We appreciate the reviewer’s insightful question. Providing formal guarantees for convergence is inherently difficult, as our regularization term $\mathcal{R}(z_i;\mathcal{B})$ relies on a knowledge bank and introduces non-decomposable dependencies. This type of regularizer poses fundamental challenges, and to the best of our knowledge, establishing general convergence theory for such structures remains an open problem in the broader learning theory community.

Given these theoretical challenges, we instead provide a thorough empirical validation of our parameter estimation. We conducted experiments on 100 randomly selected ImageNet classes to measure the divergence between our estimated distribution parameters ($\mu_k$ , $\Sigma$) and those of the ground-truth empirical distribution.

The results, averaged over 10 runs and summarized below, demonstrate the high accuracy of our estimations. As the table shows, the estimation errors are consistently low and stable. The small Frobenius distances (F-dist) for the mean ($\mu$) and covariance ($\Sigma$), along with the low overall error, confirm that the estimated distribution closely aligns with the ground-truth. This strong empirical evidence demonstrates that, while a formal proof is an open challenge, our method effectively converges to the true underlying data distribution in practice. It achieves this highly accurate and robust estimation without requiring any supervision.

---

W4: Evaluation with different VLMs.

We have already evaluated ADAPT with different VLMs in Appendix B.3. For reference, we present the summary table below, and more detailed results can be found in Figure 3 of the Appendix. ADAPT delivers consistent gains across various models, including CLIP (ViT-B/16, RN50), BLIP [1], and ALBEF [2]. This demonstrates that ADAPT is model-agnostic and broadly applicable across different VLM architectures.

[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022)

[2] Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation (NeurIPS 2021)

---

W5: Report the standard deviation of the results.

To address the concern regarding stability, we evaluated our method over 10 random permutations of the test stream, reporting the average performance and standard deviation across three tasks. The results confirm our method's high stability, with a consistently low standard deviation across all tasks. These standard deviations will be included in the tables of our final manuscript to formally report the method's reliability.

Task1:

Task2:

Task3:

---

Q1: Experiment with the bank-pruning step.

Thank you for this constructive suggestion. We implemented a bank-pruning mechanism as a new ablation study. We periodically (every 1k steps) remove a portion of the lowest-confidence samples (e.g., 10%, 30%, 50%) from the memory bank. This pruning is done on a per-class basis, as a single global confidence threshold can be unfair to naturally 'harder' classes. The results, presented below, show that this pruning strategy consistently degrades performance, especially on the more challenging Image-Net-A dataset with distribution shift.

This outcome suggests that ADAPT inherently performs a form of soft dynamic filtering, effectively mitigating the impact of noisy samples without requiring explicit pruning. Through its regularized parameter estimation process, the method naturally down-weights the influence of unreliable samples rather than discarding them entirely. Introducing an explicit or aggressive pruning mechanism may, in fact, be counterproductive, as it risks eliminating samples that could become informative as adaptation progresses. This finding highlights the robustness and stability of our original design, which avoids premature decisions while remaining resilient to noise.

---

Q2: Using separate covariance matrices.

As we detail in Appendix A.4 (Discussion), we provide strong empirical evidence that the class-conditional features approximately follow Gaussian distributions with a shared covariance matrix.

To further compare with class-wise separate covariance matrices, we conducted additional experiments on ImageNet by replacing our shared covariance with class-specific covariance matrices, keeping all other settings identical. The results, summarized below, show that this leads to a significant drop in accuracy, alongside a sharp increase in computation time and memory usage.

We attribute this performance degradation to data sparsity. Estimating a full covariance matrix for each class becomes unreliable with few high-confidence samples, leading to poor generalization and overfitting. In contrast, the shared covariance pools information across all classes, resulting in a much more stable and robust estimate, which is critical in online or data-scarce settings.

Ultimately, this ablation confirms that our shared covariance assumption is not a limitation, but a crucial design choice. It strikes an effective trade-off between accuracy, robustness, and computational efficiency, making our method practical for real-world, resource-constrained scenarios.

We sincerely appreciate the reviewer's thoughtful follow-up and their consideration of our previous explanations. We agree that the analysis provided in response to Q2 offers valuable context and will include it in the appendix of the revised version. Regarding W1 and Q1, we are pleased to provide additional details and clarifications below.

---

W1 & Q1: On Shift Robustness and the Effect of Pruning

(1) Nature of Shifts and ADAPT's Growing Advantage (W1):

(1-1) Nature of the Tested Shifts (ImageNet-C):

In W1, we follow the common protocol of evaluating on the corruption-shift benchmark ImageNet-C [1] to test robustness against challenging domain shifts. This benchmark introduces 15 corruption types (e.g., Gaussian noise, blur, JPEG compression) across 5 severity levels. These corruptions impact both the central (mode) and peripheral (tail) regions of the feature distribution. For instance, blurring and defocus affect core object representations, while noise and distortions disproportionately degrade rare or ambiguous samples. This results in a progressively more severe distribution shift across all severity levels.

(1-2) Rationale for ADAPT's Growing Advantage on Severe Shifts:

The key lies in the contrast between CLIP's static design and ADAPT's dynamic adaptation:

• CLIP's Sharp Decline: CLIP uses a fixed classifier calibrated on clean data, making it highly sensitive to corrupted or off-distribution features. As corruption severity increases, CLIP's predictions become misaligned with the target test distribution, leading to sharp performance degradation.

• TDA's Vulnerability: TDA, as a memory-based method, slightly improves upon CLIP in moderate shifts. However, its k-NN-like, instance-based strategy makes it highly vulnerable to outliers in noisy conditions. This core weakness is exposed under severe corruption, causing its performance to dramatically degrade and even drop below the static CLIP at Level 4 (36.84 vs 37.51).

• ADAPT's Resilience: In contrast, ADAPT dynamically estimates class-wise distributions using soft pseudo-labels and confidence-based memory banks. While it relies on representative examples, ADAPT does not rely on perfect pseudo-labels. Instead, its use of soft pseudo-labeling and fixed-size class-wise memory banks prevents dominance by corrupted outliers and allows it to progressively model the distribution of the corrupted data. This soft adaptation avoids overfitting to outliers while gradually aligning with the shifted data.

• Advantage is Relative: Because ADAPT degrades more gracefully than CLIP, the relative performance gap grows with increasing corruption severity. As shown in our W1 results, ADAPT (Trans.) outperforms CLIP by +2.36% at Level 1, growing to +4.79% at Level 5.

---

(2) Negative Effect of Pruning (Q1):

(2-1) Nature of the Tested Shifts (ImageNet-A):

In our Q1 experiment, we consider both ImageNet and ImageNet-A. The latter represents a natural distribution shift, consisting of naturally occurring, real-world examples that models find inherently difficult, thus representing the long tail of the data distribution.

(2-2) Why Hard Pruning Fails:

Hard pruning removes samples based solely on confidence. However, under severe shifts like ImageNet-A, many samples with relatively low confidence remain semantically valuable. Discarding them reduces diversity and limits the model's ability to adapt to unfamiliar patterns in the test data.

Instead of pruning, ADAPT dynamically updates its class-wise knowledge bank and applies soft confidence-based weighting. This ensures that even samples with relatively low confidence can contribute proportionally if they are informative. This allows ADAPT to gradually refine its distribution estimates using the full test stream, balancing robustness and adaptability even in low-data or high-noise regimes.

References:

[1] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ICLR 2019)

---

We hope this clarification resolves your concerns. If you have any further questions, we'd be glad to discuss them in more detail.