Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models

Dear Reviewer eVss,

Thank you for your positive assessment of our work. We’re delighted that you appreciate STS’s combination of black-box simplicity and semantic grounding for episodic test-time adaptation.

W1

We thank the reviewer for raising this point. We agree that ZERO’s hard‐voting scheme is important and it is not in competition with STS but rather complementary. In fact, one can seamlessly replace STS’s marginal‐probability averaging over confident views with a majority‐vote across those same views:

1. Adapt with STS as usual. Compute the low‐dimensional shift $\Delta z = B\*\gamma$ via entropy minimization (Eq. 4) and apply it to the prototypes.

2. Predict by voting instead of averaging. For each of the $N$ filtered augmentations, take the arg max over the adapted similarities to cast a “vote” for its top class, then choose the class with the most votes.

When we tried this combined STS + ZERO voting pipeline on ImageNet-A, we observed an additional +0.15 % accuracy gain over STS’s standard averaging, demonstrating that ZERO’s discrete consensus step can further sharpen STS’s predictions without any extra gradient computation.

Q1 By minimizing the marginal entropy over your filtered views, rather than the conditional entropy of each view, we gain several practical and robustness benefits:

• Robustness to outlier views. Conditional entropy forces every selected crop to be highly confident, so a single ambiguous or noisy view can pull the update in a harmful direction. Marginal entropy only requires the aggregate distribution (across all filtered views) to be sharp, allowing a few less‐confident or noisy views to be “overruled” by the majority.

• Encourages consensus without over-fitting. By focusing on the consensus distribution, marginal entropy guides STS to find a shift that works well overall, not just for each individual augmentation. This is particularly important when you have only a small number of filtered views, the fewer the views, the more you want to lean on their collective signal.

In short, marginal entropy minimization on your filtered subset yields a cleaner, more robust, and more easily optimized objective for a single-step, black-box adaptation, exactly what STS needs to steer the prototypes confidently without getting thrown off by individual noisy views.

Q2

Augmentations are related to the marginal entropy minimization and our goal to have a more robust representation for the test sample.

Q3

We tried this approach and we found that the performance improvement of having a steering vector for both the visual and textual embeddings is marginal.

Q4

Please see the response on reviewer QTGH for the results on CIFAR10-C.

Dear Reviewer QTGH,

We sincerely thank you for recognizing the novelty and motivation behind our proposed approach. We also appreciate the acknowledgment of the method’s parameter efficiency and its ability to significantly reduce computational overhead compared to existing test-time adaptation (TTA) strategies. These aspects were central to our design goals, and we are glad they resonated with the reviewer.

We address the raised concerns and questions below:

W1 - Method's Adaptation Mechanism:

We appreciate the reviewer’s point and in fact explicitly note this in our Limitation section (Sec 6): STS’s single linear shift in Eq 4. may under-fit highly complex transformations. In STS, however, we find that:

a. Across ImageNet‑A, and ‑V2 a single linear shift in our SVD‑derived subspace recovers a significant percentage of the zero‑shot accuracy gap under moderate to severe corruptions. This suggests that, even when the visual perturbation is nonlinear, the resulting change in optimal text‐prototype geometry often lies in a roughly linear manifold.

b. We also evaluated STS on CIFAR-10-C at the most severe corruption level (severity 5) and observed a +3.52 % accuracy gain over the zero-shot baseline, demonstrating that even under extreme, nonlinear corruptions, steering in our low-dimensional semantic subspace yields substantial robustness improvements.

We believe that this design reflects a deliberate trade-off that maximizes robustness and efficiency:

c. Under the $*manifold hypothesis*$, pretrained text embeddings concentrate on a low-dimensional semantic manifold. An SVD of the class-prototype matrix uncovers the principal axes along which classes vary. Even complex visual corruptions often project predominantly onto these top directions in text space, so a linear shift suffices to recover the bulk of the alignment gap.

d. By constraining adaptation to the top-k singular vectors, STS avoids over-fitting low-energy, noise-driven directions, an ever-present risk in richer, nonlinear adapters when only a few unlabeled views are available. This bottleneck both stabilizes and focuses learning on the most meaningful shifts.

In summary, although STS’s core update is linear, it is carefully designed to harness the most significant semantic directions, yielding a practical balance of expressivity, robustness, and efficiency.

W2 - Ablation Studies:

We thank the reviewer for suggesting a deeper ablation on (a) the choice of spectral subspace dimension and (b) the number of augmented views. Below we report two experiments:

a. Subspace dimension sensitivity

We compare two strategies for selecting $k_t$ over three random initializations (seeds) and the performance on ImageNet-A dataset:

(i) Gavish-Donoho hard thresholding, and (ii) 98% energy retention

b. Number of Augmented Views

We vary the number of views $N ∈ ({16,32,64,128})$ reporting accuracy on ImageNet-1K:

Increasing from 64 → 128 views gives only a +0.15 % boost, while nearly doubling time and memory. We therefore adopt $N=64$ (as in prior TPT work) to balance performance and efficiency, still remaining substantially faster than prompt‐tuning baselines.

W3 - Suboptimal on certain Datasets:

We thank the reviewer for raising this point. Here, we pinpoint two possible reasons that have to do with the visual transformations (augmentations):

a. ** Familiarity of the semantic categories:**

Many of the ImageNet‐variant benchmarks (and fine‐grained datasets like SUN397 or Caltech101) cover object classes that CLIP likely saw often during pretraining. As a result, CLIP’s predictions remain stable across simple augmentations for these “common” categories. In contrast, classes in datasets such as Flowers102 or Oxford‐Pets represent less frequent concepts, so CLIP is more sensitive to their augmented views.

b. Nature of the visual transformations:

Our augmentation pipeline uses only random resized crops and horizontal flips, essentially random “zoom‐ins.” For some domains (e.g. FGVC‐Aircraft, Stanford Cars), zooming can highlight distinctive details like logos or text, which helps CLIP recover the correct label. But on datasets like Flowers102, cropping can remove critical features (e.g. petals or stems), degrading performance.

In this work, we intentionally kept our augmentations uniform across all tasks rather than tuning them per dataset. Nevertheless, these results suggest that the choice of augmentations, and how they interact with the model’s learned visual priors, is a rich avenue for future study.

Q1

We thank the reviewer for calling attention to the need for stronger theoretical grounding. In the revised manuscript, we will add the following justifications:

a. Low intrinsic dimension of pretrained embeddings:

Prior work shows that deep‐network features—both vision and text—concentrate on a low‐dimensional manifold. In particular, Aghajanyan et al. (ICML 2020) empirically measure this “intrinsic dimension” and demonstrate that most semantic information lies in a small subspace. This motivates our choice to perform adaptation in a structured, low-dimensional basis rather than the full embedding space.

[1] INTRINSIC DIMENSIONALITY EXPLAINS THE EFFECTIVENESS OF LANGUAGE MODEL FINE-TUNING, Aghajanyan et al. (ICML 2020)

b. Domain‐adaptation generalization bound:

We reference the theoretical bound of Ben-David et al. (JMLR 2010), which states that the target risk is controlled by the source risk plus a divergence term between source and target feature distributions. By minimizing distributional discrepancy in our spectral subspace (via entropy minimization), STS directly reduces this divergence and thus the target error.

[2] Nonlinear Dimensionality Reduction by Locally Linear Embedding, Ben-David et al. (JMLR 2010)

Q2

STS can indeed be applied with no augmentations (i.e. a single view, $N=1$), since the optimization in Eq. (4) does not strictly require multiple crops. However, in our experiments, we observe that:

As $N$ increases accuracy improves but computational cost grows roughly linearly.

Q3

Thank you for asking about the TTA and STS's broader applicability. Our experiments cover a broad range of domains, from ImageNet variants to Satellite images, and Flowers variants. While our experiments focus on zero-shot image classification, STS is not tied to any specific modality or task, as long as two conditions hold:

a. We have static embeddings (e.g. text prompts, class centroids, label vectors) that represent the categories or concepts of interest.

b. You can generate multiple views of each test sample (e.g. augmentations of an image, noise-perturbed audio) without requiring labels.

Q4-7

All minor errors and typos will be corrected.

Dear Reviewer 2jDh,

Thank you for the positive recommendation of our work and the comments.

W1

We appreciate the reviewer’s interest in understanding STS’s behavior on more modern backbones. In the Supplementary Material, we report results for CLIP ViT-L/14 on ImageNet and its OOD variants. As we can see, our method still outperforms zero-shot by a large margin on average across 5 datasets, showcasing that our method generalizes well to larger-scale VLMs.

W2

Thank you for your interest in the spectral subspace.

Here, we note that PCA approach needs to center the class prototypes (substract the mean of the C rows in a CxD matrix), then compute the covariance and then perform an eigen-decomposition on the covariance matrix and take its top-k eigenvectors.

Because CLIP’s embeddings live on the unit hypersphere, their global mean is not zero, and removing it (as PCA requires) can actually distort the geometry you care about.

• CLIP optimizes ⟨zᵥ, zₜ⟩ with both zᵥ and zₜ normalized to ‖·‖₂ = 1. Those points lie on the surface of a high‑dimensional sphere, and their mean ⎡∑ᵢ zᵢ⎤/N sits strictly inside the sphere.

• PCA subtracts that mean before finding principal axes, effectively “recentering” your data cloud at the origin. That shifts your spherical manifold in a way that no longer respects the original contrastive geometry.

SVD on uncentered data respects the learned hypersphere

• A raw SVD of the C×D text‑prototype matrix Zₜ finds the directions of maximum energy,including any offset from the sphere’s center, which often encode globally relevant semantics (e.g. a “generic image” direction).

• Those directions align more naturally with CLIP’s training objective, which never removed the mean and whose singular vectors capture both mean and variation.

Empirical parity (and simplicity)

• In practice, centering then doing PCA yields virtually the same subspace span as SVD on the centered data, but differs from SVD on the raw data.

Dear Reviewer VSFB,

Thank you for your comments and positive recommendation of our work. We provide point-by-point responses to address your concerns below.

We would like to clarify that STS does not perform prompt tuning; instead, it steers the frozen text prototypes directly in a low-dimensional spectral subspace.

W1 - Limited Scope of Evaluation: Thank you for the suggestion to evaluate STS alongside other adaptation pipelines. A few points address this:

a. STS vs. DiffTPT as baseline
In Table 1, DiffTPT appears as an independent baseline; its diffusion-based augmentations are not combined with STS. While STS is fully compatible with any view generation scheme (including DiffTPT’s), our core goal is ultra-efficient, episodic black-box TTA: adopting DiffTPT’s augmentations would multiply per-sample latency and GPU memory by 5–10×, undermining STS’s real-time, low-footprint advantage.

b. Plug-and-play generality
Conceptually, STS can replace the gradient-through-prompt step in any TPT-style framework with a single spectral steering update. If a domain demands richer augmentations, one can simply feed those views into STS’s entropy-minimization loop (no changes to the SVD are required).

c. PromptSRC’s
PromptSRC [2] is designed for few-shot prompt tuning with labeled examples at test time. Because STS targets zero-shot, unlabeled, per-sample black-box adaptation, combining it with a few-shot method falls outside our stated problem setting.

W2 - Missing Baselines:

a. Directly ensembling results with negative entropy

We thank the reviewer for these helpful suggestions. Below we explain how each of these baselines fits into our black-box, episodic TTA setting, and point to where we already cover or will cover them in revision:

We appreciate the reviewer’s suggestion to include a test-time ensembling baseline via averaging predictions over augmented views (i.e., without adaptation). We implemented this “zero-shot ensemble” baseline and tested it on the challenging ImageNet-A dataset. We found that it provides a +8.63% absolute improvement over standard zero-shot CLIP, increasing top-1 accuracy from 47.87% to 56.50%.

While this confirms that multi-view ensembling can modestly improve robustness, our proposed STS method still significantly outperforms it, achieving up to 61.23% with a single hand-crafted prompt and 64.29% with ensemble.

This result demonstrates that STS goes beyond passive confidence averaging. It delivers active and semantically targeted adaptation in the latent space, leading to greater gains under distribution shift than ensembling alone.

b. Updating the last transformer layer

Such a baseline requires white-box access to CLIP’s internal weights and architecture to backpropagate into a deep layer, contradicting our design goal of black-box TTA. STS assumes only query access to the model’s logits or prototypes, never internal gradients. For real-world deployment (e.g. proprietary APIs), updating a hidden layer is infeasible, so we do not evaluate it.

c. Identity-residual on text features

Learning an matrix shift on the D-dimensional text embeddings is exactly the shared-residual variant of Test-Time Prototype Shifting (TPS) [31], and learning a full C×D residual is TPS’s class-specific variant. In TPS paper, the authors show that the class-specific variant performs slightly better than the shared-residual variant. This class-specific variant of TPS appears in Table 1 and Table 2. Across all 15 benchmarks, STS, using only a small number of spectral coefficients, consistently outperforms TPS class-specific residuals, demonstrating that our SVD-based bottleneck is a strictly stronger, more parameter-efficient adaptation mechanism that steers the textual prototypes along the most semantically meaningful directions.

W3 - More Performance-Efficiency Comparisons:

We thank the reviewer for highlighting these training-free baselines, which indeed represent an important point on the performance-efficiency frontier.

Prompt-weighting method, re-weights a fixed ensemble of hand-crafted prompts by minimizing the ensemble entropy. Unfortunately, no public code or details were released, so a faithful re-implementation is infeasible. In the revised manuscript, we will incorporate a qualitative discussion of it.

AWT method relies on an external LLM to craft dataset-specific text prompts from prior knowledge, then re-weights and transports multi-view image features via optimal transport (still no backprop through the VLM but with significant dependency on the LLM and multiple heavy forward passes). AWT’s use of the external LLM to craft dataset-specific prompts brings a practical caveat: it requires prior knowledge of the entire target dataset’s class taxonomy and domain characteristics in order to generate tailored descriptions. In true episodic, zero-shot TTA settings, where each sample arrives in isolation and no labeled or domain-wide metadata is available, this a priori dataset overview is typically unavailable, limiting AWT’s applicability. In contrast, STS operates purely on the model’s frozen prototypes and per-sample augmentations, with no dependence on external models or dataset-level information. These additions will clarify why, even among training-free methods, STS occupies a unique point on the performance-efficiency frontier.

Dear Reviewer EXj6,

We greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.

W1 - Missing Details

Spectral dimension K_t: We select K_t by applying the Gavish-Donoho thresholding rule to the singular-value spectrum of the text prototypes. For our CLIP ViT-B/16 experiments with the hand-crafted prompt “a photo of a” this yields to:

W2 - The method

We thank the reviewer for prompting a deeper comparison. Three points clarify why SVD-based steering outperforms both class-specific and shared residuals:

a) $*Structured, low-dimensional adaptation*$: As discussed in Section 3.2 (main manuscript), pretrained deep features exhibit a low intrinsic dimension [1], meaning their essential semantics lie on a smaller manifold. By performing an SVD of the full text-prototype matrix we extract the principal singular vectors that capture the most salient axes of class variation. Adapting within this constrained subspace inherently regularizes learning (preserving the rich, frozen knowledge of the VLM) and focuses updates along directions of maximal semantic relevance, rather than allowing noisy, unconstrained shifts in the D-dimensional embedding space.

b) $*Relation to TPS’s residuals*$: In Section 2 (main manuscript) we describe Test-Time Prototype Shifting (TPS) [Sui et al., NeurIPS 2024], which indeed learns either a full class-specific residual $R ∈ \mathbb{R}^{C×D}$ or a shared residual $r ∈ \mathbb{R}^D$ on top of $Z_{T_{init}}$​​, avoiding backprop through encoders similar to [A]. Our STS method advances this by introducing a spectrum-aware mechanism: rather than learning arbitrary shifts, we learn compact coefficients that steer prototypes only along the top-k singular directions. TPS’s experiments show that class-specific residuals modestly outperform shared ones, but both remain unconstrained and thus prone to overfitting under scarce, unlabeled test-time data.

b) $*Empirical evidence*$: As reported in Section 4 (main manuscript), STS, using just $k_t$ parameters per sample, consistently outperforms TPS across all 15 benchmarks. This demonstrates that constraining adaptation to a semantically meaningful, low-rank subspace yields better OOD accuracy and stability than learning full or shared residuals in the high-dimensional prototype space.

Together, these points show that SVD-driven subspace adaptation is not only theoretically motivated by intrinsic dimensionality, but also empirically superior to both class-dependent and shared residual approaches.

W3 - The experiments

Thanks for your thoughtful feedback. We acknowledge that our STS method largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8× faster with a 12× smaller memory footprint than conventional test-time prompt tuning. Here we discuss why without ensembling, STS and TPT yield closer performance on most ImageNet variants, but diverge sharply on ImageNet-A:

a1) ImageNet-A is made of “border-line” images that exploit spurious, low-level cues which CLIP has not fully suppressed. ImageNet-A was intentionally filtered to keep pictures whose foreground object is hard to spot (small, partially occluded, off-center) or whose background contains misleading texture cues. Its adversarial filtering process selects images that standard models misclassify (so they lie at the extremes of semantic variation, which align closely with the principal axes of the text-embedding manifold). Because every image already triggers the right class, the model’s output entropy is high and a tiny shift in the decision boundary flips a large fraction of errors into correct predictions. STS adapts by steering directly the text prototypes along the top-k singular vectors of the text-embedding matrix (directions that capture the dominant semantic axes e.g. object vs. background, canonical vs. extreme pose). Under the extreme distortions of ImageNet-A, the optimal correction indeed lies very close to one of those principal axes, so a single spectrum-aware steering recovers a disproportionate amount of accuracy. TPT’s prompt tokens must propagate their effect through multiple transformer layers, making it harder to affect such a large, coherent shift in a single gradient step (especially when tuning must remain lightweight). The same observation occurs for TPS method since as you can see by our experiments, TPS greatly improves performance over the TPT.

a2) Complementarity with ensembling: The framework requires a single representation per class, but if such a representation is derived from a single prompt, the amount of information that it carries is limited to the number of input tokens that the CLIP text encoder was trained with. Hence, we easily improve the robustness of class prototypes by taking the mean of each class embeddings. This allow us to leverage multiple prompting and retain the knowledge from these various representations while keeping the computational and memory efficiency of our method without sacrificing prototype representation strength. As we mention in the paper, the design of TPT does not allow leveraging this text ensemble (also pointed out by concurrent work [31]).

a3) Alternative adaptation choice: Thank you for your suggestion. We note that TPS’s class specific and shared-residual variants (which mirror two of these suggested choices [A][Z_{T_{init}}] are already compared in Table 2 and fall short of STS’s performance (confirming the value of our spectral bottleneck). Clip-adapter [8] and [B] have a prevalent assumption which is the accessibility of labeled data from the target domain, a condition that is frequently incompatible with the requirements for rapid deployment in real-world applications. In addition, a full study of every ensemble-compatible alternative is beyond our current scope. We believe the principled regularization and semantic interpretability of SVD-based adaptation make STS uniquely effective, especially in the episodic black-box TTA that we investigate.

We thank the reviewer for pointing out the ambiguity around our MaPLe initialization and for asking about the differing TPT and STS effects in Tables 1 vs 2. We follow the official MaPLe implementation of Khattak et al. (CVPR 2023), which provides three pretrained prompt sets obtained via few-shot prompt learning on ImageNet, each corresponding to a different random seed. To avoid arbitrarily favoring any single seed, we associate one of these three weight sets with each of our three independent runs. In all tables, the entry “MaPLe zero-shot” therefore means “apply the respective ImageNet-trained MaPLe prompt to the target OOD benchmark without any additional adaptation.” We will state this explicit mapping in the revision and annotate the tables accordingly to remove any confusion.

Why TPT and STS can slightly degrade MaPLe on cross-dataset TTA (Table 2)?

Specialized base prompt: MaPLe’s prompt tokens are finely tuned to the ImageNet distribution. When they are transferred to datasets with different semantics or styles, those tokens already sit close to a local optimum for many classes. As a result, subsequent shift-based updates (TPT or STS) have limited head-room and can occasionally over-correct, producing a small drop.

For the natural ImageNet variants, the ImageNet-trained MaPLe prompt remains reasonably well aligned with the target domain. In this case STS identifies truly helpful low-rank principal directions, yielding a net gain, while TPT achieves positive results as well. We will incorporate this explanation into the manuscript to make the interaction between MaPLe, TPT, and STS clear across all benchmarks.

[Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.]

Question 1 (“inference-time” wording)

Thank you for pointing out the possible ambiguity. In the submission we used the term “inference” to denote the entire test-time adaptation pipeline:

The time it takes to produce a first prediction for a previously unseen image, including any optimization / update steps that the TTA method performs on that very image. We agree that “inference” is often reserved for a single forward pass through a frozen model. To avoid confusion, we will replace “inference time” with “end-to-end adaptation latency” throughout the paper.

Question 2 (“cross-dataset generalization”)

Thank you for drawing our attention to the possible confusion. In recent TTA literature the expression has been used in the following way:

This experiment was first designed in the TPT paper to compare few-shot prompt learning vs instance-specific prompt learning (without supervision).

In our case, there is no few-shot learning, so this would not apply. To avoid confusion with the few-shot prompt learning, we will rename this setting to fine-grained generalization in Table 2.

Typos & citation links.

All figure/table references and minor errors (Lines 176, 247, 253, 256, 306) will be corrected.

I would like to thank the authors for their answers.

W1: there are still missing details about the selection of the regularization loss weight $\lambda_R$.

W2: thanks, these answers make sense, I missed TPS.

W3: a1) thanks for your answer. The provided interpretation is intuitive, however it lacks theoretical rigor and/or empirical evidence, and intuition isn't sufficient for explaining the effectiveness of SPS on ImageNet-A. However, I would not penalize this point, as it might be too much to ask for a theoretical interpretation. The remark about TPS is interesting, it seems that in the specific case of ImageNet-A, methods that tune parameters in the deep layers are better than others tuning parameters in earlier layers. Also please note that [B], which fine-tunes the projection matrix of the vision encoder, presents also remarkably strong performance on ImageNet-A is for test-time adaptation (cf. Table 9), this adds more evidence to the effectiveness of fine-tuning parameters close to the latent space for this specific dataset. This discussion is crucial to include as it can trigger future research in understanding "what to fine-tune" for a specific dataset.

a3) [8] and [B] do not necessarily need labeled data. They can be applied to the same test-time adaptation framework in a straightforward manner. [B] already performs experiments on test-time adaptation and shows strong results compared to TPT by adopting the same framework but training the projection matrix instead of tuning the prompt. This is an extremely simple baseline that seems to be powerful and can put in question the necessity of SPS method. [8] doesn't show experiments on test-time adaptation but it can be applied to this setting in the same manner, as well as the simple regularized linear adapter proposed in [B] if black-box adaptation is required.

Thanks for the explanation about MaPLe. So in both Tables 1 and 2, MaPLe is trained on ImageNet (is it 16-shot? please specify this in the paper) and evaluated on variants (Table 1) and other datasets (Table 2). Please make sure to write this clearly in the paper.

Q2 I see that the word generalization propagated from TPT, because it was showing few-shot training then generalization vs. tuning the prompts on-the-fly for datasets that were OOD for the few-shot setting. Cross-dataset generalization or fine-grained generalization can be both misleading, again because for test-time adaptation, the source domain is the pretraining dataset of CLIP and SPS/TPT is performed on-the-fly for each test image in each dataset. I see that this naming started in TPT, but it's a bit misleading and it might be better not to propagate it. That being said, this point is minor.