GS-Bias: Global-Spatial Bias Learner for Single-Image Test-Time Adaptation of Vision-Language Models

We greatly appreciate the reviewer’s encouraging and valuable comments on our paper. Below, we address the raised concerns.

Q1: The proposed global bias mechanism, which relies on global prediction consistency, is not a novel idea.

A1: We acknowledge that relying on global prediction consistency is not novel. However, our key innovation lies in introducing a learnable global bias at the output level, which effectively addresses the efficiency bottleneck in prior methods. Previous approaches require modifying model inputs, leading to substantial computational overhead. In contrast, our method operates purely at the output level, eliminating the need for expensive backpropagation while still ensuring effective test-time adaptation. Thus, our method strikes an effective balance between performance and efficiency. For instance, it improves cross-dataset and domain generalization by 2.23% and 2.72% over TPT, while using only 6.5% of its memory on ImageNet.

Q2: Could simply reducing the crop size yield a similar effect on spatial bias?

A2: Thank you for your question. In response, we conducted experiments on 11 datasets by reducing the crop size to $1\times1$ and $2\times2$ for spatial bias learning. The results show that our spatial bias effectively improves performance compared to GS-Bias without spatial bias, especially in unseen cross-dataset generalization (66.07% vs. 67.03%), demonstrating its ability to capture visual concepts missed by global bias. In contrast, simply reducing crop size does not achieve the same effect.

This is because we learn spatial bias from the spatial features of the vision encoder, where each region inherently encodes rich contextual information. In comparison, image-level cropping produces isolated patches that lack spatial coherence. Moreover, our region selection is guided by the correlation between spatial regions and class descriptions, whereas cropping is purely random.

Q3: Missed related works.

A3: We have conducted a comparison with state-of-the-art VLM TTA methods, including DPE, TDA, and HisTPT. As summarized below, TDA utilizes historical data streams to update a dynamic queue for training-free TTA, while DPE and HisTPT extend this by incorporating prototype learning and prompt tuning, respectively. In contrast, GS-Bias operates purely on a single image without accessing historical data, making it more aligned with TPT and MTA. Notably, GS-Bias remains compatible with TDA, allowing for potential integration.

To further clarify our advantages, we report the performance and memory cost of GS-Bias with TDA and DPE on ImageNet, and combine TDA and GS-Bias as a rough version with historical data flow. The results show that GS-Bias strikes a strong balance between performance and efficiency. Furthermore, when equipped with historical data flow, GS-Bias improves further without increasing memory usage. This is due to our bias learning operates solely at the output level, effectively avoiding the storage of gradient-bearing objects in the queue.

Q4: The performance of the original GS-Bias does not show substantial improvements over TPT and MTA.

A4: As shown in the table below, we aggregated the performance of the original GS-Bias across 15 datasets (Fig 1, Tab 1, and Tab 2 in the manuscript). With the same settings (BS=64), our method consistently outperforms others. More importantly, GS-Bias requires only 6.5% of TPT’s memory on ImageNet and achieving a $10\times$ speedup, demonstrating much higher efficiency. Overall, even in its original form, GS-Bias achieves an substantial improvement for the trade-off between performance and efficiency compared to TPT and MTA.

Q5: The positive effect of spatial bias is minimal?

A5: We elaborate on the effectiveness of spatial bias in A2. Regarding concerns about the performance gain, we clarify that spatial bias is designed to complement global bias, not to be used independently in CLIP. Since CLIP is trained on global features, its spatial distributions are smoother, leading to weaker gradient updates. Thus, applying spatial bias directly to CLIP has limited benefits.

Q6: More ablation studies on BS.

A6: We list all the results for BS=8,16,32,64, and 128 below. It turns out that a larger BS leads to better performance. GS-Bias outperforms both TPT and MTA even in the simplest setup.

We will include the added results and discussions in the final version as supplementary material.

Q1: A typo in the title of the paper.

A1: We sincerely appreciate your thorough and responsible review of our manuscript. We apologize for the typo caused by our oversight, and we will correct "GB-Bias" to "GS-Bias" in the final version.

Q2: Provide concrete inference examples to illustrate the effectiveness of the proposed method.

A2: Thank you very much for your insightful suggestions, which have encouraged us to present our work more intuitively. Following your advice, we have provided seven concrete inference examples, with images sourced from seven different datasets (ImageNet, ImageNet-A, ImageNet-S, Pets, Aircraft, Flowers102, and EuroSAT). For your convenience, we have placed these examples in the following anonymous link: Inference Examples.

We emphasize that GS-Bias consists of three key components: CLIP output, global bias, and spatial bias. To intuitively demonstrate the effectiveness of bias learning, we present the Top-3 probabilities and their corresponding categories for each component. The examples show that the combination of global and spatial biases effectively corrects the erroneous outputs of the original CLIP model.

Q3: The paper has limited discussion of potential failure cases or limitations.

A3: Thank you for your constructive comments, which has encouraged us to provide a more comprehensive discussion of GS-Bias. While GS-Bias achieves a well-balanced trade-off between performance and efficiency, one notable limitation is that the selection of hyperparameters is based on empirical choices. Although we performed ablation studies on the hyperparameters using as many datasets as possible and found that the model's performance is not sensitive to the hyperparameters, it is still necessary to reselect the hyperparameters for different test samples.

For instance, we set the learning rates of global and spatial biases to a fixed value of $\alpha = 1$ and $\beta = 1$ to achieve cross-dataset generalization. However, some samples may favor learning more from global information, while others may require a stronger focus on spatial information. An empirically fixed setting might lead to suboptimal adjustments. To illustrate this more intuitively, we provide two failure cases (link: Failure Cases), where fine-grained aircraft recognition tends to rely more on spatial information, whereas action recognition benefits more from global information.

Thus, we acknowledge that such empirical selection may not be optimal for every individual data sample, but it serves as a practical starting point. Future research could explore dynamic strategies for adjusting the balancing hyperparameters on a per-sample basis to further enhance model performance.

Once again, we sincerely appreciate your professional review. We will incorporate the discussion on potential failure cases and limitations into the final version and provide more concrete inference examples.

Q1: Provide a more comprehensive analysis of the number of spatial regions and the reasons for setting of the number of regions.

A1: Thank you for your insightful comments. In response, we have expanded our analysis by incorporating two new experiments:

• Exp1: Computing the number of spatial regions that are significantly related to the classification target across 11 datasets.
• Exp2: Extending Figure 3 by adding results for $K$ = 4, 128, and 196. (ViT-B/16 has 196 spatial regions)

Following Eq.10 and 11 in the manuscript, we select the top-K regions for spatial bias learning by computing the similarity scores $\boldsymbol{M}$ between all spatial regions and the class descriptions. Higher scores indicate stronger relevance to the classification target, while lower scores may correspond to irrelevant regions.

In Exp1, we normalize the similarity scores and consider regions with scores greater than 0.1 as significant, denoted by $\tilde{K} = \sum_{i} \mathbb{1} \left( \frac{\boldsymbol{M}_i - \min(\boldsymbol{M})}{\max(\boldsymbol{M}) - \min(\boldsymbol{M})} > 0.1 \right)$. Furthermore, we compute the average number of significant regions for each dataset, represented as $\tilde{K}_a$. The results indicate that the number of significant regions varies across different datasets. For example, the low-resolution satellite images in EuroSAT appear blurry, making spatial information less distinguishable and resulting in fewer significant regions. In contrast, fine-grained datasets such as pet and flower recognition provide a greater number of significant regions. Furthermore, we observe that the average number of significant regions across the 11 datasets is approximately 16.

Exp2 further confirms that as $K$ increases, performance initially improves, reaches a peak, and then starts to decline. This suggests that incorporating an appropriate number of spatial regions provides beneficial class-related information, whereas an excessively large $K$ (e.g., $K=196$) may lead to severe overfitting, causing the optimization to be trapped in irrelevant, misleading information.

In summary, we observe that $K$ = 16 achieved the best average performance and exhibited significant relevance, making it a reasonable and well-justified choice. We list the results as below.

• Exp1. The number of significant spatial regions $\tilde{K}_a$ across 11 datasets. |Method|Flower102|DTD|Pets|Cars|UCF101|Caltech101|Food101|SUN397|Aircraft|EuroSAT|ImageNet|Average| |-|-|-|-|-|-|-|-|-|-|-|-|-| |$\tilde{K}_a$|18.03|12.83|18.43|16.42|16.23|16.48|16.80|18.56|15.95|11.91|19.12|16.43|

• Exp2. Results of different numbers of spatial regions $K$ across 11 datasets. |$K$|0|4|8|16|32|64|128|196| |-|-|-|-|-|-|-|-|-| |10 Cross-Datasets|66.07|66.84|66.87|67.03|66.84|66.76|66.75|66.70| |ImageNet|70.45|70.52|70.51|70.57|70.48|70.42|70.38|70.35| |Average|68.26|68.68|68.69|68.80|68.66|68.59|68.57|68.53|

Q2: Comparison of efficiency in total computational time.

A2: Thank you for your valuable suggestions. To further demonstrate the practicality of our method, we have supplemented our analysis by reporting the total computation time of GS-Bias, MTA, and TPT across 11 datasets. Specifically, we set the augmentation batch size to 8 for cross-dataset generalization and 64 for ImageNet. All experiments were conducted on a single RTX 3090 GPU. The results indicate that GS-Bias achieves a significant speedup compared to TPT, while its total computational cost remains nearly identical to that of the parameter-free MTA. The detailed results are presented below.

• Comparison of efficiency in total computational time. |Method|Flower102|DTD|Pets|Cars|UCF101|Caltech101|Food101|SUN397|Aircraft|EuroSAT|ImageNet| |-|-|-|-|-|-|-|-|-|-|-|-| |TPT|4min|2min|5min|23min|6min|4min|60min|103min|6min|10min|660min| |MTA|1min|1min|1min|3min|1min|1min|12min|9min|1min|3min|85min| |GS-Bias|1min|1min|2min|3min|2min|1min|12min|10min|2min|4min|88min|

Q3: How GS-Bias performs on more complex and fine-grained tasks？

A3: Thank you for your insightful suggestion. The idea of GS-Bias can be extended to other foundation models for various downstream tasks. For example, in segmentation tasks, a learnable bias $B \in R^{1 \times W \times H \times C}$ can be incorporated into the output segmentation mask $M \in R^{1 \times W \times H \times C}$, making it an updatable mask $\tilde{M} \in R^{1 \times W \times H \times C}$. Applying a segmentation-specific test-time objective to $\tilde{M}$ facilitates efficient bias learning.

However, applying GS-Bias to more complex and fine-grained tasks requires designing a new test-time objective that aligns with the nature of the model and the specific downstream task. We plan to explore this direction further in future research.

We sincerely appreciate your profound comments and will incorporate the above discussion into the final version of the paper.