Training-Free Test-Time Adaptation via Shape and Style Guidance for Vision-Language Models

Thanks for your comments, and we provide our feedback on questions (Q) and limitations (L) as follows.

Q1: "Novelty and Distinction from DeYO."

A1: Thanks for your feedback. We will provide a detailed comparison and clarification between our SSG and DeYO in the following points.

First, corresponding to (4), we give the theoretical analysis about generalizable factors for the training-free VLM TTA methods for the first time. We appreciate the insight and theoretical contribution from DeYO, and we formally mention DeYO in Line151 and Line154, but DeYO only considers the entropy minimization back-propagation scenarios rather than considering the training-free (cache-based) cases. We extend the theoretical analysis to the training-free methods, and we think it is not trivial to accomplish this extension. Specially, DeYO calculates the gradient $\partial \operatorname{Ent}_\theta(\mathbf{x})/{\partial w}$ when adapting through entropy minimization loss to determine the update of $w$, then determines the change in the gap between the mean logits of samples belonging to different classes. However, our SSG utilises the properties of the cache-based method, which can directly determine the update of $w$ by the stored visual feature $\mathbf{v}(x)$ without the derivations from the gradient of entropy, such as $\Delta\left(w \cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right)\right)=\Delta w(\mathbf{x})\cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right) \approx \mathbf{v}(\mathbf{x}) \cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right)$, where stored visual feature can be regarded as the residual update of the raw classifier. Then we can obtain the change in the gap between the mean logits of samples belonging to the two classes as Equation 7 in the Appendix. Though the conclusion is the same, the core derivation of the two methods is different, as shown in Line15-Line21 in the Appendix, and we reveal the theoretical support for the cache-based (training-free) methods in a reasonable way, which can not be intuitively supported by DeYO theory without any modification.

Second, corresponding to (3), our SSG is designed for the training-free VLM TTA settings, which are different from the DeYO settings. DeYO aims to filter samples based on PLPD to benefit the entropy back-propagation, however, our SSG does not demand the entropy back-propagation. Our SSG aims to use PPD to highlight the augmented views in a novel reweight way, and supplement generalizable factors for the cache update. Based on the theoretical analysis, it is natural to highlight "lower entropy and higher PPD", but in terms of setting and specific operations, our SSG is different from DeYO. Moreover, DeYO only conducts the experiments on the single-modality scenarios, while our SSG conducts and verifies our design on large-scale vision-language models (CLIP) with promising performance and efficiency.

Third, corresponding to (2), though SSG (PPD) and DeYO (PLPD) introduce the generalizable factors, our SSG not only considers the shape-sensitive factors but also involves the style-insensitive factors. Style-related factors are important for the transfer learning methods, which have been proven in MixStyle[1], FourierDG[2], DSU[3] and more works. Therefore, the introduction of style-insensitive factors is important for the generalizable factors, which makes it more comprehensive and sound. Meanwhile, we have conducted the ablation results about style-insensitive factors in Figure 3 left in the manuscript, and style-insensitive factors show a promising improvement, which is a good supplement to their importance. Furthermore, we supplement more ablation results in the following A2, and get the same conclusion.

Corresponding to (1), we acknowledge that PPD measures prediction differences to reveal sample dependencies on specific factors, which is similar to DeYO's formula. Meanwhile, we also think it is a common and efficient operation which has been deployed in many different transfer learning fields like [4] [5], and it is reasonable to use this operation to determine the generalizable factors.

Based on the above main points, we think our SSG is different from DeYO, and we will give more discussions about DeYO to make it more explicitly credited in Sections 3.1 and 3.2 in the final manuscript.

[1] Domain generalization with mixstyle, 2021ICLR; [2] A fourier-based framework for domain generalization,2021CVPR; [3] Uncertainty modeling for out-of-distribution generalization, 2022ICLR; [4] Style Normalization and Restitution for Generalizable Person Re-identification, 2020CVPR; [5] Generalized lightness adaptation with channel selective normalization, 2023ICCV.

Q2: "Experimental Contributions. If the authors can show that the inclusion of style significantly enhances the performance or robustness of the model in ways that shape alone cannot, this would to some extent address concerns about limited novelty."

A2: Thanks for your suggestion. We have conducted detailed ablation results about the effect of PPD_st (style-insensitive factors) in Figure 3 left in the manuscript, and we supplement the average OOD accuracy based on CLIP-RN50 for further support. In the below table, the baseline method does not contain style-insensitive and shape-sensitive factors, PPD_st refers to style-insensitive factors, PPD_sh refers to shape-sensitive factors, and PPD refer the full model.

From the above table, it is obvious that incorporating only the style-insensitive factors (PPD_st), the performance is improved from 46.33%/63.46% to 46.85%/64.08%, and further incorporation with shape-sensitive factors, the performance further enhance. The improved performance obviously demonstrates the effectiveness of the inclusion of style, and makes the generalizable factors more comprehensive and sound.

L1: "The authors should provide a detailed comparison of computational efficiency with other training-free methods, highlighting any specific optimisations or improvements that their method offers."

A3: Thanks for your suggestions.

First, SSG utilised shape and style augmentation is extremely simple to implement. For the shape augmentation, we use the patch-shuffle operation, which can be efficiently implemented by erarrange func in einops packages. For the style augmentation, we use the colour transformation with hue adjustment, which can be easily implemented by the kornia package.

Second, we apply the shape/style augmentation to the raw input images before the model forward procedure, and we concatenate them into a batch to feed into the model, which can be considered as the processing time is nearly similar.

Third, after model feedforward and obtaining the logits, the following PPD calculation, reweighting and cache update is a simple mathematical calculation, which does not introduce much more inference burden.

In conclusion, our SSG shows the promising inference efficiency, and we give the detailed inference time comparison below.

It is obvious that SSG needs more data processing time due to the shape/style augmentation, and the other two parts need similar inference time, but the increased inference time is relatively small. Furthermore, we supplement the efficiency comparison in ImageNet-V, which contains 10000 images, in the following table.

From the above table, even for the 1w image scale, our SSG only increase 1 minute compared with TDA while improving the OOD accuracy by about 1.3%. At the same time, our SSG only demands 56% inference time compared with DEP while achieving better performance.

L2: "It is reasonable not to heavily use augmented views in test-time adaptation, as such operations (e.g., cropping, colour perturbations) may lead to the loss or displacement of key semantic information. Forcing consistency across these augmented views can suppress the model's ability to discern fine-grained semantics, thereby affecting the final prediction."

A4: Thanks for your comments. We do not force consistency across the raw images and their shape or style augmented images. Actually, we use shape and style augmentation to corrupt the raw images' shape and style information, aiming to measure the shape-sensitive and style-insensitive factors to further represent the generalizable factors.

We sincerely thank you for your constructive comments, and we provide our feedback on weaknesses (W) and questions (Q) as follows.

W1&W2&Q1: "How does PPD and the theoretical analysis differ from DeYO's work, and shouldn't this prior work be more explicitly credited rather than claimed as a contribution?"

A1: Thanks for your feedback. We will depict the difference between our SSG and DeYO in the following points.

First, we give the theoretical analysis about generalizable factors for the training-free VLM TTA methods for the first time. We appreciate the insight and theoretical contribution from DeYO, and we formally mention DeYO in Line151 and Line154, but DeYO only considers the entropy minimization back-propagation scenarios rather than considering the training-free (cache-based) cases. We extend the theoretical analysis to the training-free methods, and we think it is not trivial to accomplish this extension. Specially, DeYO calculates the gradient $\partial \operatorname{Ent}_\theta(\mathbf{x})/{\partial w}$ when adapting through entropy minimization loss to determine the update of $w$, then determines the change in the gap between the mean logits of samples belonging to different classes. However, our SSG utilises the properties of the cache-based method, which can directly determine the update of $w$ by the stored visual feature $\mathbf{v}(x)$ without the derivations from the gradient of entropy, such as $\Delta\left(w \cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right)\right)=\Delta w(\mathbf{x})\cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right) \approx \mathbf{v}(\mathbf{x}) \cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right)$, where stored visual feature can be regarded as the residual update of the raw classifier. Then we can obtain the change in the gap between the mean logits of samples from two classes as Equation 7 in the Appendix. Though the conclusion is the same, the core derivation of the two methods is different, as shown in Line15-Line21 in the Appendix and stated in Line 165 in the manuscript. We reveal the theoretical support for the cache-based (training-free) methods in a reasonable way, which can not be intuitively supported by DeYO theory without any modification.

Second, our SSG is designed for the training-free VLM TTA settings, which are different from the DeYO settings. DeYO aims to filter samples based on PLPD to benefit the entropy back-propagation. However, our SSG does not demand the entropy back-propagation, and aims to use PPD to highlight the augmented views in a reweight way, and supplement generalizable factors for the cache update. In terms of setting and specific operations, our SSG is different from DeYO. Moreover, DeYO only conducts the experiments on the single-modality scenarios, while our SSG conducts and verifies our design on large-scale vision-language models (CLIP) with promising performance and efficiency.

Third, though SSG (PPD) and DeYO (PLPD) introduce the generalizable factors, our SSG not only considers the shape-sensitive factors but also involves the style-insensitive factors motivated by the MixStyle, which makes the generalizable factors more comprehensive and sound. And the ablation results about style-insensitive factors in Figure 3 left in the manuscript show the promising improvement, which is a good supplement to style-insensitive factors, as denoted in A5. As for the specific operations of PPD, we think it is common to measure prediction differences to reveal specific factors.

Based on the above main points, we think our SSG is different from DeYO, and we will give more discussions about DeYO to make it more explicitly credited in Sections 3.1 and 3.2 in the final manuscript.

W2-1: "The paper lacks comparison against straightforward augmentation baselines that combine geometric and photometric transformations."

A2: Thanks for your valuable comments. We supplement the comparison experiments between: (a) straightforward augmentation baselines, which means our SSG model without PPD-related operations but considers geometric and photometric transformations in the augmentations, and (b) our full SSG model.

From the above table, we can find that our full model shows an obviously better performance than model (a). The reason is that geometric and photometric augmentation usually does not provide a more accurate prediction than the regular augmentation as shown in TPT/TDA, as they cause perturbations to the original image.

W2-(2)&Q2-(1): "Can you provide comparisons against simple baseline augmentations (e.g., random combinations of Gaussian noise and rotations) with PPD computation to demonstrate that your specific perturbation choices are essential?"

A3: Thanks for your suggestions. We supplement the comparison experiment between (a) PPD with Gaussian noise and rotations, and (b) PPD with shape patch-shuffle and style colour transformation with hue adjustment.

From the above table, we can find that model (b) gives a much better performance than model (a). In fact, to obtain the PPD_st and PPD_sh, we need to corrupt/perturb the image's shape/style information, and patch-shuffle and colour transformation are suitable to reach this goal, because the augmented images obviously lose or change the shape/style information. But for the rotation and Gaussian noise augmentation, the shape information is perturbed slightly and the more than style information is destroyed, which even gives a negative impact on the performance.

Q2-(2): "Additionally, please clarify whether the entropy computation uses the same shape and style perturbations as PPD, or different augmentations such as those employed in TPT?"

A4: Thanks for your question. We follow the augmentation in TPT/TDA to make the 63 augmented image views, and compute the entropy for these 64 (along with the raw image) images. For obtaining PPD_st and PPD_sh, we apply shape and style perturbations to the selected images (with high confidence) from 64 views, as described in Line182-Line183 and Line194-Line195.

W4&Q3: "Can you provide the missing ablation studies for ResNet-50, particularly showing the individual effects of PPD_sh and PPD_st, as well as the impact of patch shuffling on CNN-based architectures?"

A5: Thanks for your feedback. We supplement the ablation results with average OOD accuracy based on the CLIP-RN50 backbone.

First, we supplement the ablation results about shape-sensitive and style-insensitive factors. Furthermore, we supplement the baseline method performance as described in A8. From the table, both shape-sensitive and style-insensitive factors give the better performance, while the combination gieves the best performance.

Second, we report the ablation results about hyperparameters (patch number) on the shape perturbation (patch shuffle) below. From the table, the same conclusion is show as patch number as {4, 6} gives the better performance, and we set patch number as 4 for all experiments.

Third, to show the effect of the patch-shuffle, we supplement the ablation experiments about the different shape perturbation types as shown below, corresponding to the results in Table 1 in the appendix. It is obvious that patch-shuffle gives the best performance. The reason is that, pixel-shuffle is difficult to clarify the background and object, center-occlusion misses the image where the object is not at center, and rotation does not impact the object shape information efficiently.

W5: "Typos."

A6: Thanks for your comments. We will correct all typos in the final version.

W6: "Target Distribution Access."

A7: Thanks for your clarification. We will modify the Line91 in Section 2 to "...yet they still require training on the target data with ground-truth labels. Differently, SSG aims to ..., where the model does not demand training on the target data with ground-truth labels.".

W7: "Method Foundation."

A8: Thanks for your feedback. We build our SSG on the naive cache-based method, and this baseline method only contains one positive cache and can be considered as the cache-based method TDA without a negative cache, which is different from the entropy-based TPT.

For the cache implementation details of baseline and our SSG, we follow the TDA default setting for the positive cache. Specifically, the utilised cache is a dynamic key-value cache, whose memory size is 3 for all datasets following TDA. For the cache collection process, the key-value cache is initially empty and then accumulates a sufficient number of key-value pairs during the test-time adaptation.For the similarity function in adapted prediction, we utilise Eq.2 in the manuscript following TDA and Tip-Adapter.

We will add the method foundation in the Subsection "Implementation details".

We sincerely thank you for your great efforts and insightful questions. We provide our feedback on weaknesses (W) and questions (Q) as follows.

W1: "The method assumes shape features are always more reliable than style, but that’s not always true."

A1: Thanks for your valuable feedback. Motivated by Shape-Bias[1] and MixStyle [2], SSG utilizes the shape features (shape-sensitive factors and style-insensitive factors) to represent the generalizable factors, aiming to improve model's robustness. Meanwhile, we agree that there are always some tasks (e.g., texture-based medical image analysis) that depend on style information, and we will supplement this limitation in the "Conclusion" section.

[1] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, 2018ICLR; [2] Domain generalization with mixstyle, 2021ICLR.

W2-(1)&Q3: "The comparison result on the ImageNet dataset is missing."

A2: Thanks for your suggestion. We supplement the SSG performance on the ImageNet below.

As shown in the above table, SSG/SSG+ gives superior performance compared with training-free methods TDA/BoostAdapter.

W3&Q6: "Too many abbreviations (SSG, SHS, SIS, PPD, STI) in the paper."

A3: Thanks for your valuable feedback. For simplicity, we standardize the acronyms for "style-insensitive factors" as STI and "shape-sensitive factors" as SHS. Meanwhile, we maintain the SSG for the "style and shape guidance", and modify the perturbed prediction difference from PPD to SSG-score, while naming the operation "PPD reweight" as SSG-RW and "PPD comparison" as SSG-CP. We will modify all abbreviations in the final manuscript to make it clear.

Q1: "How would you describe the SSG method, is it a cache with an adapter? What is the cache capacity? Can you please explain the cache collection process?"

A4: Thanks for your feedback. As you pointed out, our SSG can be described as a cache-based method with an adapter. As described in Section 3.1, SSG contains the dynamic visual cache, and SSG's final prediction is the combination of zero-shot prediction $P(f_{\mathrm{test}})$ from Equation 1 and the adapted prediction $P_{cache}(f_\mathrm{test})$ from Equation 2.

We supplement more details about cache design to clarify. Following the positive cache in TDA, we utilize the dynamic key-value cache for SSG, and keep the cache capacity as 3 for all datasets. The key-value pair corresponds to the testing features $f_{\mathrm{test}}$ and the pseudo label $\hat{l}$ from the prediction $P(f_{\mathrm{test}})$.

We supplement more details about cache collection for clarify. For the cache collection process, the key-value cache is initially empty and then accumulates a sufficient number of key-value pairs during the test-time adaptation. SSG progressively incorporates test predictions with lower update criterion while limiting cache capacity. Specifically, if the number of the collected key-value pair is less than the cache capacity (3 for SSG), current testing key-value will add to the cache directly. If the number of collected key-value pair is more than the capacity, SSG will choose the existing key-value pair with highest update criterion, and compare it with the current key-value pair, and add the key-value pair with lower update criterion to the cache.

The update criterion is set to entropy of prediction $\mathrm{H}(P(f_{\mathrm{test}}))$ for previous cache-based method such as TDA or BoostAdapter, and our SSG modifies the update criterion as the combination of entropy and PPD (SSG-score), such as $\mathrm{H}(P(f_{\mathrm{test}})) - \mathrm{PPD}$ as described in last paragraph (Line213 - Line218) in Section 3.2. Meanwhile, SSG only contain one cache without any other negative cache like TDA.

Following your suggestion, we will add more introduction at the beginning of Section 3.2.

Q2: " How are entropy and PPD criteria combined? Does a higher PPD indicate better?"

A5: Thanks for your valuable question. For the first question, we combine the entropy and PPD criteria by the minus operation, such as $\mathrm{H}(P(f_{\mathrm{test}})) - \lambda*\mathrm{PPD}$. Through this operation, a sample with low entropy and high PPD will be cached just as you suppose. We set $\lambda$ as 1 for all experiments, and the ablation experiments about the $\lambda$ are shown in Section B.2 in the appendix, which are shown below.

For the second question, a higher PPD means the sample contains more generalizable factors (shape-sensitive and style-insensitive factors), which is complementary to the entropy criteria for better performance. The ablation experiments about the effect of PPD reweighting (SSG-RW) and PPD comparison (SSG-CM) in Figure 3 in the manuscript show the improvement from the high PPD, and we show the results here.

Q4: "Results of the SSG+ method in Table 3."

A6: Thanks for your question. We supplement the results of the SSG+ method in Table 3 below, and SSG+ shows the better performance in almost all datasets compared with SSG in Table 3 in the manuscript.

Q5: "Equation 4 lacks intuition."

A7: Section A in the appendix shows more evidence for Equation 4, and we briefly summarise here.

Equation 4 in the manuscript (Equation 10 in the appendix) is derived from the definition of the harmful samples. We define the harmful sample as one that reduces the differences in the mean logits between classes during the test-time procedure. Intuitively, we want the margin or differences between classes larger as much as possible, to obtain robust classification results.

Based on this definition, we first use the properties of training-free (cache-based) VLM TTA to obtain the change in logits due to model's update in Equation 6, shown as $\Delta\left(w\cdot\mathbf{v}\left(\mathbf{x}^{\text {test}}\right)\right)=\Delta w(\mathbf{x})\cdot\mathbf{v}\left(\mathbf{x}^{\text {test}}\right) \approx \mathbf{v}(\mathbf{x}) \cdot \mathbf{v}\left(\mathbf{x}^{\text {test}}\right)$, where stored visual feature can be regarded as the residual update of the raw classifier. Then we obtain the change in the gap between the mean logits of samples from two classes as Equation 7, shown as $\mathbf{v}(\mathbf{x})\cdot(E_{X_{+1}^{{test}}}[\mathbf{v}(\mathbf{x}^{{test}})]-E_{X_{-1}^{{test}}}[\mathbf{v}(\mathbf{x}^{{test}})]).$ Finally, based on the disentangled factors, we approximate Equation 7 to Equation 10 (Equation 4 in the manuscript) as $\mathbf{v_pp}(\mathbf{x})\cdot(E_{X_{+1}^{ {test }}}[\mathbf{v_pp}(\mathbf{x}^{ {test }})]-E_{X_{-1}^{{test}}}[\mathbf{v_pp}(\mathbf{x}^{{test}})])+\mathbf{v_pn}(\mathbf{x})\cdot(E_{X_{+1}^{ {test}}}[\mathbf{v_pn}(\mathbf{x}^{{test}})]-E_{X_{-1}^{{test}}}[\mathbf{v_pn}(\mathbf{x}^{{test}})]).$

To make the differences large, we need to make Equation 10 positive. According to Equation 9, the term multiplied by v_pp is positive, while the term multiplied by v_pn is negative, so we need to highlight the generalizable factors v_pp. Meanwhile, according to Equation 10, the sample with high confidence but with significant v_pp factors is also harmful for the differences between the two classes.

Q7: "Can you show in Figure 3 the effect of reweighting and a comparison using the ViT backbone?"

A8: We supplement the average OOD accuracy results about the effect of reweighting (SSG-RW) and comparison (SSG-CM) based on the ViT-B/16 backbone below. According to the results, we can find that both reweighting and comparison operations give an improvement, and a combination of them gets the best performance.

Moreover, we give the OOD average accuracy for the ablation results in the middle of Figure 3 to make the ablation consistent. This ablation experiment is about the patch number on the shape perturbation (patch shuffle). According to results, we set the patch number to 4 for all experiments.

Q8&Typos: "There is no reference to Figure 1 in the manuscript. And Formatting Concerns."

A9: Thanks for your valuable suggestions. We refer the Figure 1 in Line203, and we will move it to the first paragraph of Section 3.2 in the final version. And we will correct all typos in the final version.

We appreciate your thorough review of our paper, along with your constructive comments. We provide our feedback on weaknesses (W) and questions (Q) as follows.

W1: "While the paper is thoughtfully designed and thoroughly evaluated, the core idea feels like an incremental improvement of existing cache-based TTA techniques."

A1: Thank you for the approval of our thoughtful design and thorough evaluation. While our SSG is easy to implement with existing cache-based VLM TTA methods, we do not think it is an incremental improvement.

First, SSG reveals the effect of generalizable factors for the cache-based (training-free) VLM TTA, which has not been studied before. Similar to TDA (2024CVPR) or BoostAdapter (2024NeurIPS), our SSG gives a new perspective for the cache-based TTA framework, and the corresponding experimental results prove the effectiveness.

Second, we give the theoretical analysis for the generalizable factors under training-free TTA scenarios. Different from the entropy minimization scenarios, it is not trivial to derive the proof based on the properties of cache-based methods, as shown in Line15 - Line21 in the Appendix.

Third, SSG gives a potential direction to explore a further cache-based approach by designing more advanced generalizable factors.

W2&Q5: "Efficiency concern. Could you please clarify whether forward passes are reused or shared across branches?"

A2: Thanks for your thoughtful feedback. The forward passes are indeed shared across branches in our SSG, and we give the detailed explanations below.

First, SSG utilised shape and style augmentation is extremely simple to implement. For the shape augmentation, we use the patch-shuffle operation, which is efficiently implemented by erarrange func in einops packages. For the style augmentation, we use the colour transformation with hue adjustment, which is easily implemented by the kornia package.

Second, SSG applies the shape/style augmentation to the raw input images before the model forward procedure, and we concatenate them into a batch to feed into the model, thus the forward passes can be shared and maintain the forward time nearly unchanged.

Third, after model forward and obtaining the logits, the following PPD calculation, reweighting and cache update are simple mathematical calculations, which do not introduce much more inference burden.

Finally, we give the detailed inference time comparison below. SSG needs a little more data processing time due to the shape/style augmentation, and the total increased inference time is relatively small.

Furthermore, we supplement the efficiency comparison in ImageNet-V, which contains 10000 images, in the following table.

From the above table, even for the 1w image scale, our SSG only increase 1 minute compared with TDA while improving the OOD accuracy by about 1.3%. Our SSG only demands 56% inference time compared with DEP while achieving better performance.

W3&Q1: "Cache design ambiguity."

A3: Thank you for pointing it out. We build our SSG on the naive cache-based method, which only contains the positive cache and can be considered as the TDA without a negative cache.

For the cache implementation details, we follow the TDA default setting for the positive cache. Specifically, the utilised cache in SSG is a dynamic key-value cache, whose memory size is 3 for all datasets. For the similarity function in adapted prediction, we utilise Eq.2 in the manuscript following TDA and Tip-Adapter.

Last but not least, we combine BoostAdapter with SSG as SSG+. In SSG+, we only apply our SSG operations for the positive cache in BoostAdapter, while keeping the negative cache and related operations identical to BoostAdapter.

W4&Q2: "Missing augmentation-only ablation."

A4: Thanks for your suggestions. We supplement the augmentation-only ablation results below. Model-(a) means our SSG applying shape and style perturbation without PPD-based reweighting or cache updates, and model-(b) means our full SSG.

From the above table, we can find that our full model shows an obviously better performance than model (a).

W5&Q3: "Details and analysis of k*."

A5: Thanks for your questions. We follow the operation in TPT and TDA, choosing k* as the top 10%, and we keep the k* value for all experiments. We supplement the ablation results about the k* value, as shown below.

As shown in the above table, we can find that setting k* as the top-10% achieves the best performance, which is consistent with the conclusion from TPT.

W6: "Inconsistent ablations."

A6: Thanks for your valuable suggestions. We unify the ablation experiment based on the ViT-B/16 backbone and report the average OOD accuracy.

First, we report the ablation results about shape-sensitive and style-insensitive factors. As shown in Fig.3 in the manuscript, we have given the comparison results, and we summarise below, while we supplement the baseline performance on which we built our SSG as stated in A3.

Second, we report the ablation results about hyperparameters (patch number) on the shape perturbation (patch shuffle) below. From the table, the same conclusion is shown as patch number as {4, 6} gives the better performance, and we set patch number to 4 for all experiments.

Third, the ablation results about the combination hyperparameters between PPD_st and PPD_sh are shown in Line317 in the manuscript, and we omit them for saving space.

Fourth, we report the ablation results about the effect of the PPD operation below. According to the results, we can find that both reweighting and comparison operations give an improvement, and a combination of them gets the best performance.

W7: "Missing statistical variance in main results. Reporting mean ± std across runs would improve transparency and trust in the reported gains."

A7: Thanks for your feedback. We supplement the statistical variance and show it below. Due to the space limitation, more variance will be provided in the final manuscript.

W8-(1)&Q4: "When computing PPD, are the same top-k low-entropy indices used across the original, shape-perturbed, and style-perturbed augmentations?"*

A8: We indeed utilise the same top-k* low-entropy indices used across the original, shape-perturbed and style-perturbed augmentations.

W8-(2): "Rename the subsection."

A9: We will rename this subsection as "Hyperparameters about the combination between PPD_st and PPD_sh" to clarify.

W8-(3): "Typos."

A10: We will accordingly modify the typos and standardise the acronyms for style-insensitive factors as STI.

Q6: "Have you tested SSG on other vision-language models?"

A11: Thanks for your constructive comments. Our SSG can easily be applied to other vision-language models, and we follow DPE to involve the OpenCLIP (ViT-L/14) as an example for a fair and public comparison. We show the comparison results below.

We can observe that our SSG still outperforms training-free TDA by 1.57%/ on average across 4 datasets, showing that our method generalizes well to larger-scale VLMs. And more vision-language models such as SigLIP or DFN will be explored in the future.

Q7: "The ensemble row is somewhat ambiguous."

A12: We follow TPT and DPE to involve the "Ensemble" as the comparison method. "Ensemble" means the baseline zero-shot CLIP prediction using the ensemble of 80 hand-crafted prompts from [1].

[1] Learning transferable visual models from natural language supervision. ICML 2021.