AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation

We sincerely appreciate your constructive feedback. Please allow us to address your concerns below.

W1: I'd like to see performance-efficiency trade-off under few-shot settings.

Thank you for your suggestion. We presented this experiment in Figure 2 of our attached PDF. In comparison to the zero-shot scenario, the few-shot settings incorporate two additional adapter modules at each transformer layer to facilitate transfer learning. This configuration results in our method being initially slower than other prompt learning approaches. This slowdown is likely due to the adapters needing sequential processing, whereas the prompts are processed in parallel. Taking into account the computational costs, we propose that AWT could be more efficiently transferred using test-time zero-cost methods like LoRA [1], which allows for structural reparameterization after training.

W2: How do the authors decide the hyper-parameters?

Thank you for your question. As detailed in Table 4 of our paper, AWT involves four main hyper-parameters: number of augmented image views ($N$), number of generated class descriptions ($M$), and the weighting temperatures $\gamma_v$ and $\gamma_t$. We maintain consistent hyper-parameter values across all scenarios—whether zero-shot or few-shot, and for both image and video tasks. This uniformity is due to our primary focus on the zero-shot capabilities, where typically no validation set is available for further tuning of hyper-parameters. Here is how we determined these values:

• We used a subset of the ImageNet training set as the initial and only validation set for selecting hyper-parameters. We chose the hyper-parameter values based on the accuracy:
  • For $\gamma_v$ and $\gamma_t$, we conducted a broad search and observed optimal performance around the value of 1/2, which we adopted as the default value.
  • For $N$ and $M$, our selection process was similar, but we opted for the most cost-effective values rather than the absolute best. This decision was based on the observation that while increasing $N$ and $M$ can marginally improve results, it also significantly raises the computational costs associated with querying GPT.

W3: Why choose flipping as one of the image augmentation strategies?

You are correct in noting that cropping alone can capture fine-grained information. However, we believe that flipping introduces an additional layer of diversity in visual perspectives, which can potentially enhance the cross-modal matching process. To support our choice, we've conducted an ablation study detailed below. Our findings suggest a modest improvement, but given the minimal additional cost, we have decided to retain this method.

We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.

[1] Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022.

We sincerely appreciate your insightful feedback. Please allow us to address your concerns below.

W1: 1) The dataset description is a slight limitation. 2) Results for prompting without dataset description. 3) The LLM version for AWT, VisDesc [1], and CuPL [2] might differ.

Thank you for raising these issues.

• Dataset Description Limitation: We acknowledge this limitation. However, to enable the LLM to perceive visual domains (e.g., sketches), manually inputting this prior information is necessary. CuPL (in its full version) addresses this by manually designing multiple questions for each dataset. We have streamlined this process by directly providing LLMs with dataset-level descriptions, enabling them to automatically generate dataset-tailored questions.
• Prompting without Dataset Description: Your suggestion to explore scenarios without dataset descriptions is valuable, considering real-world applications where such information might not always be available. We have designed a "base" version for AWT where we replaced the instruction in Lines 146-147 with "Generate questions to classify images." while keeping other parts unchanged.
• LLM Version Consistency: For VisDesc, we regenerated the descriptions using GPT-3.5 (the same version as used for AWT) using their official code. For CuPL, the descriptions were initially from SuS-X [3] and generated by GPT-3. We apologize for this oversight. To correct this, we have regenerated CuPL descriptions with GPT-3.5 and implemented both base and full versions of CuPL.

In summary, we now provide additional results for AWT (base) and GPT-3.5-based CuPL (base and full) in the table below.

W2: Results for weighted averaging are missing.

Thank you for pointing this out. We aimed to demonstrate the significance of OT by contrasting A(Img.+Name) and A+T in Table 4(a). We have also included the results for weighted averaging below for your reference.

W3: The related work section could be significantly extended.

We apologize for the oversight in our related work section. CLIP-Adapter [4] and Tip-Adapter [5] are pioneering studies proposing an adapter module and a training-free cache module for few-shot adaptation, respectively. APE [6] further leverages a CLIP prior for initialization in a more efficient manner. We will incorporate these studies into our related work section. Moreover, to enable a comparison between APE and AWT, we implemented APE-T (its highest-performing variant) using the CLIP-base/16 architecture across 11 few-shot learning datasets, utilizing the official code. We detail this comparison in Figure 1 of our attached PDF. APE-T will also be included in Figure 3 of our paper.

Minor: Are $\gamma_v=2$ and $\gamma_t=2$ or are the equal to 1/2 as in tables 4e and 4f?

We apologize for the confusion; both $\gamma_v$ and $\gamma_t$ are indeed set to 1/2.

We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.

[1] Menon, Sachit, and Carl Vondrick. "Visual Classification via Description from Large Language Models." ICLR 2023.

[2] Pratt, Sarah, et al. "What does a platypus look like? generating customized prompts for zero-shot image classification." ICCV 2023.

[3] Udandarao, Vishaal, Ankush Gupta, and Samuel Albanie. "Sus-x: Training-free name-only transfer of vision-language models." ICCV 2023.

[4] Gao, Peng, et al. "Clip-adapter: Better vision-language models with feature adapters." IJCV 2024.

[5] Zhang, Renrui, et al. "Tip-adapter: Training-free clip-adapter for better vision-language modeling." ECCV 2022.

[6] Zhu, Xiangyang, et al. "Not all features matter: Enhancing few-shot clip with adaptive prior refinement." ICCV 2023.

We appreciate the time and effort you have dedicated to reviewing our paper. Please allow us to address your concerns below.

W1: Relationship and differences between prior OT methods and AWT.

Thanks for your comment. For a detailed response, please refer to our responses to common reviewer concerns. We will include this discussion in the final version of our paper.

W2: Some recent works should be acknowledged and discussed.

Thank you for bringing these excellent studies to our attention. We have carefully considered their relevance with AWT and plan to include the following discussions in our revised paper:

• DMN [1] introduces a versatile adaptation framework suitable for various settings. However, DMN's applicability is constrained as it necessitates historical test data and is limited to scenarios where test and candidate classes are identical. AWT, while effective, also has its limitations: it requires additional LLMs and incurs higher computational costs. Noted that AWT and DMN could be potentially combined: DMN could utilize its capability to store AWT's augmented image features in memory alongside LLM augmented textual features, then replace the final dot product with our proposed entropy-based weighted OT distance.
• MMA [2] introduces advanced adapter-based methods for VLMs. In zero-shot settings, MMA falls short compared to AWT due to its dependency on additional training data. In contrast, for few-shot adaptation/generalization settings, while AWT uses a standard adapter module, MMA introduces an advanced adapter that balances discrimination and generalization. Integrating MMA's adapter into AWT could potentially enhance performance.
• TDA [3] proposes the use of a memory cache to mitigate the extensive computational demands of test-time adaptation methods. TDA is akin to the zero-shot adaptation version of DMN, which necessitates historical test data from identical candidate classes. Based on our analysis alongside DMN, our AWT method is also orthogonal to TDA, and integrating these two approaches could potentially enhance performance.

W3: The computational complexity should be analyzed and compared against prior methods.

Thank you for your suggestions. We have discussed the computational limitations in Sections B.2 and F of our Appendix, and compared our method with previous methods in Figure 8 of our paper. Below, we provide additional details to more comprehensively address your concern:

• Augment and CLIP forward: Given that the text features can be pre-computed (as suggested in the CLIP paper), AWT theoretically involves computing complexity of $N+1$ times that of the original CLIP model, where $N$ is the number of augmented image views.
• Weight Calculation: This process is relatively swift and primarily involves several matrix operations such as Softmax function.
• Optimal Transport: Thanks to the efficient Sinkhorn’s Algorithm, we can solve the optimal transport problem with high efficiency.

The bulk of the computational effort is concentrated in the CLIP forward process. Thanks to the parallel processing capabilities of modern GPUs and efficient deep learning libraries like PyTorch, the computational cost of AWT remains manageable. Notably, AWT is significantly more efficient than test-time adaptation methods, which require model training during inference, such as TPT [4] (refer to Figure 8 of our paper). Moreover, the cost-performance trade-off can be easily modulated by adjusting $N$. We believe that these limitations can be further addressed through future research, as discussed in Lines 776-779 of our paper.

We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.

[1] Zhang, Yabin, et al. "Dual memory networks: A versatile adaptation approach for vision-language models." CVPR 2024.

[2] Yang, Lingxiao, et al. "MMA: Multi-Modal Adapter for Vision-Language Models." CVPR 2024.

[3] Karmanov, Adilbek, et al. "Efficient Test-Time Adaptation of Vision-Language Models." CVPR 2024.

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS 2022.

Thank you for your detailed review and insightful comments! Please allow us to address your concerns below.

W1 and Q1: 1) Random resized cropping and flipping is naïve. 2) Cannot see any improvement by visual augmentations. 3) What about applying more complex visual augmentations?

Thank you for raising these points.

• We opted for random resized cropping and flipping primarily due to their efficiency when compared to more advanced augmentation methods. For instance, employing diffusion models can significantly slow down inference speed, as discussed in the last part of Section 4.3 in DiffTPT [1], where inferring 10 test images takes 36 minutes. Meanwhile, methods involving data retrieval often require substantial storage space, such as LAION-5B used in SuS-X [2].

• The incremental improvements with visual augmentations are also noted by us, as mentioned in Lines 277-278 of our paper. Directly applying such augmentations can generate information-sparse image views. This motivates us to propose the weighting strategy. We provided an additional ablation study below, which demonstrates that, with the weighting module, even simple visual augmentations can yield significant improvements.

• We have explored more complex augmentations like diffusion models in Appendix Section D and found them superior to naive methods. Additionally, we included results for the EuroSAT dataset this time, summarized in the table below. However, given the efficiency concerns, we choose to use simple augmentation by default.

W2: Difference between prior OT methods and AWT is not clearly claimed.

We apologize for any confusion caused. For a detailed response, please refer to our responses to common reviewer concerns. We will ensure to clarify any ambiguities in the final version of our paper.

Q2: Why only 50 class descriptions are generated?

Thank you for your query. As detailed in Table 4(c), increasing the number of descriptions from 50 to 100 provides only marginal gains (0.03% and 0.14% improvements, respectively), while doubling the query costs for GPT. We believe that maintaining 50 descriptions strikes a good cost-effectiveness balance.

Q3: Any inference speed limitation?

Thank you for your question. We have discussed the inference speed limitations in Sections B.2 and F of our Appendix.

Q4: Are the evaluation results stable or constant?

The results are stable and almost constant. The stability of AWT is supported by error bar analysis (mean and variance) in Section B.1 of the Appendix. For each run, images are augmented using different seeds, and descriptions are regenerated. Your suggestion to explore the impact of different GPT versions is insightful. Following your advice, we conducted further experiments with an alternative GPT version and another LLM, and presented the results below. Overall, AWT maintains robust performance across these variations. An interesting observation is that more advanced LLMs do not necessarily yield significant performance improvements. This may be attributed to the fact that, in this scenario, AWT primarily relies on the simple instruction-following and knowledge-recall capabilities of LLMs, making models like GPT-3.5 sufficient.

L1: No limitation discussion in the original paper.

Thank you for your comment. The limitations have been discussed in Section F of the Appendix.

We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.

[1] Feng, Chun-Mei, et al. "Diverse data augmentation with diffusions for effective test-time prompt tuning." ICCV 2023.

[2] Udandarao, Vishaal, Ankush Gupta, and Samuel Albanie. "Sus-x: Training-free name-only transfer of vision-language models." ICCV 2023.

Dear Reviewer CYNL,

We greatly appreciate the time and effort you have dedicated to reviewing our submission. As the discussion phase is set to conclude in 48 hours, we wish to ensure that all your queries and concerns are thoroughly addressed. Please do not hesitate to reach out if there are additional aspects of our submission that you would like us to clarify or elaborate on.

Thank you once again for your insightful feedback.

Best regards,

Authors of Paper #8459

Dear Reviewer CYNL,

We would like to remind you that there are only 4 hours remaining in the discussion phase. Please let us know if all your concerns and questions have been addressed satisfactorily.

We look forward to your feedback.

Best regards,

Authors of Paper #8459