Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model

Response to Weaknesses

1. Performance of One-step Label Propagation

   Thank you for the suggestion on evaluating one-step versus iterative label propagation. As detailed in Appendix A.2.5, ECALP demonstrates increased accuracy with more iterations, as shown by our results on the DTD dataset. While a single iteration achieves 53.31% accuracy, three iterations enhance this to 54.49%, balancing efficiency and performance. This analysis highlights the trade-off between computational simplicity and accuracy, aiding practitioners in making informed choices for applications where computational resources are limited.

2. Combination with CoOp

   Thank you for your insightful question. In Appendix A.2.4, we explored integrating ECALP with traditional prompt learning methods like CoOp, using 1 to 16-shot samples as part of the graph rather than for training. Our results showed that combining ECALP with CoOp significantly enhances CoOp performance, demonstrating that ECALP can effectively complement traditional methods, further validating its applicability and versatility.

Response to Questions

1. Mistake in Figure 1
   We appreciate your careful review and apologize for the oversight. In Fig. 1, we have corrected the error by switching the positions of the second image encoder and the text encoder. Thank you for bringing this to our attention.

2. Multiple Textual Prompts

   For each category, multiple textual prompts are generated using various templates, such as 'a photo of a {}', 'a sculpture of a {}', and 'a rendering of a {}'. These templates are suggested by the original CLIP paper. This method produces multiple text embeddings, which we then average to achieve a robust text representation. Crucially, we apply the same set of templates to all categories within a dataset, ensuring consistency across classes.

We hope these additions address your concerns thoroughly and demonstrate the robustness and practicality of our approach. Thank you again for your feedback, which has significantly contributed to improving our manuscript.

Kind regards,
The Authors

Response to Weaknesses

1. Motivation of ECALP

   Our method is motivated by the need to enhance VLMs without task-specific tuning, leveraging test samples more effectively. We propose a graph-based approach for label-efficient adaptation, dynamically constructing graphs over text prompts, few-shot examples, and test samples. This allows for inference without the need for additional support sets, improving feasibility and memory efficiency. Our model includes context-aware feature re-weighting to address modality differences and enhance task adaptation accuracy. By supporting efficient graph expansion and iterative label propagation, our method enables real-time inductive inference. Extensive evaluations demonstrate superior performance across various tasks, including fine-grained categorization, style-transfer classification, and out-of-distribution generalization.

2. Method Implementation

   Thank you for your suggestion regarding implementation complexity. Our method is designed to be straightforward and does not require task-specific tuning. We have provided detailed implementation guidelines in Section 3.5, along with Algorithm 1. Additionally, we promise to release the source code upon acceptance, aiming to contribute to the community and foster the development of more advanced methods in the future.

3. Generalization to Significant Domain Shifts

   We appreciate the concern regarding the generalization of context-aware feature re-weighting across different tasks. Our approach leverages robust text and few-shot re-weighting, which combines prompt text embeddings with target distribution information, minimizing the impact of domain shifts. As shown in Appendix A.2.8, we provide visualizations of severe corruption from the ImageNet-C dataset, illustrating our method's resilience. Furthermore, as detailed in Appendix A.2.6, ECALP consistently outperforms CLIP-ResNet50 by 3%-5% across all corruption severity levels, including minimal levels, demonstrating its robustness and effectiveness even under significant domain shifts.

We hope these additions address your concerns thoroughly and demonstrate the robustness and practicality of our approach. Thank you again for your feedback, which has significantly contributed to improving our manuscript.

Kind regards,
The Authors

Response to Weaknesses

1. Limited Performance Improvement & Highlight Second-highest Performance

   We appreciate the reviewer's feedback on performance gains. Enhancing VLMs in a training-free setting is inherently challenging. Despite this, ECALP achieves a 1-2% accuracy improvement over state-of-the-art methods in tasks like fine-grained recognition, style transfer, and out-of-distribution scenarios. These improvements are notable given the absence of hyperparameter tuning, highlighting ECALP's robustness and practicality.

   Additionally, we have underlined the second-highest performance in Tab. 1, 2, and 3 for improved clarity. Thank you for this suggestion.

2. Clarify the Computational Efficiency Measurement

   In Tab. 6, the testing time for ECALP includes the entire process: dynamic graph construction, label propagation, data loading, and CLIP encoder operations. This comprehensive measurement demonstrates our method's computational efficiency in practical scenarios.

Response to Questions

1. CUDA Memory Comparison

   ECALP is designed to be memory-efficient. On the DTD dataset, ECALP requires approximately 2.4 GB of CUDA memory per test image, compared to 2.3 GB for the CLIP ResNet 50 baseline and 3.0 GB for ZlaP. In contrast, training-required methods like TPT consume about 21.9 GB during training. Our efficiency is primarily due to: (1) being training-free; (2) storing only the image features as graph nodes, each occupying just 4 KB of CUDA memory; and (3) implementing label propagation with sparse operations, which conserves memory.

2. Robustness to Incorrect Adjacency Graph Initialization

   To evaluate the robustness of our dynamically constructed graph for label propagation, we conducted experiments under various data-stream conditions, as detailed in Appendix A.2.7. We test ECALP's performance with three different test set configurations: Standard Random Sampling: A typical random sample of the test stream. Hard Samples First: The initial 5% of the test stream consists of only hard samples (those misclassified by CLIP baseline), followed by a random sampling for the remaining 95%. 10% Hard Samples First: The initial 10% of the test stream contains hard samples, with the rest being randomly sampled.

   The results below suggest a minimal performance drop, with less than a 0.4% decrease in accuracy even with a 10% hard sample initialization. This resilience is due to our graph's dynamic nature, where edge connections between data points are continuously updated as the test sequence progresses. This adaptability ensures that our method remains robust against poor initializations, preventing cumulative errors and maintaining high accuracy even under challenging conditions.

3. Performance on Different Corruption Severities

   ECALP remains effective at different severity levels in corrupted downstream tasks. As detailed in Appendix A.2.6, we examined ECALP's performance using the ImageNet-C dataset across severity levels from 1 to 5. ECALP consistently outperforms CLIP-ResNet50 by approximately 3%-5% across all severity levels. This demonstrates ECALP's robustness and effectiveness regardless of the corruption severity.

4. Impact of Label Propagation Iterations

   The performance of ECALP is related to the number of label propagation iterations $T$. As detailed in Appendix A.2.5, we explored varying T from 1 to 6 on the DTD dataset. The results suggest that slightly more iterations may improve the accuracy. This is due to a larger receptive field by more propagation iterations, which better utilize the test data's manifold structure. However, increasing T also increases computational costs, and we chose T = 3 to strike a balance between efficiency and accuracy.

The table below shows the accuracy at different iterations:

We hope these additions address your concerns thoroughly and demonstrate the robustness and practicality of our approach. Thank you again for your feedback, which has significantly contributed to improving our manuscript.

Kind regards,
The Authors

Response to Weaknesses

1. Mistake in Figure 1
   We appreciate your careful review and apologize for the oversight. In Fig. 1, we have corrected the error by swapping the positions of the second image encoder and the text encoder. Thank you for bringing this to our attention.

2. Contribution Relative to ZLaP
   While ECALP shares a similar foundational idea with ZLaP, specifically in leveraging data manifolds via label propagation, our approach introduces several novel contributions. These include i) reduced data requirements, ii) supporting few-shot adaptation, iii) context-aware edge re-weighting, iv) improved label propagation strategy, and v) improved performance. Notably, our zero-shot ECALP does not require an additional support set for graph construction, enhancing feasibility and memory efficiency in real-world applications. In contrast, a separate support set is required by ZLaP. ECALP also supports Few-shot Adaptation by incorporating few-shot samples into the graph with reweighting, unlike ZLaP, which is limited to zero-shot adaptation. To address modality differences and reduce irrelevant information impact, we introduce a Context-aware Edge Re-weighting. Our approach enhances label propagation efficiency through Iterative Label Propagation and Dynamic Graph Expansion, enabling low-cost inductive inference. We demonstrate superior performance across diverse tasks, including fine-grained datasets, style-transfer datasets, and OOD datasets.

3. Few-shot Performance on OxfordPets and Food101
   We appreciate the observation regarding performance variations on the Oxford Pets and Food101 datasets. We analyze the causes for the relatively poorer few-shot results on these datasets through a qualitative visualization of testing data features in Appendix A.4. Two datasets, Flowers102 and EuroSAT, are brought in for comparative studies. The testing sample features are projected into 2D space via t-SNE. We make the following observations. The testing samples are more cluttered and mixed up in OxfordPets and Food101 than in Flowers102 and EuroSAT. There are more separated sub-classes, i.e. isolated clusters within each semantic class, for OxfordPets and Food101. This suggests having a few-shot labeled samples is less effective for OxfordPets and Food101 datasets. In certain cases, an inappropriate selection of few-shot samples may even bias the adaptation.

Response to Questions

1. Analysis of Text Reweighting on ImageNet
   In zero-shot scenarios, text re-weighting effectively bridges modality differences between text and image embeddings by mitigating the impact of irrelevant features. However, in few-shot adaptation scenarios, feature selection is mainly carried out by the few-shot samples which share the same representation space with other testing samples. This explains why improvement is mainly brought in by few-shot reweighting, rather than text reweighting. An insignificant drop of performance on ImageNet might be caused by the conflict between few-shot and text reweighting. Nevertheless, they are complementary to each other on DTD and UCF datasets.

We hope these clarifications comprehensively address your concerns and underscore the robustness and applicability of our approach. We are grateful for your insightful feedback, which has greatly contributed to enhancing our manuscript.

Kind regards,
The Authors