Attribute-based Visual Reprogramming for Vision-Language Models

FLexibility with Other LLMs (i.e., Smaller-Scale or Open-Source ones)

(W1) While the paper mentions using GPT-3.5 for generating descriptive and distinctive attributes (DesAttrs and DistAttrs), it lacks detailed reasoning on why GPT-3.5 was specifically chosen over other potential models.

Thank you very much for your concern. In response to the question regarding the selection of other potential large language models (LLMs), we have added the results for additional models (Phi 3.1 Mini 128k, Llama 3.1 Mini, and GPT-4o Mini) in the table below.

Key conclusions

• Less powerful LLMs also deliver competitive results. AttrVR with other LLMs also achieves significant performance improvements compared to the baseline. Therefore, choosing which LLM to use might not be that important.
• LLMs with higher generation quality (e.g., the GPT series) yield better results, making the selection of such models more reasonable.
• However, AttrVR is not tied to GPT-3.5. It can seamlessly integrate smaller or open-source LLMs, and even exclude using them to achieve tangible performance. You may also refer to our another response for more discussions.

AttrVR is Robust to Noise/Inaccuracy in Attributes

(W1 cont'd) Further, the method relies heavily on the accuracy and quality of DesAttrs and DistAttrs generated by a language model. However, there is limited discussion on how inaccuracies or noise in these attributes might affect the model's performance.

We confirm that the generated attributes do contain some noise and irrelevant information. However, the $k$-nn attribute selection module (proposed on Page 6) is specifically designed to select the most relevant attributes for each sample. This step effectively filters out low-quality attributes, ensuring the quality and relevance of the final chosen attributes. To demonstrate its effectiveness, we conducted additional experiments comparing the performance of random attributes selection with our $k$-nn attributes selection. The findings are presented in the table below.

Key conclusions

• Low-quality attributes degrade performance. For example, in datasets marked with '^' (e.g., Cars, EuroSAT, UCF101, Resisc), results from randomly selected attributes perform worse than not using attributes at all (i.e., the baseline AR).
• $k$-nn module ensures that only high-quality attributes are used. AttrVR with $k$-nn consistently outperforms both baseline and random selection, confirming its ability to select high-quality and relevant attributes.
• When the generated attributes are of sufficiently high quality, such as in datasets marked with '*', the advantage of knn over random attribute selection gradually diminishes.

The results show that our $k$-nn module is critical for ensuring attribute quality and addressing concerns about reliability.

We appreciate your questions and this supplementary analysis has been included in Appendix C.7 and C.8 of the revised version.

Querying $k$-NN Attributes does NOT Incur Significant Time Cost

(W4) Although the k-NN aggregation outperforms other methods like max, avg, and mean aggregation in attribute selection, the iterative querying process for the k-nearest attributes per sample per epoch raises concerns about computational efficiency and scalability, especially for large datasets. In addition, the sample-specific optimization approach using k-NN may cause the model to overfit to specific samples instead of capturing generalizable features.

Thank you for your concern. In fact, the additional computational overhead introduced by the $k$-nn attribute selection is negligible because

• The attributes' text embeddings are precomputed and remain unchanged throughout training.
• The additional overhead of the $k$-nn module lies solely in the sorting and selection processes once the similarity matrix between image samples and attributes is computed.
• Concretely, in our $k$-NN selection, the complexity for sorting and averaging the top $k$ attributes is $O(n(m\log⁡ m+k))$, where $m$ is the number of attributes; while the time complexity for computing the mean or maximum (i.e., alternative selection strategies) is $O(nm)$.
• For each attribute class ($m=20$ items) and top-$k=3$ selection, these operations incur minimal computational cost compared to the forward pass of CLIP, which actually constitutes the dominant time complexity in practice.

To illustrate this, we measured the time required for different methods. The table below shows the time per batch, epoch, and total training time for AttrVR using different modules (mean, avg, max, and $k$-nn). As observed, the additional time cost of using knn is indeed negligible.

In the revised version, we have included this explanation in Appendix C.5.

$k$-NN Selection is Not Empirically Prone to Overfitting

Additionally, regarding the potential for overfitting caused by knn, the results in Appendices C.3 and C.8 demonstrate that $k$-nn has an advantage in selecting and filtering attributes. The results show that it outperforms other methods, and the performance improvement far outweighs the potential impact of overfitting.

Once again, thank you for your valuable suggestions, which have contributed to improving the quality of our paper. We hope that our reply addresses your concerns.

Thank you very much for your review. For each of your concerns, we first quote the original question, and present point-by-point clarification accordingly.

AttrVR is Not Sensitive to Hyperparameter Selection

(W2) The effectiveness of the model appears sensitive to hyperparameters like λ (balance between DesAttrs and DistAttrs) and k (the number of nearest attributes). While the authors conduct a hyperparameter analysis, this could raise concerns regarding the robustness of the model in practical settings where optimal hyperparameters may vary widely across datasets.

Thank you for your concerns. In fact, as shown in Figure 6, as long as $k$ and $\lambda$ are within a reasonable range, their fluctuations do not have a significant impact (e.g., $k \in [1, 5], \lambda \in [0.25, 0.75]$). Additionally, we have conducted further experiments where we selected the optimal $k$ and $\lambda$ for each dataset. The results are presented in the table below.

Key conclusions

• The differences between the optimal $k$ and $λ$ for each dataset and the results obtained with $k=3$ and $λ=0.5$ are minimal.
• We believe that this choice can be widely used across datasets.

We further note that AttrVR is under a standard supervised learning paradigm, where hyperparameter tuning can be a routine and manageable aspect of model training. Thus, it is easy to include any hyperparameter selection processes in AttrVR when necessary. We thus conclude that there is no further concern about hyperparameter sensitivity with our method.

Thank you again for your valuable feedback, we have included this supplementary analysis in Appendix C.6 of the revision.

Difficulty in Isolating Hard Negative Completely

(W3) While standard contrastive learning ablation is provided, the study lacks a comparison between hard negatives and simple negatives. Additional experiments could assess performance when hard negatives are removed, highlighting whether they significantly enhance alignment and fine-grained recognition capability.

Thank you very much for your interesting suggestion. Below are clarification and additional context regarding hard and simple negatives and their contributions to our method.

Why is isolating hard negative difficult? Identifying if a single attribute description represents a hard or simple negative is inherently ambiguous. For example, the sentence "The British Shorthair is a medium-sized cat with a solid, muscular build" may contain both a hard negative description ("a medium-sized cat") shared with another class (in Fig. 1), and a simple negative description ("a solid, muscular build"). This overlap makes it impractical to establish an absolute categorization.

Validated Contributions via Ablation Studies. Besides, the DesAttr in our method is somewhat analogous to hard negatives, while DistAttr resembles simple negatives. To assess their contributions, we have performed ablation studies where DesAttr was removed, as shown in Table 3 in the paper, which has already demonstrated its impact on the results.

After reviewing the authors' rebuttal, I find my concerns are almost addressed. Given the responses, I recommend maintaining the original score.

Dear Reviewer ELTS,

Many thanks for your reply! Since there are still five days until the end of the discussion, may we know your concerns if any (as your concerns are "almost addressed" instead of "fully addressed")?

We are not sure if there is anything we can do to make our paper even better? We can further strengthen our paper based on your new comments until our paper is good to be accepted. ^^

Looking forward to hearing back from you.

Best,

The authors of Submission 6637

Thank you very much for your review. For each of your concerns, we first quote the original question, and present point-by-point clarification accordingly.

(W1) Is there a way to ascertain that the distinctive captions generated are actually distinct in the embedding space? While lines 78-79 claims this is the case in Figure 1, I disagree. The distinctive attributes seem to not necessarily be farther from the cluster center compared to descriptive attributes. The authors could compare of cosine distance of descriptive attributes and distinctive attributes with the class labels. Ideally, there should be a higher distance from other classes for distinctive attributes than descriptive attributes.

Thank you very much for your great suggestion. Now we have calculated the average cosine distance (i.e., $1-$ cosine similarity) between the prompted label embeddings and the distinctive attribute (DistAttr), descriptive attribute (DesAttr) embeddings for 100 attribute samples of the two classes in Figure 1. The results with mean and std are shown in the table below.

The results confirm that DistAttr is indeed farther from the label embeddings compared to DesAttr. Thank you again for your valuable suggestion. We have incorporated two bar charts in Figure 1 of the revised edition to visually display the results of this comparison.

(W2) How does $k$-nn attributes selection perform compared to random attributes selection?

Thank you very much for your valuable suggestion. Following your request, we conducted additional experiments and obtained new findings, highlighting the importance of our $k$-nn module when the quality of the generated attribute descriptions is low.

Key conclusions

• Low-quality attributes degrade performance. For example, in datasets marked with '^' (e.g., Cars, EuroSAT, UCF101, Resisc), results from randomly selected attributes perform worse than not using attributes at all (i.e., the baseline AR).
• $k$-nn module ensures that only high-quality attributes are used. AttrVR with $k$-nn consistently outperforms both baseline and random selection, confirming its ability to select high-quality and relevant attributes. When the generated attributes are of sufficiently high quality, such as in datasets marked with '*', the advantage of $k$-nn over random attribute selection gradually diminishes.

The results show that our $k$-nn module is critical for ensuring attribute quality and addressing concerns about reliability. This supplementary analysis has been included in Appendix C.8 and we also update the plot in Appendix C.3 of the revision.

(W3)&(Q1)&(Q3) Typos

Thank you very much for your corrections. In the revised version, we have updated the relevant references (W3) and fixed the typo (Q3). Additionally, the error of the plus/minus signs in the definition formula for Class Separability (CS) (Q1) has been corrected. The reviewer is correct, CS should be maximized when the intra-class variance is minimized and the inter-class distance is maximized.

(Q2) Clarification of AttrZS in Table 1

Thank you very much for raising this concern. Here, AttrZS refers to the scenario where we do not train the VR noise pattern. For a single sample, AttrZS first resizes the sample to the required input size for the model, then calculates the similarity between the sample and the attribute descriptions of each class using our AttrVR method in a single pass. The details of this process have been updated in Appendix A.3 of the revised edition.

Once again, thank you for your valuable suggestions, which have contributed to improving the quality of our paper. We hope that our reply addresses your concerns.

Thank you very much for your question regarding AttrVR's reliance on large language models (LLMs) and the feasibility of alternatives. For each of your concerns, we first quote the question and provide clarifications accordingly.

FLexibility with Smaller or Open-Source LLMs

(W1) Dependence on Large Language Models: The AttrVR method relies on large language models (e.g., GPT-3.5) to generate descriptive and distinctive attributes. This dependence may increase computational costs and limit the method's applicability in resource-constrained environments.

(Q2) Is it possible to use smaller or open-source language models to generate attribute descriptions to reduce dependence on large models?

AttrVR is not restricted to GPT-3.5. It can seamlessly integrate smaller or open-source LLMs while maintaining competitive performance. To validate this flexibility, we conducted supplementary experiments using GPT-4o Mini, Phi 3.1 Mini 128k, and Llama 3.1 Mini which can still deliver strong results while substantially lowering computational demands. See the Table below for detailed results.

Effectiveness of AttrVR Without LLMs

(Q1) How significantly would limited or unavailable access to these large language models impact the method's effectiveness?

Even in scenarios where LLMs are unavailable, AttrVR remains effective due to

• Handcrafted Prompts with Labels: The VR training and $k$-nn attributes selection modules can still be utilized to obtain optimized handcrafted prompts with labels for individual samples, enhancing performances over the baseline.
• No Training Overhead: Note that attribute generation is independent of the training process. Thus, replacing GPT-3.5 with smaller or open-source models—or even excluding LLMs—does not increase training time.

Supplementary Experimental Results

To further demonstrate the flexibility and usability of AttrVR when used with other LLM options, we evaluated 6 configurations on multiple datasets: (1) the baseline method AR (from this paper), (2) handcrafted prompts with labels (without relying on LLMs),(3) LLM Phi 3.1 Mini 128k, (4) LLM Llama 3.1 Mini, (5) LLM GPT-4o Mini, (6) GPT-3.5 (as used in this paper).

Key conclusions

• Even when using only handcrafted prompts with labels without LLM-generated attributes, AttrVR still outperforms the baseline.
• Using smaller or open-source models can also achieve significant improvements compared to the baseline.
• Higher-quality LLMs (such as the GPT series) tend to yield slightly better results, making them more suitable for generating attributes.

These insights confirm that AttrVR applies to diverse attribute generation strategies and computational settings. We have included this supplementary analysis in Appendix C.7 of the revised edition.

Thank you again for your thoughtful questions.

Thank you for your concerns. For each of your concerns, we first quote the question and provide clarifications accordingly.

AttrVR is Robust to Low-quality Attributes

(W2) Reliability of Attribute Generation: The attribute descriptions generated by large language models may have issues with accuracy and relevance. If the generated attributes do not match the target categories, they could negatively impact the model's performance.

(Q4) If the generated top attributes do not match the target categories, how does it impact model performance?

Thanks for raising this concern. We have included additional ablation studies to investigate this possibility, and find that:

• Low-quality generated attributes can negatively impact the training of VR models on CLIP (AttrVR variant using random attributes selection)
• However, it will not impact the performance of our AttrVR, as the proposed $k$-nn attribute selection module filters out those low-quality or irrelevant attributes.

The table below substantiates this claim with detailed experimental comparisons.

Mechanisms Ensuring the Attribute Quality

(Q3) Does the paper provide mechanisms to assess and ensure the quality and relevance of attribute descriptions generated by large language models?

Yes, the $k$-nn attribute selection module (proposed on Page 6) is specifically designed to select the most relevant attributes for each sample. This step effectively filters out low-quality attributes, ensuring the quality and relevance of the final selected attributes. To demonstrate its effectiveness, we conducted the following experiments. The results are presented below.

Supplementary Experimental Results

We present a comparison between attributes randomly selected from the generated set and those selected using our knn module.

Key conclusions

• Low-quality attributes degrade performance. For example, in datasets marked with '^' (e.g., Cars, EuroSAT, UCF101, Resisc), results from randomly selected attributes perform worse than not using attributes at all (i.e., the baseline AR).
• $k$-nn module ensures that only high-quality attributes are used. AttrVR with $k$-nn consistently outperforms both baseline and random selection, confirming its ability to select high-quality and relevant attributes.
• When the generated attributes are of sufficiently high quality, such as in datasets marked with '*', the advantage of knn over random attribute selection gradually diminishes.

The results show that our $k$-nn module is critical for ensuring attribute quality and addressing concerns about reliability. We appreciate your questions and this supplementary analysis has been included in Appendix C.8 of the revised version.

(W3) Computational Complexity and Efficiency: AttrVR queries 𝑘 nearest neighbor attributes for each image sample in every training cycle, increasing computational overhead. This may impose pressure on training time and resource requirements for large datasets.

Querying $k$-NN Attributes does NOT Incur Significant Time Cost

In fact, the additional computational overhead introduced by the $k$-nn attribute selection is negligible because

• The attributes' text embeddings are precomputed and remain unchanged throughout training.
• The additional overhead of the $k$-nn module lies solely in the sorting and selection processes once the similarity matrix between image samples and attributes is computed.
• For each attribute class ($m=20$ items) and top-$k=3$ selection, these matrix operations incur minimal computational cost compared to the forward pass of CLIP, which constitutes the dominant time complexity in practice.

To substantiate this, we measured the time required for the baseline method and AttrVR.

Supplementary Experimental Results

The table below shows the time per batch, epoch, and total training time for the baseline method and AttrVR. As observed, the additional time cost of using $k$-nn is indeed negligible.

We appreciate your concern and this supplementary analysis has been included in Appendix C.5 of the revised edition.

Scope of Experiments

(W4) Scope of Experimental Validation: The paper primarily conducts experiments on 12 downstream tasks. It remains unclear whether the method's effectiveness can be validated on larger or more challenging datasets.

(Q5) Has consideration been given to validating the method's effectiveness on larger or more challenging datasets to ensure its generalizability?

(Q6) Although the AttrVR method leverages learnable noise, how does its performance improve in more complex image scenarios?

Thank you for your feedback about the scope and generalizability of our experiments. We address these below:

1. Dataset Scope:
   • Our experiments cover 12 diverse tasks, following widely used benchmarks from previous classification works [1, 2], and include datasets spanning diverse visual domains such as scenes, actions, textures, and fine-grained details.
   • Additionally, we incorporated the results from the ImageNet dataset, a well-established benchmark containing a rich variety of categories and images.
2. Generalization to Challenging Scenarios:
   • To the best of our knowledge, ImageNet may have already constituted a sufficiently large and challenging benchmark dataset in the field of image classification.
   • Thus, we believe the results on these datasets should be enough to demonstrate that AttrVR possesses strong generalizability.

We appreciate your questions and we hope our response meets your expectations.

[1] Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In CVPR, 2023.

[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.

AttrVR is Not Sensitive to Hyperparameter Selection Across Datasets

(W5) Impact of Hyperparameter Selection: Hyperparameters in AttrVR (e.g., the number 𝑘 of nearest neighbor attributes) significantly affect model performance. While the paper selects the optimal 𝑘=3, it does not demonstrate whether this choice is equally suitable in more general scenarios across different datasets, failing to prove the method's reliability in various practical environments.

Thank you for raising your concern regarding hyperparameter selection in AttrVR, specifically the choice of the number of $k$. To address it, we conducted a detailed analysis comparing the performance of AttrVR with a shared $k=3$ across all datasets against using dataset-specific optimized $k$.

The results presented in the table below show that the performance (accuracy) differences between these two settings are minimal, with variations no greater than 0.3% across datasets. This may have demonstrated that $k=3$ is a broadly applicable choice for diverse datasets.

We also note that AttrVR is under a standard supervised learning paradigm, where hyperparameter tuning can be a routine and manageable aspect of model training. By selecting $k=3$, we leverage its general applicability while avoiding extensive dataset-specific adjustments. That being said, it is easy to include any hyperparameter selection processes in AttrVR when necessary. Thus, we believe there is no further concern about hyperparameter sensitivity with our method.

Supplementary Experimental Results

We appreciate your concern and this supplementary analysis has been included in Appendix C.6 of the revised paper.

Thank you very much for the your response to my questions and the detailed experimental evidence provided, which has resolved many of my doubts. However, I still have some questions that need further confirmation:

Description Generation in the Inference Process:

Does the inference process require generating descriptions for each image through an LLM? Without seeing the image, how is your prompt designed? The templates provided in the paper are relatively simple and may not generate very detailed and accurate descriptions. Then, the inference and classification are performed through AttrVR. The inference process of this method may be relatively complex.

Possibility of Model Replacement:

Can this process be replaced by more advanced LVLMs (Large Vision-Language Models)? These methods have excellent visual emergence capabilities.

Detailed Explanation of Attribute Differences:

Could you elaborate on the differences between DesAttrs and DistAttrs? The description in the paper is somewhat vague; it would be best to provide examples and summarize the explanation.

Dear Reviewer uGif,

Thank you for taking the time to review our submission. We appreciate your thoughtful feedback and the opportunity to clarify our work through our responses and the revision.

It has been a few days since we submitted our responses. If you have any further questions or require additional clarifications, we would be delighted to discuss and address them. Otherwise, we kindly hope you might consider revisiting your evaluation and possibly reflecting these clarifications in your final score.

Thank you again for your time and effort in reviewing our work and your valuable contribution to improving the quality of our submission.

Looking forward to hearing back from you.

Best, The authors of Submission 6637

Thank you very much for your additional questions. Below, we still first quote your question, and then provide detailed responses along with additional experiments addressing your concerns.

Description Generation in the Inference Process:

Does the inference process require generating descriptions for each image through an LLM?

No, AttrVR does not generate descriptions during inference. Instead, attribute descriptions are precomputed for each class before training and stored as an attribute set. This eliminates the dependency on querying LLMs during inference.

Without seeing the image, how is your prompt designed? The templates provided in the paper are relatively simple and may not generate very detailed and accurate descriptions.

Although the images are not used, the attribute-generation prompts leverage the [target class name] and [downstream task name] to generate class-specific attributes in the precise downstream task context, using the following two types of templates:

• DesAttr: Describe the appearance of the [downstream task name] [target classes name]
• DistAttr: Describe the unique appearance of a/an [target classes names] from the other [downstream task name]

Simple as the templates are, they are effective enough to generate useful descriptions that enhance AttrVR's performance (as shown in Tables 1 and 3).

While some descriptions may lack detail or accuracy, the knn module in AttrVR addresses this by: (1) filtering out vague or inaccurate attributes, and (2) selecting sample-specific attributes.

Then, the inference and classification are performed through AttrVR. The inference process of this method may be relatively complex.

Thank you for raising this concern. However, there might exist misunderstandings to our method and we would like to clarify them further as follows. In fact, the inference process in AttrVR is straightforward and avoids unnecessary complexity. Here’s how it works:

• All attribute descriptions are generated, and their text embeddings are computed before training and inference.
• Thus, inference only involves a standard forward pass as in training, calculating similarity between image embeddings and precomputed text embeddings from the attribute set.
• As the attributes text embeddings are precomputed, the time cost during inference is negligible (Table 10, "Batch Forward Time").

Possibility of Model Replacement:

Can this process be replaced by more advanced LVLMs (Large Vision-Language Models)? These methods have excellent visual emergence capabilities.

Yes, Multimodal Large Language Model (MLLM or LVLM) can generate attributes and integrate seamlessly into our AttrVR framework.

To substantiate this, we conduct additional experiments using the GPT-4o-mini VLM module (known to have excellent visual emergence capabilities) to generate attributes for each class based on five randomly sampled images from the training set and the same prompts used in LLMs. Still, both VLM- and LLM-generated attributes are provided before training.

The comparative results with generating attributes using the GPT-4o-mini LLM module are presented in the table below.

Our Concolusions:

• VLM-generated attributes yield better results in visually distinctive tasks (e.g., texture in DTD dataset). while their advantage over LLM-generated attributes diminishes in tasks with subtle visual differences (e.g., different models of airplanes in Aircraft dataset).
• AttrVR outperforms the baseline in all cases, using either LLM-generated or VLM-generated attributes, demonstrating its robustness regardless of the attribute generation model.

These results are included in Appendix C.9. of the latest revision.

Could you elaborate on the differences between DesAttrs and DistAttrs? The description in the paper is somewhat vague; it would be best to provide examples and summarize the explanation.

What distinguishes DesAttr and DistAttr

DesAttr refers to the most common visual features across multiple image samples within the same class, describing the class by capturing its general characteristics. For instance, for classes 'pink primrose' and 'hard-leaved pocket orchid', DesAttr would describe their characteristics 'delicate appearance' and 'thin stems'.

• Example of DesAttr for 'pink primrose'

  • "The pink primrose flower has a delicate and dainty appearance.",
  • "The flower pink primrose is a dainty, delicate flower that grows on thin stems."

• Example of DesAttr for 'hard-leaved pocket orchid'

  • "The flower hard-leaved pocket orchid is a small, delicate plant with a unique appearance.",
  • "The flower hard-leaved pocket orchid is a small, delicate flower that grows on a thin stem."

DistAttr highlights visual features unique to a class, distinguishing it from other classes. For class 'pink primrose', DistAttr would emphasize characteristics such as 'soft, pale pink color' and 'slender stem', which are unique to class 'pink primrose'. In contrast, for class 'hard-leaved pocket orchid', DistAttr would focus on characteristics like 'tightly clustered flower' and 'bell-shaped flower', features that are rarely observed in other classes.

• Example of DistAttr for 'pink primrose'

  • "A pink primrose typically has a soft, pale pink color with delicate, ruffled petals.",
  • "A pink primrose has a delicate and slender stem, typically ranging from 15-30 cm in height."

• Example of DistAttr for 'hard-leaved pocket orchid'

  • "A hard-leaved pocket orchid is a small, tightly clustered flower that typically grows on trees or rocks.",
  • "Canterbury bells (Campanula medium) are highly recognizable flowers due to their distinct, bell-shaped flowers that hang down from sturdy stems."

In summary, their key difference is that DesAttr mainly describes the more common features within the samples of a given class, while DistAttr primarily describes the features that distinguish the class from samples of other classes. Together, these attributes enable AttrVR to balance the generality and distinctiveness of the generated attributes.

Once again, we thank you for your thoughtful questions, helping us to improve the quality of our paper further. Would these clarifications and additional experiments have addressed your concerns? Looking forward to hearing back from you.

Thank you for considering the possibility of increasing our score and some further concerns. We will address your concerns to clarify any potential misunderstandings, highlight the contributions of our work, and hope you will make your decision to increase the score. ^-^ We will quote your concerns and then provide the clarification.

The core of this work lies in how to construct a description dataset that possesses both common and distinctive features to enhance task performance.

In fact, proposing the attributes dataset is just one of our contributions. Our primary contribution lies in introducing the AttrVR method to Reprogramming the vision-language model for downstream classification tasks (i.e., Reprogramming is a way to adapt pre-trained models to downstream tasks by only modifying the input and output spaces of downstream tasks). Our innovation compared to other reprogramming methods is that we pair images with generated attributes for learning the reprogramming patterns, rather than pairing them with prompted labels, which better aligns with the capability of vision-language model. We also leverage Class Separability (i.e., reducing intra-class variance and increasing inter-class separation) in Section 5 to theoretically explain the effectiveness of our method.

However, as far as I know, your method has very limited scalability, requiring retraining and data construction with LLMs for zero-shot tasks.

In fact, our method has two versions:

(1) AttrZS: This approach performs zero-shot classification using only generated attributes, requiring no retraining and involving a single CLIP forward pass, as detailed in Appendix A.3. It is fully scalable.

(2) AttrVR: This version is designed for methods that need to train a visual reprogramming pattern (i.e., a method of adding a trainable noise pattern to the input image). This is the version we mainly discussed in the rebuttal and elaborated on extensively in the paper.

Both versions show improvements over their respective baselines and do not introduce any additional computational overhead compared to the baselines. Therefore, the work presented in this paper should not face limitations in scalability.

Additionally, I am skeptical about whether the dataset you described is truly as effective as you claim and whether it is open-source.

Please rest assured regarding the effectiveness and open-source concerns. Our method ensures reproducibility, and the code is already provided via an anonymous link in this paper. If the paper is accepted, we will update the camera-ready version with a GitHub link and upload all generated attributes, including those generated by other LLMs and VLMs during our rebuttal experiments.

Moreover, with current large-scale visual-language models already performing well in fine-grained scene recognition, the value of this method is diminishing.

In this response, we would like to emphasize the significance of our method and the value of such research direction.

Firstly, as shown in Table 1, even though vision-language models perform well in many cases, they still struggle on certain tasks, such as ESAT (a remote sensing dataset). Our method helps to unlock better performance for vision-language models in such tasks.

Secondly, although the current development of vision-language models has led to high accuracy in many classification tasks, it is still crucial to study their training and alignment across different distributions and propose improvements. Such research not only aims to improve performance but also seeks to promote the application of traditional mathematics and machine learning theories in the era of foundation models.

Do our responses solve your further concerns? Would it help you to make the decision about increasing the score? ^-^ Hope to hear from you.

Responses to Reviewer ELTS

1. LLMs and Attribute Noise

• Concerns: Can other LLMs be used? How does noise in generated attributes affect performance?
• Response: We demonstrate that AttrVR is compatible with smaller (GPT-4o mini) or open-source (Phi 3.1 Mini 128k, and Llama 3.1 Mini) LLMs. We add a section (Appendix C.7) to include the results from these additional experiments and discuss the noise in generated attributes (Appendix C.8).

2. Hyperparameter Sensitivity:

• Concerns: Is AttrVR sensitive to $k$ or $\lambda$?
• Response: We conduct additional experiments to show hyperparameter-induced performance variations are under 0.4% (added in Appendix C.6).

3. Comparison of Negatives:

• Concern: How do hard and simple negatives compare?
• Response: We clarify that DesAttr acts as hard negatives with Ablation studies (Table 3), while DistAttr serves as simple negatives.

4. Time Complexity of $k$-nn:

• Concerns: Is $k$-nn efficient compared to alternatives?
• Response: We conduct time complexity analysis and measure the training time, confirming $k$-nn selection results in negligible overhead. We include the details in Appendix C.5..

We believe these responses have comprehensively addressed all reviewer concerns, incorporating additional experiments and clarifications. The reviewers have not raised further issues and have indicated scores leaning towards acceptance.