Tree of Attributes Prompt Learning for Vision-Language Models

Thank you for the constructive feedback and the positive assessment of our work! We address the detailed concerns below.

W1: LLM Robustness

Thank you for highlighting this concern. We note that since our use of LLM is mainly retrieving simple facts without requiring complex capabilities, the quality of the generated attributes is generally reliable.

1. Robustness Against Different LLMs

We regenerated the descriptions using a small LLM, Qwen-2-7B-Instruct [1], and obtained comparable results:

2. Robustness Against Different Generation Prompts

We reran ToA generation pipeline using prompts rewritten by ChatGPT. This process resulted in a slightly different set of attributes. The results were as follows:

These results demonstrate that our method maintains its performance across different LLMs and generation prompts. The consistency in results indicates that our method's effectiveness does not heavily rely on the choice of LLM, given its straightforward nature. As LLM capabilities continue to improve, we anticipate further enhancements in robustness.

W2: Effectiveness of Model Regularization

Thank you for pointing this out. We conducted an additional experiment without regularization, while reducing the learning rate to 1/4 of the original values to avoid overfitting. The results are as follows:

As expected, the performance significantly decreases without model regularization, which aligns with findings from previous works [2,3]. For example, MaPLe, which did not use model regularization, reported Base: 82.28, Novel: 75.14, HM: 78.55, whereas PromptSRC, which added model regularization to MaPLe, achieved an average HM of 79.97. These results demonstrate that model regularization is crucial for prompt tuning to prevent overfitting and catastrophic forgetting.

Notably, our work outperforms existing methods both with and without model regularization, showing better performance than MaPLe when regularization is not used, and surpassing PromptSRC when it is used.

W3: Clarity of Figure 2

We apologize for any confusion caused by Figure 2. In the figure, each “color” represents an attribute (e.g., orange -> fur pattern). In each attribute:

• $I_1$ represents the visual expert token $p^v_a$ of attribute $a$.
• $T_1$, $T_2$, etc. are $v_c^a$, representing the output of the VCP layer for class 1, class 2, etc., in attribute $a$.
• $I_1T_1$, $I_1T_2$, etc. represent the cosine similarity calculations between the visual expert token and the corresponding pooled textual features of all classes, which are the prediction logits of attribute $a$.

We calculate prediction logits for each attribute and obtain the final prediction via a weighted sum of all prediction logits (refer to Equation 7). We will refine the figure to make this clearer.

Q1: Attn. Max Pooling vs. VCP

Both Attn. Max Pooling and VCP are attention-based pooling, but VCP can be viewed as a "soft" version of Attn. Max Pooling. In Attn. Max Pooling, we calculate the attention score between $p^v_a$ and the description set $D_c^a$ in the same way in first three lines of Equation 4. Instead of obtaining $v_c^a$ using the attention score for a weighted sum of $D_c^a$, we perform a max pooling of $D_c^a$ based on the attention score, selecting the description with the highest attention score to the visual expert token. The better performance of VCP than Attn. Max Pooling shows VCP's flexibility in capturing multiple relevant descriptions, whereas Attn. Max Pooling only pools one description.

Q2: Adaptive attribute number

We use an adaptive number of attributes because of variations and granularities between datasets. For instance, 4 attributes (Action Pose, Number of People, Background Setting, Objects Present) may suffice for UCF101 but are inadequate for ImageNet, which has greater class variation.

The generated attributes are generally in the same granularity, such as shape, pattern, etc. for ImageNet and fur pattern, eye pattern, etc. for Pets. Once sufficient attributes for class differentiation are available, additional attributes do not provide further benefits and may increase overfitting risks.

Q3: Deep prompting

Yes, deep prompting introduces learnable prompt tokens in every transformer layer, as opposed to shallow prompting, which introduces prompt tokens only at the input level. This is a common trick for improving model's performance used in previous works [2,4,5]. To reduce complexity, we applied deep prompting only to the vision encoder. We apologize for not showing this in Figure 2.

Q4: Effect of $\alpha$

$\alpha$=0.4 was empirically found to balance global and local information optimally. When $\alpha$=1.0, the performance significantly decreased:

This shows the importance of expert tokens. Additionally, in Table 4, we tried using CLS token instead of expert tokens to align with each attribute feature during training, but also found it significantly worse. These results support the notion that domain-specific expert tokens enhance the model’s ability to grasp fine-grained details by focusing on distinct aspects of the image, as opposed to the CLS token’s global focus.

Q5: Unstructured Set of Descriptions

Yes, they were converted by putting all descriptions from the same tree of attributes in a unstructured set of descriptions.

[1] Yang et al. Qwen2 Technical Report. arXiv:2407.10671 (2024)

[2] Khattak et al. Self-regulating Prompts: Foundational Model Adaptation without Forgetting. ICCV (2023)

[3] Yao et al. Visual-Language Prompt Tuning with Knowledge-guided Context Optimization. CVPR (2023)

[4] Khattak et al. MaPLe: Multi-modal Prompt Learning. CVPR (2023)

[5] Jia et al. Visual Prompt Tuning. ECCV (2022)

Thank you for your valuable comments! We address the detailed concerns below. We hope that our responses will reflect positively on your final decision.

W1: Computational and Time Costs

Thank you for highlighting the concern. We would like to clarify how TAP manages these costs effectively compared to previous works.

1. Efficient Use of Learnable Parameters: While TAP introduces textual and visual prompts, the number of newly introduced learnable parameters is actually less than in previous works such as PromptSRC [1] and MaPLe [2]. Specifically, methods like PromptSRC and MaPLe utilize "deep prompting," introducing "N" learnable prompt tokens in each layer of both the vision and text encoders. In contrast, TAP employs "deep prompting" only for the vision encoder. For the text encoder, we use "shallow prompting," where learnable prompts are introduced only at the input level. This approach significantly reduces the number of additional parameters for the text encoder.
2. Inference Efficiency: At inference time, both the vision and text embeddings in TAP can be independently pre-extracted and saved for future retrieval tasks. This means that once the embeddings are generated, they can be reused without recomputing. In contrast, MaPLe [2] utilizes a vision-language (VL) coupling function, which couples the vision and text encoders. This coupling means that the embeddings cannot be cached independently; different images can alter the text feature embedding due to the VL coupling, necessitating re-inference for new images. TAP avoids this issue, making it more efficient for repeated inferences and retrieval tasks.

W2: How TAP uses the hierarchical structure

Thank you for your feedback. We would like to clarify how the hierarchical structure in TAP is utilized to enhance the integration of text encoder outputs and the overall model performance.

Our approach leverages the tree structure in two significant ways:

1. Description generation in Top-Down: We first generate a set of attributes from the class name, followed by generating descriptions for each attribute. This hierarchical approach ensures that the descriptions are structured and contextually relevant. Unlike previous works that generate an unstructured set of descriptions, our method organizes them into a coherent tree structure, as illustrated in Figure 1 of the paper.
2. Utilization of Tree of Attributes in Bottom-Up:
   • From leaf nodes to attribute-layer, we use the VCP layer to aggregate descriptions in each attribute to form attribute-level features. These features are then aligned with corresponding visual expert tokens, ensuring that each visual expert token focuses on specific, fine-grained attributes.
   • From attribute-layer to root node class prediction, we aggregate attribute-level features to make class predictions via a weighted sum of the prediction logits (refer to Equation 7). This process allows the model to utilize the structured relationships within the tree, enhancing the alignment and integration of visual and textual data.

By using a top-down approach to generate the tree and a bottom-up approach to utilize it, TAP effectively integrates hierarchical relationships within the prompt graph. This dual usage ensures that the model leverages both the high-level structure and fine-grained details, leading to improved performance and interpretability.

We hope this clarifies the role and utilization of the hierarchical structure in TAP.

W3: Difference with MAPLE and ALIGN

Compared with multimodal prompt tuning methods MAPLE and ALIGN, our method significantly differs from them in two key aspects: 1) Main focus. Our method focuses on augmenting category names with prior knowledge from LLMs to better utilize the rich information inherent in the category names. In contrast, these methods rely solely on the original category names and focus on multimodal feature fusion/alignment. 2) Methodology. Our method constructs a tree of attributes to organize LLM-generated descriptions and leverages this structured information through prompting, whereas MAPLE and ALIGN fail to learn such structured information, as they do not associate textual descriptions with category names.

[1] Khattak et al. Self-regulating Prompts: Foundational Model Adaptation without Forgetting. ICCV (2023)

[2] Khattak et al. MaPLe: Multi-modal Prompt Learning. CVPR (2023)

Thank you for the constructive feedback and the positive assessment of our work! We address the detailed concerns below.

W1: Potential for Hallucinated Content from LLM

Thank you for your insightful comment. We considered using Wikipedia for description generation; however, this approach still necessitates an LLM to extract structured descriptions from extensive Wikipedia pages, which may still result in hallucinated content. Additionally, processing long Wikipedia pages can be resource-intensive. Since our task is essentially retrieving simple facts from LLM that does not require sophisticated reasoning or planing capabilities, the risk of hallucination is significantly reduced.

W2: Structure-Aware Encoding of the Tree

We apologize for any confusion caused. Our tree structure serves dual purposes:

1. Generation of Descriptions: The tree structure guides the generation of descriptions by first generating a set of attributes from the class name and then generating descriptions for each attribute. This approach contrasts with previous works that generate an unstructured set of descriptions (refer to Figure 1).
2. Utilization of the Tree of Attributes: The generated Tree of Attributes is used as follows: The VCP layer aggregates descriptions in each attribute (leaf nodes) to form attribute-level features, which align with the visual expert tokens. These attribute-level features are then aggregated to make class predictions via a weighted sum of the prediction logits (refer to Equation 7).

Thus, our approach generates the tree in a top-down manner and utilizes it in a bottom-up manner.

W3: Equation (5) clarification

We apologize for the confusion. $y$ in $v^a_y$ represents the ground truth, thus $v^a_y$ represents attribute $a$'s VCP-pooled text embedding of the ground truth class.

W4: Potential for Spurious Correlation in Vision-Conditional Pooling

We acknowledge your concern. We note this issue to be a common problem of the attention mechanism. To address this, we experimented with "Quiet Attention" proposed by Evan Miller [1], obtaining comparable results:

This demonstrates that our current implementation of the VCP module, which emphasizes relevant descriptions while de-emphasizing irrelevant ones (as showcased in Figure 4), is effective for the current study.

Q1: Structed Description

We argue that visual data implicitly has a tree structure. The raw image can be seen as the root node, the expert tokens as the attribute layer, and the text description features linked to the expert tokens by the VCP module as the leaf nodes. In this framework, the image corresponds to the class label in text, visual expert tokens align with text attributes, and leaf nodes are shared between vision and text via the VCP layer. This tree structure facilitates fine-grained alignment between visual and textual data.

Q2: One Expert for One Attribute

We make sure one expert token only learn from one attribute by only aligning one expert token to one specific attribute. That is, we have the same number of expert tokens as the number of attributes in the generated Tree of Attributes, and each expert token aligns with one of the attribute via contrastive learning.

Q3: Downstream tasks

Yes, the model works well on downstream tasks. Our base-to-novel experiments, where the model is trained on base classes and tested in zero-shot on novel classes, demonstrate superior performance compared to other baselines. Additionally, we conducted a cross-dataset experiment, where we trained on ImageNet with 16 shots and tested in zero-shot on the remaining 10 datasets. As shown in Table 1 of the Global response, TAP outperforms PromptSRC by 1.03% on ImageNet and 0.75% on average across the other 10 datasets. This indicates that our model generalizes well to downstream tasks.

[1] Evan Miller. Attention Is Off By One. (2023)

text_embed = text_encoder(ToA)  # ToA is a dictionary, text_embed is E x N x D
image_tokens = vision_encoder(image)  # E x D
pooled_embeds = []
for i in range(E):
    expert_token = image_tokens[i]  # 1 x D
    attribute_embed = text_embed[i]  # N x D
    pooled_embed = VCP(expert_token, attribute_embed)  # 1 x D
    pooled_embeds.append(pooled_embed)

Thank you for your valuable comments! We address the detailed concerns below. We hope that our responses will reflect positively on your final decision.

W1: Human Review Requirement

We appreciate the reviewer highlighting the concern regarding the human review process in our method. We apologize for any confusion caused by our explanation. The human efforts mentioned refer to the process of curating a 1-shot example for in-context learning when prompting LLMs. Unlike previous works [1] that manually curate descriptions, we have streamlined the process by making it semi-automatic. LLMs generate the examples, followed by a brief human review. Typically, the generated examples are sufficiently accurate and require less than 30 seconds per dataset for a quick read-through, as they mostly retrieve simple facts from LLMs. This minimal human involvement ensures quality without significant effort or bias.

W2: Cross-Dataset Experiments

We have conducted additional "cross-dataset" experiments where we trained the model on ImageNet with 16 shots and tested it directly on the remaining 10 datasets in a zero-shot manner. The results are presented in Table 1 of the Global response. Compared to PromptSRC, TAP achieved a +1.03% improvement on ImageNet and a +0.75% average improvement across the 10 datasets, demonstrating robust domain generalization capabilities.

W3: Use of Different Learning Rates

We understand the concern regarding the use of different learning rates. The primary reason for splitting the datasets into two groups with different learning rates is due to the variability in ease of learning from LLM-generated descriptions.

However, we also note that using different hyperparameters for different datasets is not uncommon in existing prompt-learning papers. For instance, TaskRes [2] used different learning rates for ImageNet and other datasets. Similarly, CoOP [3] and DMN [4] utilized different numbers of epochs for various datasets, and PromptSRC applied different GPA hyperparameters across dataset groups.

Q1: Dataset Information

As stated in Appendix A.3, the dataset information provided to the LLM is the description from the dataset's official website. For example, the dataset information for EuroSAT is: "EuroSAT dataset is based on Sentinel-2 satellite images covering 13 spectral bands and consisting of 10 classes."

Q2: Number of Descriptions

We apologize for the confusion. As indicated in our prompts in Appendix A.3, the number of descriptions per class ranges from 2 to 5. The figure is an illustrative example and does not limit the descriptions to two sentences per image.

Q3: Relevant Example

At higher hierarchy, the model captures details from different attributes in the image through the alignment of vision expert tokens and corresponding text attribute features, as showcased in Figure 3. At the attribute level, the model captures the variations within the class via the use of the VCP module as showcased in Figure 4.

[1] Menon et al. Visual Classification via Description from Large Language Models. ICLR (2023)

[2] Yu et al. Task Residual for Tuning Vision-Language Models. CVPR (2023)

[3] Zhou et al. Learning to Prompt for Vision-Language Models. IJCV (2022)

[4] Zhang et al. Dual memory networks: A versatile adaptation approach for vision-language models. CVPR (2024)

[5] Khattak et al. Self-regulating Prompts: Foundational Model Adaptation without Forgetting. ICCV (2023)

Thank you for your valuable comments! We address the detailed concerns below.

W1: Marginal Performance Improvement

We appreciate the reviewer's observation.

a. Generalizability in Base-to-Novel Experiments: The base-to-novel experiment is crucial for evaluating the generalizability of prompt learning methods. According to CoCoOP [1], models can overfit to base classes in a few-shot setting. Our results show that TAP excels in this aspect, indicating robust generalization capabilities.

b. Significance of Performance Increase: While the few-shot setting is less important, an average performance increase of 0.5% in the few-shot classification is still noteworthy. For context, the improvement from CoOP to ProGrad is 0.8%, and CoCoOP performs nearly 5% worse than CoOP in few-shot settings. Thus, the improvements brought by TAP, though appearing small, are significant enough within the domain.

c. Improvement of VCP Module: Regarding the VCP module, Table 6 indicates that VCP achieves a 1.08% improvement over average pooling. This enhancement is substantial, demonstrating the effectiveness of VCP in refining attribute relevance to the visual context, which translates to better overall performance.

W2: Differences between TAP and previous works

The key differences between TAP and existing methods are the structured approach to integrating LLM-generated descriptions and the use of visual expert tokens for fine-grained alignment. As for the two referenced papers, although some similarities exist, significant differences between our work and theirs are evident. Additionally, TAP outperforms them by large margins (TAP HM 81.04 vs. AAPL 76.01 and MAP 79.36).

Regarding AAPL: While both AAPL and TAP aim to enhance vision-language models through attribute integration, they diverge fundamentally in their methodologies and objectives. AAPL focuses on managing learnable prompts via data augmentation, using adversarial token embedding to mitigate bias and enhance attribute-specific learning through augmented image features. However, this approach is constrained by the limitations of image data augmentation, which cannot cover all possible variations. In contrast, TAP leverages LLM-generated knowledge to construct a structured Tree of Attributes. This method explicitly captures fine-grained, instance-specific attributes, such as differentiating between a pan-fried crescent-shaped dumpling and a round steamed dumpling, which image augmentations in AAPL cannot achieve. Thus, TAP's utilization of LLM-generated descriptions allows for a more comprehensive and accurate adaptation of VLMs to diverse and nuanced visual concepts.

Regarding MAP: While both TAP and MAP leverage LLM-generated descriptions and introduce learnable prompts in the vision encoder, several key differences distinguish TAP. First, MAP generates descriptions in an unstructured manner, as depicted in Figure 1(b) of our paper, whereas TAP organizes this information into a structured Tree of Attributes (ToA). Second, although both methods aim to enhance fine-grained alignment, MAP aligns at the individual description level, which can lead to redundancy and misalignment due to similar descriptions falling under the same attribute category or containing irrelevant information for the specific image. TAP, on the other hand, aligns visual expert tokens at a higher "attribute" level, ensuring each token focuses on a specific attribute class. This structured approach mitigates the risk of misalignment and enhances the specificity and relevance of the visual prompts. Furthermore, TAP employs a vision-conditional pooling module to filter out irrelevant descriptions, which is not addressed in MAP, providing a more robust and contextually accurate alignment.

W3: Dependency on Prior Information for ToA Construction

We understand the reviewer's concern. To address this, we performed additional "cross-dataset" experiments to evaluate the zero-shot generalization capabilities of our method.

In the experiment, following the common practice, we trained our model on ImageNet under 16-shot and tested it directly on the remaining 10 datasets in a zero-shot manner. The results are presented in Table 1 of the Global response. Compared to PromptSRC, TAP achieved a +1.03% improvement on ImageNet and a +0.75% average improvement across the 10 datasets. These results underscore TAP's robust performance in domain generalization settings, reinforcing its utility in open-vocabulary contexts.

W4: Clarity of Figure 2

Thank you for your constructive feedback. In Figure 2, the input for the vision encoder is the tokenized image tokens and learnable visual expert tokens. The input for the text encoder is the generated descriptions and learnable text prompts. Each attribute has its prediction and the fusion of all attributes is treated as the final output. We will update the figure to ensure it clearly illustrates the input and output streams of our model in the revised version.

W5: Clarification of equation (5)

We apologize for the confusion caused. In equation (5), $v_c^a$ is the pooled textual embedding from attribute $a$ of class $c$, where pooling is done by the VCP module (cross attention between $p^v_a$ and the descriptions in the attribute description set $D_c^a$). The reason why we didn't contrast $p^v_a$ against $D$ is because $D$ is a set of textual embeddings. Note that the vision-conditional pooling module pools suitable textual descriptions with the visual instance condition so the $v_c^a$, as an instance-specific text feature, can be better aligned with the vision embeddings.

[1] Zhou et al., Conditional Prompt Learning for Vision-Language Models. CVPR (2022)

Dear Reviewers,

Thank you for your comments. Please read through all the reviews and the rebuttal and see if authors' responses have addressed your and others' concerns.

Best,

AC