Class Concept Representation from Contextual Texts for Training-Free Multi-Label Recognition

We appreciate your valuable comments and suggestions.

[W1] Clear explanation for difference between TaI-DPT and our method.

To clarify, our method differs from TaI-DPT in two key ways:

• Training-free approach: Unlike TaI-DPT, our method performs multi-label recognition without any training using massive text descriptions inspired by how human forms concept on words. (as detailed in L75-80).
• Enhanced adaptability: Our method employs a sequential attention mechanism to adaptively project visual features onto the same linear space as class concept representation. This adaptive approach, in contrast to TaI-DPT, leads to improves the visual-text alignment at inference. We will revise this explanation for further clarity.

[W2] Unfair comparison in Table 5.

The class concept representation in Table 5 is based on the MS-COCO captions, potentially leading to inherent advantages from dataset-specific information on only "MS-COCO" dataset. However, several points should be considered:

• Ablation Study: We isolated our method's effectiveness by using training set of MS-COCO captions in an ablation study.
• Consistency Across Datasets: We observed consistent performance improvements on the VOC2007 dataset, which lacks MS-COCO-specific information.
• Generalizability to NUS-WIDE: Using GPT-3.5 generated texts from only target class names, we demonstrated performance gains on NUS-WIDE (Table 1), supporting our approach's broader applicability.

[W3] Model implementation for fair comparison.

To ensure a fair comparison with existing methods, we intentionally opted to implement ResNet-based CLIP. It's important to note that TaI-DPT and DualCoOp, which are SOTA benchmark methods, were also developed based on only ResNet architectures.

[W4] Explanation of terminology, "training-free" and "test-time adaption" in our method.

our approach can be accurately described as both training-free and employing test-time adaptation.

• Test-time adaptation: At inference, the visual features of a test image are adaptively projected onto the linear space of class concept representation $\mathcal{Z}$ through sequential attention. This adaptive process dynamically enhances visual-text feature alignment and occurs entirely at test time without requiring any training, making it a form of test-time adaptation.

This aligns with Reviewer jAJ4's strength comment that "The proposed method does not require training and can be seen as a form of test-time adaptation.".

[W5] Dependent on source text description.

Thank you for interesting suggestion. We acknowledge the reviewer's concern regarding the potential dependence of our method on the quality of text descriptions. To address this, we conducted experiments using both high-quality (MS-COCO captions) and low-quality text descriptions. The low-quality texts (e.g. "a art of a chair, mouse, car, and bus.") were generated naively by combining 1~4 random class names and simple handcrafted templates like "a photo of {}" and ""a art of {}".

Table 1: Evalation of our method with low-quality texts

Even when using low-quality, context-free text descriptions, our approach significantly improves the baseline (ZSCLIP). This indicates that while carefully curated descriptions can enhance performance, our method effectively leverages diverse text information from even simplistic combinations of random class names.

[Q1] Explanation of Softmax operation for measuring the similarity between visual and text features.

The cosine similarity between L2-normalized visual (f and F) and text (T) features is computed using matrix multiplication as detailed in Eq. 2 and 3. ($t$ refers to the transpose operation.) The softmax operation is subsequently applied to these cosine similarity values, and Figure 3 visualizes the resulting softmax values. We will clarify it in more detail within the paper to avoid any further misunderstandings.

[Q2] Performance improvement from class concept representation.

Class concept representation $t_{c}^{concept}$ combines vectors of text descriptions, capturing co-occurrence patterns with other classes and rich contextual information. Thus, $t^{concept}_{c}$ encapsulates essential information about relevant classes and nuanced context for the target class. We hypothesize that this process mirrors how human form concept on words from past experience and is crucial for multi-label recognition. This point implies that these phrase-level embeddings are more effective than using word-level default prompts.

To verified the effectiveness of $t_{c}^{concept}$, we compared the closest text captions in MS-COCO caption to both (1) hand-crafted prompts embeddings $t_{c}^{hand-crafted}$ (e.g., "a photo of") and (2) class concept representation $t_{c}^{concept}$.

• Top-3 co-occurrence objects of class "knife" in MS-COCO caption
  • dining table, person, cup
• Top-3 cloest texts with $t^{concept}_{c}$
  • A knife sticking out of the top of a wooden table.
  • A person holding a knife on a cutting board.
  • A guy looking down at a big knife he is holding.
• Top-3 cloest texts with $t^{hand-crafted}_{c}$
  • A knife is sitting on a plate of food
  • A sharp bladed knife left out on the ground.
  • A knife about to cut a birthday cake in pieces.

Results indicates that our $t_{c}^{concept}$ is better aligned with text descriptions containing multiple class names that are frequently co-occurring objects with target class "knife". This demonstrates suitability of $t_{c}^{concept}$ for multi-label recognition tasks where images depict complex scenes with multiple objects.

Thank you for your response. As mentioned in "[W3] Model implementation for fair comparison," the primary reason for omitting the implementation of a transformer backbone is that existing state-of-the-art (SOTA) methods (DualCoOp and TaI-DPT) are solely based on the CLIP ResNet architecture. To ensure a fair performance comparison with these methods, we also employ CLIP ResNet.

To explore the potential of our method with a different backbone, we extended our experiments to the ViT/B-16 architecture (Table 2). However, due to ViT's inherent lack of locality compared to CNNs [1,2], CLIP ResNet-101 shows better performance than CLIP ViT/B-16 in zero-shot CLIP (ZSCLIP). The performance improvement was also especially pronounced for CLIP ResNet-101. When it comes to capturing spatial information through local features, ResNet-based CLIP models are generally better suited for multi-label recognition tasks. To effectively handle local features, modifications to the vision transformer architecture are necessary, as exemplified by MaskCLIP [3].

Nevertheless, our proposed approach significantly improved zero-shot CLIP (ZSCLIP) performance for both ResNet and ViT backbones. Consequently, we have demonstrated the general applicability of our method to both ResNet and transformer architectures.

Table 2: Evalation of our method on CLIP ViT/B-16.

[1] Mao, Mingyuan, et al. "Dual-stream network for visual recognition." NeurIPS, 2021.

[2] Chen, Zhiwei, et al. "Lctr: On awakening the local continuity of transformer for weakly supervised object localization." AAAI, 2022.

[3] Zhou, Chong, et al. "Extract free dense labels from clip." ECCV, 2022.

We appreciate your valuable feedback and comments!

[W1] Additional multi-label few-shot method in Table 3.

As you mentioned, Tip-adapter is a training-free few-shot method proposed in 2021, primarily designed for single-label classification. To incorporate a recent few-shot method for evaluating multi-label recogntion, we adopt BCR[1], published in 2024, which is specifically designed for multi-label few-shot learning. BCR[1] has demonstrated superior performance, achieving 58.6 mAP for 1-shot and 66.5 mAP for 5-shot on the MS-COCO dataset. To provide a comprehensive evaluation, we will compare our method to BCR[1] in Table 3 of the main paper.

[1] An, Yuexuan, et al. "Leveraging Bilateral Correlations for Multi-Label Few-Shot Learning." IEEE Transactions on Neural Networks and Learning Systems (2024).

[W2] Emphasizing the "lable-free" in the abstract.

Thank you for pointing out the importance of emphasizing the 'label-free' aspect of our method. We agree that this is a crucial aspect of our work. We will revise the abstract to clearly highlight the 'training-free' and 'label-free' properties of our approach.

[Q1] Is generation of massive sentences time consuming?

The MS-COCO caption dataset we used is a publicly available dataset and was not generated using GPT-3.5.

For evaluating NUS-WIDE, we generated a large set of sentences using GPT-3.5. This process was relatively quick, taking only a few hours. In comparison to acquiring a labeled multi-label dataset, generating sentences was significantly less time-consuming.

I have read the author's response and maintain my rating.

Thank you for your instructive comments and suggestions.

[W1 & Q1] Clarification on Sec.3.2 Context-Guided Visual Feature.

We appreciate the reviewer's careful reading of Section 3.2. Iterative reduction of the number of groups $G$ is formalized in Eq.2 and 3. For a detailed explanation of the dimension changes in the sequential attention, please refer to the response of Reviewer 7zr3 [W2]. We will provide a detailed algorithm and correct Sec.3.2 for clear understanding.

For each dataset, we set $G=3$ and ($M_1,M_2,M_3$) as (40,40,25) for MS-COCO, (40,40,40) for VOC2007 and (40,40,36) for NUSWIDE. The modulation parameter $\alpha_{f}$ and $\alpha_{F}$ function as temperature values within the softmax operation.

As suggested by the reviewer, we explored selecting the top-K descriptions with the highest similarities before softmax in Eq.2 and 3 with $G=1$. The Table 1 demonstrates that this approach effectively improves performance without sequential attention. However, our proposed sequential attention on context-guided visual features achieved the best overall performance.

Table 1: Alternative approach using Top-k similarity selection

[W2] Scalability with a large number of classes.

As the number of target classes increases, our method demands a correspondingly large number of text descriptions. Handling such massive amounts of text can pose scalability challenges. To address this, we propose sequential attention, which efficiently manages numerous text descriptions as demonstrated in Figure 3(b).

[Q2] Correction for double-grained prompt (DPT) structure.

Thank you for your detailed feedback. As you correctly point out, while our method utilizes visual features at both local and global levels, similar to TaI-DPT, we does not incorporate text representations at either level. Consequently, our approach does not employ a double-grained prompt (DPT) structure. We will clarify this crucial distinction in the text to prevent any misunderstandings.

[Q3] Spatial aggregation sensitivity to the spatial extent of class.

Following the DualCoOp and TaI-DPT, we employ the spatial aggregation for obtaining local similarity score, $S^{loc}$, which is aggregated according to the $\textstyle\sum_{j=1}^{HW}\text{Softmax}(s_{c,j}^{loc})$ $\cdot s^{loc}_{c,j}$ as shown in Eq.4. The softmax operation makes this approach robust to variations in the spatial extent of each class.

[Q4] Intermediate setting in Table 5.

We conducted the sequential attention process with class concept representation $t^{concept}_{c}$ for the class-guided visual feature. In the table, $T$ refers to the text features matrix of all text descriptions. We will incorporate these results into the paper.

Table 2: Ablation study on each components of our method

[Q5] Performance convergence with increasing number of text descriptions.

In the Table 6, the rapid performance convergence at 1K texts for VOC2007 is primarily attributed to the dataset's small number of target classes (20 classes). In contrast, MS-COCO (80 classes) requires a larger number of texts for convergence.

While using exclusively MS-COCO captions might introduce redundancies as the texts grows, we believe that the quantity and diversity of descriptions are crucial to our method's effectiveness. Sampling strategies might limit the model's exposure to valuable information. Instead, we believe that incorporating text data from multiple sources would be a more promising approach to enhance performance and address potential redundancy issues.

[Q6] Leveraging the ground truth descriptions for test images.

Interesting question! As you noted, our method might be seen as retrieving the most relevant descriptions for multi-label recognition. However, it's crucial to emphasize that our proposed sequential attention mechanism differs from a simple retrieval approach by performing a weighted averaging of vectors within the linear space $\mathcal{Z}$ spanned by text features

We acknowledge that GT captions could potentially provide valuable information. However, our observation revealed that most GT captions lack comprehensive coverage of all object classes, especially for smaller or less prominent objects. This limitation motivated our decision to leverage a massive dataset of text descriptions, which offers a richer and more diverse words.

• Example of GT captions from MS-COCO dataset
  • Label of a text image - (person, bicycle, car, stop sign, backpack, handback)
• night is falling on an empty city street.
• a street view of people walking down the sidewalk.
• a street with a few people walking and cars in the road.
• woman walking down the side walk of a busy night city.
• a street with cars lined with poles and wires.

As you can see, these captions often omit certain object classes, such as the bicycle, backpack, and handbag. Note that we use the training annotation of MS-COCO caption for evaluation on MS-COCO and VOC2007.

[Q7] Performance discrepancies of DualCoOp and why DualCoOp++ results were not reported?

(DualCoOp) For Table 2, we use mean average precision (mAP) for zero-shot learning, while DualCoOp reports F1 score. Table 4 presents our reproduced DualCoOp results, which exhibit minor variations from the reported performance.

(DualCoOp++) Due to the unavailability of public code for DualCoOp++, we were unable to report its performance.

After reading all the reviewers' comments and the authors' answers, I believe my initial rating of 5: Borderline accept is a fair assessment of this contribution and thus maintain it.

We appreciate your insightful feedback and suggestions.

[W1] Effectiveness of class concept representation $t^{concept}_{c}$.

(Verification of class concept): To evaluate the class concept representation, we compared the closest text descriptions in the MS-COCO caption dataset to both (1) hand-crafted prompts embeddings $t_{c}^{hand-crafted}$ (e.g., "a photo of") and (2) class concept representation $t_{c}^{concept}$.

• Top-3 co-occurrence objects of class "knife" in MS-COCO testset
  • dining table, person, cup
• Top-3 closest texts with $t^{concept}_{c}$
  • A knife sticking out of the top of a wooden table.
  • A person holding a knife on a cutting board.
  • A guy looking down at a big knife he is holding.
• Top-3 closest texts with $t^{hand-crafted}_{c}$
  • A knife is sitting on a plate of food
  • A sharp bladed knife left out on the ground.
  • A knife about to cut a birthday cake in pieces.

Results indicates that our $t_{c}^{concept}$ is better aligned with text descriptions containing multiple class names that are frequently co-occurring objects with target class "knife". This demonstrates suitability of $t_{c}^{concept}$ for multi-label recognition tasks where images depict complex scenes with multiple objects.

(Difference from prompt ensembling[4]): While [4] demonstrate the effctiveness of equal weight averaging for text features from several hand-crafted template(e.g., "a photo of" and "a art of"), we averages the vectors of text descriptions including diverse contextual information associated with the target class name. Therefore, this enriched representation results in significant improvement of performance.

[W2] The explanation for reshaping the text feature T in "Context-Guided Visual Feature".

(Detailed explanation): In Eq. 2 and 3, the t refers to transpose as you mentioned. At the $k$ iteration, the transpose operation $t$ swap the dimension corresponding $M_k$ and $D$ of $T$ $\in$ $R^{M_1 \times \cdots \times M_G\times D}$, so $T^t$ $\in$ $R^{D \times \cdots \times M_G\times M_1}$. Let L2-normalized global feature $f$ $\in$ $R^{1\times D}$ and global context-guided visual feature $v^{(0)}=T, v^{(0)}$ $\in$ $R^{M_1 \times \cdots \times M_G\times D}$ and $(v^{(0)})^t$ $\in$ $R^{D \times \cdots \times M_G\times M_1}$.

For example in the case of Eq.2 at $k=1$, we process $v^{(1)} = Softmax_{\dim_1} \left(\frac{f (v^{(0)})^t}{\alpha_{f}} \right)v^{(0)}$ $\in$ $R^{M_2 \cdots \times M_G\times D}$, where $f (v^{(0)})^t$ $\in$ $R^{M_2 \times \cdots \times M_G \times M_1}$ and $Softmax_{\dim_1}$ applied along the dimension of $M_1$. Finally, we obtain the $v^{G}$ $\in$ $R^{1 \times D}$. We will clarify it in the main paper and add the detailed algorithm in the supplementary material.

(Ablation study on $G$ and $M_g$): The text feature $T$ is reshaped by pre-defined shape, not random reshaping. We have conducted the ablation study varying with the value of $G$ and $M_g$ in $T$ $\in$ $R^{M_1 \times \cdots \times M_G\times D}$.

Table 1: Ablation study for the number of groups $G$

Table 2: Ablation study for varing the value of $M_g$ for 64000 texts

The experimental result demonstrate that increasing $G$ and balancing the number of texts $M_g$ for each group is important to process sequential attention process. Note that we set ($M_1,M_2,M_3$) as (40,40,40) for VOC2007.

[W3] The implementation of partially labeled setting with our method.

In the partial label setting, we combine the prediction values of DualCoOp and our method by averaging them with a weight of 0.5 for each. Since DualCoOp was originally designed for practical partial label scenarios (limited-annotation MLR), our approach can enhance DualCoOp by providing complementary information during inference.

[Q1] Exploring equal weight averaging for class concept representation.

We have explored weighted averaging for class concept representation $t^{concept}_{c}$ using an attention mechanism. Weights $w$ were calculated by applying a softmax operation to cosine similarities between context-guided visual feature ($v^{(G)}$, $V^{(G)}$) and the set of corresponding text feature vectors for target class.

By varying the temperature $\alpha_{concept}$ for softmax in attention module, we observed that nearly uniform weight averaging yielded optimal performance with large values of $\alpha_{concept}$. For simplicity and efficiency, we adopt equal weight averaging.

Table 3: Ablation study for varing the $\alpha_{concept}$

[Q2] Incorrect grammar in L182.

Thank your for the comment. We will correct the grammar in L182.

[Q3] The class "sofa" is present in VOC2007 but absent from MS-COCO classes. How can we construct the class concept from MS-COCO captions?

While the class "sofa" is not explicitly defined as a category in the MS-COCO dataset, the term "sofa" appears frequently within its numerous text descriptions from MS-COCO caption dataset.

[Q4] Exploring different ratio $\alpha_{f,t}/\alpha_{F,t}$.

The main reason for $\alpha_{f,t}$/$\alpha_{F,t}=0.5$ is because it yielded the best overall performance across the dataset. We fixed the $1/\alpha_{f,t}$ at 80, 60, and 40 for MS-COCO, VOC2007, and NUS-WIDE, respectively.

Table 4: Ablation study for varing the ratio $\alpha_{f,t}$/$\alpha_{F,t}$ in the sequential attention