Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin

Thanks for your time and encouraging comments! We address each of your concerns below.

Q1: Ablations of hyperparameters are not included

Following your suggestion, we have conducted ablations of several hyperparameters in CAP.

Ablation of $t$

In CAP, we use $t$ to identify concept mismatched classes. Below, we report the results under UL setting with different values of $t$.

Table 1. Impact of threshold $t$.

$C$ is the number of classes in the dataset. The results show that CAP maintains consistent performance across different threshold values.

Ablation of $k$

In CAP, we use $k$ to determine the number of pseudo labels generated for each class. Results of ablation of different values of $k$ under SSL setting are presented below.

Table 2. Impact of pseudolabel number $k$.

The results demonstrate that CAP is generally robust to different choices of $k$.

Ablation of $\tau$

In CAP, we use $\tau$ as the confidence threshold to dynamically generate pseudolabels. We conducted experiments on different values of $\tau$ under UL setting. The results are as follows:

Table 3. Impact of confidence threshold $\tau$.

The results confirm that our method is robust to changes in $\tau$.

Q2: Confusing Figure 5

Following your valuable suggestion, we have revised Figure 5 to provide clearer visualization of CAP's key components while simplifying its overall presentation. The new figure is available at:

https://anonymous.4open.science/r/CAP-C642/framework_updated.pdf

We appreciate your insights and hope this revision improves clarity. We will incorporate all of the above experimental results and figures in the next version, and we welcome any additional feedback you may have!

Thanks for your time and helpful comments! We address each of your concerns below.

Q1: The number of baselines is limited

In our paper, we primarily compare against state-of-the-art methods specifically designed for leveraging CLIP’s zero-shot capabilities through pseudolabel generation. The number of such methods is relatively small—for instance, CPL only compares with three baselines.

To further enrich our comparisons, we have incorporated general semi-supervised learning methods as additional baselines, and the results under SSL setting are presented in the table below.

Table 1. Comparison of different methods under SSL setting.

The results show that CAP consistently outperforms these baselines on all datasets, with a significant improvement of 6.7% on RESISC45 and 9.8% on DTD.

[1] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Proc. of NeurIPS, 2020.
[2] FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning. In Proc. of ICLR, 2023.
[3] SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning. In Proc. of ICLR, 2023.

Q2: The scope of evaluation is somewhat narrow

We appreciate your suggestion to evaluate our method on more challenging tasks such as segmentation. However, our method is primarily designed to address the issue of pseudolabel imbalance caused by the image-level classification bias of VLMs, making it less directly applicable to segmentation tasks.

A key requirement of our approach is the ability to extract image features for each class and use them for clustering or similarity computation at different stages of the algorithm. In segmentation, however, a single image typically contains multiple classes, prevents the extraction of clean, class-specific visual prototypes. This intrinsic difference in problem formulation currently limits the applicability of our method.

In fact, existing works in top machine learning conferences that address this problem, such as CPL [4], LaFTer [5], and FineSSL [6], have similarly concentrated on classification tasks. This is because classification serves as a fundamental benchmark for evaluating improvements in pseudolabeling and adaptation strategies for VLMs.

While extending these techniques to segmentation or other structured prediction tasks is an interesting direction, it would require task-specific modifications beyond pseudo-labeling (e.g. in segmentation, one often needs to design a more advanced segmentation network). That being said, we fully acknowledge the importance of extending these techniques to structured prediction tasks like segmentation, and we will include this promising avenue in our future work discussion.

[4] Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proc. of ICML, 2024.
[5] LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections. In Proc. of NeurIPS, 2023.
[6] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. In Proc. of ICML, 2024.

---

We will incorporate all of the above experimental results and analyses in the next version.

Thanks for your time and helpful reviews! We address each of your concerns below.

Q1: A more formal or quantitative definition of concept mismatch and concept confusion

Generally, concept mismatch arises from a severe form of the semantic gap while concept confusion is more commonly observed between similar classes. We provide more formal definitions below:

Concept Mismatch: A class $y$ is considered to exhibit concept mismatch if the predicted label $\hat{y}$ for its visual prototype $\boldsymbol{v}_y$ (the average of visual features for class $y$) is incorrect, i.e.,

$$\hat{y} = \arg\max p(\boldsymbol{v}_y) \neq y$$

where $p(\boldsymbol{v}_y)$ is the predicted probability distribution over all classes for prototype $\boldsymbol{v}_y$.

Concept Confusion: Given the confusion matrix $\mathbf{M}$ predicted by CLIP, two classes $i$ and $j$ are considered to exhibit concept confusion if the misclassification between them is significant, specifically:

$$\mathbf{M} _ {ij} + \mathbf{M} _ {ji} > 0.25 \times (N_i + N_j)$$

where $N_i$ and $N_j$ are the total number of samples belonging to class $i$ and $j$, respectively.

Based on these definitions, we report the number of classes exhibiting concept mismatch and concept confusion in RESISC45 and DTD.

Table 1: Number of classes exhibiting concept mismatch and concept confusion.

We will incorportate these definitions into the next version of our paper to improve clarity and precision.

Q2: Computational cost of CAP

CAP, CPL, GRIP share two stages:

1. Pseudolabeling stage – Responsible for generating pseudo-labels, only needs to be run once per dataset.
2. Fine-tuning stage – Where the model is trained using the generated pseudo-labels.

Additionally, CAP employ MismatchDetection algorithm to fix mismatched concepts.

Below, we compare the computational cost of these two stages of training on EuroSAT for CPL, GRIP and CAP, using an NVIDIA RTX 4090 GPU and a Intel(R) Xeon(R) Silver 4314 @2.40GHz CPU:

Table 2: Computation time for each stage of CPL, GRIP and CAP.

In can be observed that the mismatch detection algorithm of CAP takes about 47s. In fine-tuning, CPL requires more time as it takes an iterative strategy requiring training to convergence multiple times (10 iterations in the original implementation). In each iteration, CPL expand the pseudo-labeled dataset. GRIP shares a similar trend with CPL.

Table 3: Computation time for each iteration of CPL.

Q3: Discussion of the limitations

One limitation of our method (CAP) is that it relies on a mismatch detection algorithm when handling concept mismatch. Currently, our detection method is a simple clustering-based algorithm, which, despite its computational efficiency, has room for improvement in detection accuracy and robustness.

In future work, we plan to explore more precise mismatch detection algorithms to better identify mismatched concepts, thereby further improving the robustness and accuracy of our method.

---

We will incorporate all of the above experimental results and analyses in the next version.

Thanks for your time and helpful comments! We address each of your concerns below.

Q1: The ablation studies could be more comprehensive

Following your feedback, we have conducted additional ablation studies on the components you mentioned.

Ablation of gamma Selection

We conducted experiments on different values of $\tau$ (we assume that the reference to gamma was intended to refer to $\tau$, the confidence threshold) under UL setting. The results are as follows:

Table 1: Performance regarding $\tau$ under UL setting.

It can be observed that our method is generally robust to changes in $\tau$.

Ablation of Independent Adapters

We evaluate our method with and without the independent adapters under UL setting. The results are as follows:

Table 2: Performance of different adapter configurations under UL setting.

It can be observed that both model gives better results than CPL, and shared adapters gives generally comparable results to independent adapters.

Impact of Prompt Tuning Methods

We replace MaPLe with two prompt tuning methods (i.e., UPT [1] and VPT [2]) and compare the performance under UL setting.

Table 3: Impact of different prompt tuning methods.

The results demonstrate that CAP can be effectively integrated with various prompt tuning methods.

[1] Unified Vision and Language Prompt Learning. ArXiv, abs/2210.07225, 2022.
[2] Visual Prompt Tuning. In Proc. of ECCV, 2022.

Q2: Abbreviations (e.g., CA, CACM) are not properly explained

Thank you for pointing this out. We apologize for the oversight. CA and CACM are abbreviations for Concept Alignment and Confusion-Aware Calibrated Margin. We will ensure that these abbreviations are properly explained in the next version.

Q3: Analysis of underperformance on CUB TRZSL

Thanks for pointing this issue out. By analyzing the confusion matrices, we found that our method has more classes with zero accuracy than CPL. To further study this, we conduct a in-depth analysis of the results on CUB.

CUB consists of 200 fine-grained bird species and exhibits frequent concept mismatches. For instance, CUB includes four species of orioles, but CLIP extracts highly similar text features for their category names. As a result, all four species are aligned with the visual features of a single dominant oriole species, leading to incorrect pseudo-labeling. Due to the simplicity of our mismatch detection algorithm, many of these mismatches go undetected, preventing affected classes from receiving correct pseudo-labels and limiting fine-tuning improvements.

In contrast, CPL assigns multiple pseudo-labels to each sample. Since the misclassified categories are semantically related, there is a higher chance that the correct label appears within CPL’s candidate label set. This enables CPL to improve accuracy on these classes more effectively, which explains its superior performance under this particular setting. However, such extreme concept mismatches are relatively rare across datasets.

Q4: Rationale behind (2)

The choice of similarity: In Equation (2), we use cosine similarity to measure the similarity between the text and visual prototypes of two classes, as cosine similarity is the most commonly used similarity metric in CLIP and relevant literature [3,4].

We take the maximum similarity value across the text and visual features because if two classes have high text feature similarity but low visual similarity, misclassification can still occur, and vice versa. By selecting the maximum value, we ensure that our method accounts for class pairs where concept confusion is most likely to happen.

[3] Learning Transferable Visual Models From Natural Language Supervision. In Proc. of ICML, 2021.
[4] Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?. In Proc. of ICLR, 2024.

---

We will incorporate all of the above experimental results and analyses in the next version.