Weak Supervision from Vision-Language Models to Self-Improve on Downstream Tasks

We thank the reviewer for positive feedback on the clarity of the paper's structure, the thoroughness of our ablation studies, and the usefulness of the qualitative examples in illustrating the model’s performance and limitations. Below, we provide detailed responses to the questions and suggestions.

Lack of explanation for some hyperparameters.

We have revised the manuscript to ensure all the hyper-parameters are clearly mentioned by name throughout the paper. Specifically

$λ$: loss factor controlling the importance of partial label learning loss

$q$: number of quantiles, which defines the number of quantiles that the unlabelled samples are divided into before filtering.

$p$: number of samples selected from each cluster for pseudo-labelling.

For the number of quantiles ($q$), we initially reported a wide range (including the optimal value) for brevity. However, we also explored other intermediate values. In the revised manuscript, we have expanded Table 7c (included below) to show these intermediate values (e.g., $q=10$). Our experiments indicate that the best performance is achieved at $q=5$.

Why does CPL under 4-shot experiments fall short against SelfPrompt?

We believe CPL falls short against SelfPrompt in 4-shot experiments because of its random selection of labelled samples, which often includes noisy and less representative data. This results in a suboptimal model that struggles to generalize effectively. The suboptimal model, in turn, generates noisy pseudo-labels through CPL's incremental pseudo-labelling approach, which results in inefficient use of unlabeled data and a decline in the final model's performance. On the other hand, our proposed solution selects the most representative set of samples as the labelled set, and our clutter-guided pseudo-labelling and confidence-aware semi-supervised learning ensures optimal utilization of the labelled set budget and learning from the unlabelled set.

Results for number of samples larger than 50?

The performance saturates at $p = 50$. In the revised manuscript, we now include results for a larger number of samples selected per cluster ($p$). Specifically, we show that increasing $p$ to 75 results in a slight performance decline. This decrease occurs because higher values of $p$ increase the likelihood of including incorrect pseudo-labels, as cluster-guided pseudo-labelling tends to select samples farther from the cluster centers.

Potential impact of dropping different sizes of high and low confidence sets?

Thank you for this interesting question. Performance can vary with the portion of high and low confident samples removed from the unlabelled set. As discussed in Section 3.2, we control the number of samples to be dropped with a quantile value, $q$. For example, $q = 20$ indicates that the unlabeled dataset is divided into 20 quantiles and removes the lowest and highest 5% of samples while retaining 90% of the samples. As shown in Table 7 (b), the best result is obtained when we filter the most and least confident 20% of the sample ($q=5$).

Does cluster-guided pseudo-labelling not rely on VLM predictions for pseudo-labelling?

The clusters are formed using only the image embedding from the vision encoder of the VLM and do not utilize the prediction (using both vision and text encoder) for pseudo-labelling. Specifically, unlike prior works, we do not utilize the zero-shot prediction from the VLM to generate the pseudo-labels. To avoid any confusion, we have revised the concerning line from "We propose a novel clustering-guided pseudo-labelling approach that does not rely on the VLM to generate the pseudo-label." to the following: "We propose a novel clustering-guided pseudo-labelling approach that does not utilize the zero-shot prediction from the VLM as the pseudo-label."

Additionally, the bias in the zero-shot predictions of the VLM does not necessarily carry over to our cluster-guided pseudo-labelling approach, as it only utilizes the vision encoder of the pre-trained VLM, while the VLM prediction requires both text and vision encoder and is often sensitive to the prompt used with the text encoder for zero-shot prediction. Our empirical evaluation, shown in Figure 1 (left), demonstrates that the proposed pseudo-labelling method consistently generates more accurate pseudo-labels compared to prior approaches.

Values of the hyper-parameters, q, τ, and λ?

In the revised manuscript, we discuss the values of q, τ, and λ in the implementation details, along with the sensitivity study on q, and τ in Table 7. Specifically, all the main results are reported with the values of $q = 5$, $τ = 0.05$, and $λ = 1$.

Undefined and unclear variables.

Thank you for pointing this out. Following is a clarification of the variables:

• $j \in \{1,2,...,N\}$ is the cluster index.

• $X_{weak}$ is composed of the remaining samples from the unlabelled set $U$ that are not part of the pseudo-label set $\mathcal{X}^{+}$. We have revised the definition of $X_{weak}$ as: $\mathcal{X}_{\text{weak}} = \{(x_i, s_i) \mid x_i \in U \setminus \mathcal{X}^{+}\}$

• $X_{weak}$ was a typo intended to be $\mathcal{X}_{weak}$.

• $f(x)$ is the prediction of the VLM, $f(x) = p(y| x; \theta, \phi, P)$.

• Subscripts confusion. To avoid any confusion with the subscripts, we have revised Eq. 6 by using superscript to represent the element of vector $s$.

We have now clarified all of these in the paper on lines 254, 291, 295, 184, and 299, respectively.

Minor problems.

Thank you very much for pointing these out. All the minor problems are now addressed in the revised manuscript.

How do the three datasets $X_{L}$ , ${X}^{+}$, and $X_{weak}$ change during the tuning process

$X_{L}$ is the labelled set, which remains constant throughout training. Following prior works such as GRIP and CPL, the training is conducted over $S$ sessions, where $\mathcal{X}^{+}$ and $X_{weak}$ are linearly expanded by including samples from unlabeled set $U$.

Could authors provide a pseudo-code to better understand how the method is implemented?

Certainly. Please refer to the pseudo-code (Algorithm 1) in the Appendix of the revised manuscript.

We thank the reviewer for the positive feedback on our paper's extensive experiments and the method's performance. Below, we address the specific weaknesses and provide detailed responses.

Authors should discuss related weakly supervised learning papers.

In the revised manuscript, we have added a section to discuss the related work on weakly supervised learning. Specifically, we have included the following:

"Weakly supervised learning encompasses a class of methods that train models with limited or imprecise supervision (Peyre et al., 2017; Li et al., 2019; Zhou, 2018). Unlike fully supervised learning, which requires large amounts of precisely labelled data, weakly supervised learning leverages weak annotations, such as noisy, incomplete, or coarse-grained labels. This paradigm significantly reduces the reliance on costly and time-intensive data annotation processes. This paradigm has demonstrated broad applicability across various domains, including vision-language models (Wang et al., 2022b), medical image analysis (Kanavati et al., 2020). Despite its potential, it has been underexplored in the context of semi-supervised learning, where pseudo-label predictions frequently introduce noise. This positions weakly supervised learning as an ideal candidate to address the challenges posed by noisy labels in semi-supervised frameworks."

Labelled set being selected randomly in other works.

Thank you for this interesting question. Please note that standard semi-supervised learning papers do indeed select the labelled set randomly from the whole dataset. It's neither fixed nor provided. Specifically, a subset of samples is randomly selected from the unlabeled set along with the ground truth labels to form the labelled set. Please refer to the seminal SSL work FixMatch [1] (page 17, Section C). As mentioned in this paper, the method selects the labelled set randomly, trains the model with different random labelled sets, and reports the average and standard deviation. Other popular methods, such as FlexMatch, MixMatch, and ReMixMatch, followed the same protocol of random selection of labelled sets. The previous SOTA (CPL [3]) on tuning VLMs with semi-supervised learning also followed the same protocol. We would like to clarify that there is no mention of how the labelled set is created in the original paper, but the official implementation of the method shows that the labelled set is, in fact, selected randomly, like all methods mentioned above: https://github.com/vanillaer/CPL-ICML2024/blob/master/methods/main_SSL.py#L74

We acknowledge that the idea of diverse and representative labelled set selection can be utilized with other existing methods too. In the table below, we show the results of our weakly-supervised sampling (WS) with existing methods and also present the performance of SelfPrompt with and without a weakly-supervised sampling module. As we find from this table, SelfPrompt (without WS) outperforms the prior works by 4.71%, while WS with GRIP, CPL, and SelfPrompt improves the performance by 1.72%, 1.67%, and 3.21%, respectively. In the revised manuscript, we have now included a new section on Page 7 to include these results.

Thank you to the authors for their detailed rebuttal. However, my main concerns about problem setting and motivation still exist. Specifically,

1. Problem setting. Please note that in semi-supervised learning, we are given with a small-size labeled dataset and a large-size unlabeled dataset. Data and labels in the labeled dataset are sampled from an unknown distribution p(x, y) but not the unlabeled dataset, and data from the unlabeled dataset are sampled from an unknown distribution p(x) (refer to the problem background or problem setting in [4,5]). The labeled dataset is not from the unlabeled dataset and we can not control the samples in the labeled dataset. Moreover, I have carefully checked the paper of Fixmatch and did not find that the authors selected labeled data from the unlabeled datasets (if I missed it, please point that out).

2. Motivation. I understand that N is the number of labeled data. However, to me, N does not have any semantic meaning, so it is not clear why authors conduct k-mean clustering with N clusters and claim that N clusters will have distinct features.

Given that these concerns still exist, I will currently maintain my score.

[4] A survey on semi-supervised learning

[5] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

I agree with Reviewer homK that this framework is arguably valid. As noted in my initial review, labeling important data within a limited budget is a significant and practical problem. However, it is fundamentally different from semi-supervised learning. In my view, the authors may consider reframing the narrative of the paper to focus on the context of labeling important data with a limited budget rather than positioning it within the scope of semi-supervised learning. However, in my view, considering the substantial changes required and the limited remaining time available for ICLR discussion, the current paper may not yet be ready for publication and may be better revised and submitted to other conferences later.

Please note, as you mentioned, in Fixmath, the labeled data are sampled from the dataset (data distribution) rather than the unlabeled dataset. As I previously stated, this experimental setup is designed to mimic real-world conditions where we cannot choose which data to label. Therefore, unfortunately, this approach is flawed within the context of semi-supervised learning.

Dear Reviewer homK and Reviewer 352h,

We deeply thank the reviewers for their continued engagement. This means a lot, regardless of the outcome of this submission, and has without a doubt increased the quality of our paper. We would like to clarify that the overarching goal of our method is to fine-tune a pre-trained model on a downstream task. Nonetheless, semi-supervised learning is one of the benchmarks we use to evaluate our solution (alongside base-to-novel generalization), following prior works addressing the same problem. However, since our setup assumes the ability to select the labelled set (inspired from the active learning literature), based on your recommendation, we have now slightly re-positioned this work as active semi-supervised learning. Accordingly, we present our results in two distinct setups as follows. First, we use a randomly selected labelled set, similar to standard semi-supervised learning literature (please see Section 4.2 of the revised manuscript). Second, we evaluate our method with the additional ability to choose the strategic labels to sample. These results are now presented in Section 4.3. We observe that in both setups (as well as the third setup of base-to-novel generalization), our method achieves very strong results, outperforming other methods. Kindly see the revised manuscript, where the changes regarding this discussion have been highlighted in red.

We hope that this repositioning of the work, along with using two explicitly separate experiment setups (covering both random sampling and selected sampling), addresses the concerns regarding this particular component of our method. If the reviewers have any additional questions or concerns, we would be more than happy to provide further clarifications or make additional changes to the paper in the remaining duration of the rebuttal period.

Dear Reviewer 352h,

As we approach the end of the rebuttal period, we would like to offer any further clarifications regarding our paper if there are additional questions. As mentioned in our previous response, we have slightly adjusted the positioning of our paper and presented results for both the standard and our proposed active semi-supervised learning setups. We hope this change, along with additional results (Section A.2), sensitivity analysis (Table 7), further explanations of the proposed solution (Section 3), expanded related work (page 3), comparisons to more methods (Table 1, 2), pseudo-code (Section A.3), and other minor changes made during the rebuttal, have addressed your concerns and strengthened the overall quality of our paper. If you believe these updates have successfully addressed your concerns, we kindly ask you to consider raising your score to reflect the improved state of the paper.

Once again, we would like to sincerely thank you for your comments and constructive discussions, which have without a doubt significantly improved our paper.

Best regards,

Authors

We appreciate the reviewer’s positive feedback, highlighting that the paper is well-written and easy to follow, as well as recognizing the promise of our results and the comprehensiveness of our experiments, including the ablation study. Below, we address the questions and comments in detail.

The number of baselines is too small.

Our work focuses on the semi-supervised tuning of VLMs, a relatively new area compared to the more established fields like few-shot tuning, where the pre-trained encoder is fine-trained using a combination of a few labelled samples and large amounts of unlabeled data. We have compared our results with all prior methods in this domain, similar to the previous SOTA CPL (published at ICML 2024), which we present in Tables 1 and 2. Additionally, as per your suggestion, we have now reproduced the SOTA (PromptKD) on the base-to-novel generalization task (Table 5) in the semi-supervised learning setup using the same training details (encoder and prompt) as our proposed solution. As we find from the Table below (please refer to Table 1, 2 in the revised manuscript for more details), SelfPrompt (ours) outperform PromptKD by a large margin, indicating the effectiveness of our solution in utilizing the unlabelled data while fine-tuning the pre-trained VLM.

VLM probabilities are used to determine which samples are used in clustering.

The VLM is only used to filter out some of the samples in the weakly-supervised sampling module, but not for pseudo-labelling. Specifically, unlike prior works, we do not utilize the zero-shot prediction from the VLM to generate the pseudo-labels. To avoid any confusion, we have revised the concerning line from "We propose a novel clustering-guided pseudo-labelling approach that does not rely on the VLM to generate the pseudo-label." to the following: "We propose a novel clustering-guided pseudo-labelling approach that does not utilize the zero-shot prediction from the VLM as the pseudo-label."

By labels, is this referring to the pseudo-labels?

In this context, labels refer to ground truth labels and not pseudo-labels. In a semi-supervised setting, we have a large unlabelled set, and we select a small subset (randomly or strategically) from the unlabelled data to form the labelled set. This set is sampled from ground truth labels provided with the dataset.

What does the loss function on line 282 optimize?

The loss function optimizes the learnable tokens/prompts added to the pre-trained encoder. This is discussed in line 181 of section 3.1 of the paper. We used the same model and prompt tuning approach as the prior works in this area (CPL, GRIP) and did not introduce/change any details to ensure a fair comparison to prior work.

Please note that it is not unrealistic to select samples from an unlabelled set for labelling. There is an entire field of research called active learning [1,2] that focuses on selecting and labelling samples from unlabeled data that not only selects the initial labelled set but also uses a trained model to identify and select the most informative samples for further labelling. The question asked by Reviewer 352h concerns whether, in existing works, the labelled set is pre-defined or selected randomly. We consider this an important question because if prior works used a pre-defined labelled set, our approach of strategically selecting the labelled set would result in an unfair comparison. However, if the labelled set was randomly sampled from the whole dataset in prior works, our method of strategic sampling can not be considered unfair. Our response clearly addresses this concern by referencing prior papers and their implementation, which explicitly indicate that the labelled set was randomly selected from the entire dataset. Therefore, we believe our problem setup is not flawed. For convenience and clarity, we have copied our full discussion to Reviewer 352h at the end of this response.

Nonetheless, as per the suggestion of Reviewer 352h, we have now included new results for both our method and prior methods by considering strategic samples as a new setup. Specifically, we demonstrate the impact of the SelfPrompt's other two components separately while incorporating the sampling technique into both our method and prior works. We believe this completely eliminates any concerns regarding fair comparison. We believe our response clearly resolves any concerns regarding unfair/flawed experimental design, but we are more than happy to answer any further questions on this matter.

References:

1. Brame, Cynthia. "Active learning." Vanderbilt University Center for Teaching (2016).
2. Felder, Richard M., and Rebecca Brent. "Active learning: An introduction." ASQ higher education brief 2, no. 4 (2009): 1-5.

Dear Reviewer homK,

As we approach the end of the rebuttal period, we would like to offer any further clarifications regarding our paper if there are additional questions. In response to your previous comment (agreeing with Reviewer 352h), we have slightly adjusted the positioning of our paper and presented results for both the standard and our proposed active semi-supervised learning setups. We hope this change, along with additional results (Section A.2), sensitivity analysis (Table 7), further explanations of the proposed solution (Section 3), expanded related work (page 3), comparisons to more methods (Table 1, 2), pseudo-code (Section A.3), and other minor changes made during the rebuttal, have addressed your concerns and strengthened the overall quality of our paper. If you believe these updates have successfully addressed your concerns, we kindly ask you to consider raising your score to reflect the improved state of the paper.

Once again, we would like to sincerely thank you for your comments and constructive discussions, which have without a doubt significantly improved our paper.

Best regards,

Authors

We thank the reviewer for the positive feedback, acknowledging the significant performance gains achieved by our method, the insightful identification of the sampling issue, and the comprehensiveness of our experiments. Below, we provide detailed responses to the questions raised.

Lack of technical novelty

The proposed method is not primarily based on pseudo-label thresholding and selection. Out of the 3 modules in our proposed method, only the 3rd one (confidence-aware semi-supervised learning) uses the concept of pseudo-label thresholding. Specifically, our proposed confidence-aware semi-supervised learning is a hybrid approach of fully-supervised and weakly-supervised learning, which learns from the high-confident samples in fully-supervised learning while learning from the low-confident samples in a weakly-supervised setting. Overall, This paper introduces three key technical novelties that distinguish it from existing literature. Specifically, we identify and address 3 key limitations in existing approaches to tuning VLMs in a semi-supervised learning setup: (a) the random selection of labelled set by existing methods does not adequately represent the underlying data distribution, leading to inefficient use of the limited label budget, (b) utilizing the zero-shot capabilities of pre-trained VLMs leads to a considerable number of incorrect pseudo-labels, (c) incremental pseudo-labelling approach of existing approaches lead to accumulation of noisy pseudo-labels, ultimately leading to performance degradation. Each of these issues contributes to a significant drop in the final performance of the model and requires careful consideration and unique solutions to mitigate their adverse impact. Following are our solutions to each of these problems:

Contribution (a): We introduce the concept of weakly supervised labelled set sampling to overcome the limitation of existing solutions of not selecting the most representative set of samples as the labelled set. The proposed weakly supervised labelled set sampling is a novel technique that consists of a two-step protocol. The first step removes the least informative and noisy samples with our filtering with weak supervision approach, and the second step selects a diverse set of samples as the labelled set using our diversity sampling technique. No prior works have explored such an approach for sampling such a representative set of labelled sets for semi-supervised learning.

Contribution (b): We introduce the concept of cluster-guided pseudo-labelling. Unlike existing methods that utilize the zero-shot capabilities of pre-trained VLMs, leading to a considerable number of incorrect pseudo-labels due to miscalibration in the VLM, we propose to leverage the clusters formed in the weakly-supervised sampling module to generate the pseudo-labels. To our knowledge, this approach to pseudo-labelling has never been explored in the context of semi-supervised learning.

Contribution (c): Finally, we introduce confidence-aware semi-supervised learning. The proposed solution is a novel technique that combines fully-supervised learning with weakly-supervised learning to enable efficient utilization of the unlabelled data. Our solution is the first of its kind to use such a hybrid approach to utilizing the labelled data.

Does it involve dataset-specific tuning?

No. Our proposed solution does not involve any dataset-specific tuning. A key strength of our method is that the solution is dataset-invariant and does not require any data-specific tuning. All the results reported for datasets use a single set of hyper-parameters. All the hyper-parameters and implementation details are provided in section 4.1 to ensure the reproduction of our proposed solution.

Two recent baselines are neglected.

Thank you for pointing out these two papers. In the revised manuscript, we have discussed these two papers under the prompt tuning section (Page 3) of the related work. However, we can not directly compare our solution with these methods as our approach is based on semi-supervised learning, whereas the two mentioned papers focus on few-shot learning. Unlike our solution, these methods are unable to leverage unlabeled data.