Semi-Supervised CLIP Adaptation by Enforcing Semantic and Trapezoidal Consistency

Dear Reviewer JQns,

We deeply appreciate the reviewer's thoughtful comments. We are encouraged by comments such as well written and experiments are comprehensive. We will respond to each of your concerns below.

Weakness #1: The underlying motivation for surrogate caption generation process.

Response: Following your suggestion, we generated captions based on keywords using a large language model (GPT-4o). The comparison results are as follows:

In the table, we report the average performance of ZS (zero-shot), I2T (image→text retrieval), and T2I (text→image retrieval) on the remote sensing datasets. From the results, we found that the surrogate captions generated by SemiCLIP generally perform better than the captions generated by GPT-4o based on keywords. We believe this is due to the lack of relationships between concepts, which makes it difficult to reconstruct a complete caption based solely on keywords. As a result, the captions generated by GPT-4o still contain a significant amount of noise.

The surrogate captions in SemiCLIP are generated by the concatenation of keywords and learnable prompts, where learnable prompts can help surrogate captions better adapt to the task-specific domains, leading to improved performance. In fact, generating captions for images is a challenging task in scenarios with limited data and models. The surrogate captions we proposed offer a simple solution to this task, with hopes for better methods in the future.

Weakness #2: Why diagonal and legs regularization are beneficial?

Response: We first explain the reasons why diagonal and legs regularization are beneficial below:

(1) Traditional contrastive learning methods, such as CLIP, aim to reduce the distance between matching image-text pairs when learning image-text representations. However, they do not impose constraints on the overall geometric structure of the data, which leads to inconsistent predictions between the image and text spaces, especially in semi-supervised scenarios with only a small number of image-text pairs. Trapezoidal consistency addresses this issue by introducing equal-length legs and diagonals consistency regularization terms. These regularizers constrain the similarity gaps between mismatched image-text pairs as well as image-image and text-text pairs, resulting in a more consistent and structured representation space, thereby improving prediction consistency.

(2) Trapezoidal consistency can ensure geometric alignment between image and text representations, allowing the model to make more consistent predictions when reasoning in both the image and text spaces. This means that the image and text representations learned by trapezoidal consistency can be more easily interchanged, leading to improved performance on downstream tasks.

(3) The rigid separation between the positive pairs and negative pairs enforced by the contrastive loss in CLIP may degrade performance when some pairs in the negative batch belong to a similar entity. Trapezoidal consistency poses constraints on the overall geometry of all the data pairs rather than forcing a rigid separation, which enables the interaction of information between intra-modal and cross-modal samples, even in data-scarce scenarios.

Based on your suggestion, we also conducted relevant experiments using a soft-label approach, and the results are as follows:

The soft-label version 1 refers to the use of similarity between unlabeled images and surrogate captions to directly weight the contrastive loss applied to them, with the aim of mitigating the impact of noise in the surrogate captions. The soft-label version 2 is the Soft-PL, which employs soft nearest neighbor to achieve alignment for unlabeled images and has been compared in most tables in our paper.

The results indicate that the soft-label methods fail to effectively mitigate the noise in the captions under this scenario. However, our proposed diagonal and legs regularization, through indirect constraints on the unlabeled images and surrogate captions, effectively enhances the model's alignment performance.

I appreciate authors’ additional experiment and clarification, which resolve my concerns. I update the rating accordingly.

Dear Reviewer Psom,

We are grateful for the thoughtful reviews, and for the comments like effectively, results are promising, and easy to understand. We will address the concerns below.

Weakness #1: The analysis of prompting strategy.

Response: Thank you for your valuable suggestions to analyze the prompting strategy experimentally. We believe that a more in-depth investigation is indeed necessary. Based on your advice, we carried out the corresponding experiments, and report the average performance of ZS (zero-shot), I2T (image→text retrieval), and T2I (text→image retrieval) on the remote sensing datasets below:

(1) Comparing performance with prompts at different positions.

We conducted experiments on the effect of prompts at different positions, with all prompts initialized using a normal distribution. Overall, the performance is slightly better when the prompts are positioned in the middle, which is why we use this approach in our experiments.

(2) Different initialization.

From the results above, we found that zero initialization performed poorly in zero-shot tasks, with 1.1% lower performance compared to SemiCLIP, but showed no significant decline in retrieval performance. Normal initialization performs slightly worse than our proposed method in both zero-shot and retrieval tasks. Overall, SemiCLIP's initialization method achieves the most robust and stable performance.

(3) Fixed vs learnable prompts.

In fact, we have presented the relevant results in the ablation study, specifically in Table 6, and we have reorganized the results as follows:

The results show that learnable prompts achieve an average performance advantage of 0.9% over fixed prompts.

The experiments above demonstrate the effectiveness of the prompt strategy in SemiCLIP, and we will include the relevant results in the revised version of the paper.

Weakness #2: The quality of pseudo labels.

Response: We appreciate your insightful suggestions regarding the quality of pseudo labels, and the experimental results based on your advice are as follows:

For Common concepts, we use the class names that truly exist in the zero-shot test set of remote sensing as the concepts. For oracle fine-grained concepts, we extract the concepts from the real captions corresponding to the unlabeled images.

From the results, we can see that SemiCLIP achieves an average performance improvement of 0.9% compared to using common concepts, while using oracle fine-grained concepts provides an additional 0.8% improvement over SemiCLIP. This suggests that finer-grained concepts may have a more positive impact on performance. However, in practice, oracle fine-grained concepts are not accessible. SemiCLIP effectively leverages predicted concepts to achieve performance close to that of oracle fine-grained concepts.

Dear Reviewer Psom,

We sincerely thank the reviewer for valuable comments. We have addressed them in our responses and updated the manuscript accordingly. If the reviewer has any further questions, we are always ready to provide additional clarifications.

Best regards,

The Authors

Dear Reviewer 52fH,

We are grateful for the valuable reviews, and for the positive comments such as new method. We will address the concerns below.

Weakness #1: The method used to mine semantic concepts is not novel.

Response: During the semantic concepts mining (SCM), the model is adapted to a task-specific domain and trained with a classifier that aligns with the concepts in the images, which lays an important foundation for generating surrogate captions in the second stage. We believe SCM is an innovative approach to utilizing unlabeled data, and significant improvement in model performance is achieved through image-concept alignment in Figure (2a) and Figure (2b).

In fact, mining semantic concepts is not the primary innovation of this paper, and it mainly serves to subsequent generation of surrogate captions and further performance improvement via consistency loss. The improvement in mining semantic concepts does contribute to the performance enhancement for the problem addressed in this paper, but it is not the primary factor. Therefore, we chose to keep its design simple, allowing for further improvements in future research.

Weakness #2 and Question #2: Why it is necessary to restrict the trapezoid to have equal-length legs and diagonals?

Response: We will explain the reasons why it is necessary to restrict the trapezoid to have equal-length legs and diagonals below:

(1) Traditional contrastive learning methods, such as CLIP, aim to reduce the distance between matching image-text pairs when learning image-text representations. However, they do not impose constraints on the overall geometric structure of the data, which leads to inconsistent predictions between the image and text spaces, especially in semi-supervised scenarios with only a small number of image-text pairs. Trapezoidal consistency addresses this issue by introducing equal-length legs and diagonals consistency regularization terms. These regularizers constrain the similarity gaps between mismatched image-text pairs as well as image-image and text-text pairs, resulting in a more consistent and structured representation space, thereby improving prediction consistency.

(2) Trapezoidal consistency can ensure geometric alignment between image and text representations, allowing the model to make more consistent predictions when reasoning in both the image and text spaces. This means that the image and text representations learned by trapezoidal consistency can be more easily interchanged, leading to improved performance on downstream tasks.

(3) The rigid separation between the positive pairs and negative pairs enforced by the contrastive loss in CLIP may degrade performance when some pairs in the negative batch belong to a similar entity. Trapezoidal consistency poses constraints on the overall geometry of all the data pairs rather than forcing a rigid separation, which enables the interaction of information between intra-modal and cross-modal samples, even in data-scarce scenarios.

Question #1: Suggestions for improving the paper title.

Response: Thank you for your suggestion! We will carefully consider your suggestion and make some revisions to the title.

Dear Reviewer 52fH,

We sincerely thank the reviewer for valuable comments. We have addressed them in our responses and updated the manuscript accordingly. If the reviewer has any further questions, we are always ready to provide additional clarifications.

Best regards,

The Authors

Dear Reviewer WcrK,

We appreciate the reviewer for the thoughtful reviews and the encouraging comments such as clear manner, easy to understand, and grasp the key ideas without difficulty. Following are our responses to the concerns.

Weakness #1: Semantic Concepts Mining method is not novel.

Response: During the semantic concepts mining (SCM), the model is adapted to a task-specific domain and trained with a classifier that aligns with the concepts in the images, which lays an important foundation for generating surrogate captions in the second stage. We believe SCM is an innovative approach to utilizing unlabeled data, and significant improvement in model performance is achieved through image-concept alignment in Figure (2a) and Figure (2b).

Compared to concept labeler in [1], the differences are as follows:

(1) We train a linear classifier to achieve better image-concept alignment in task-specific domain, while concept labeler in [1] obtains concepts through image-concept retrieval, which may not enable effective retrieval in certain specialized domains due to the niche nature of some concepts.

(2) Concepts generated in [1] are considered as conditional input to the cross-modal decoder, which requires a large amount of data for training, significantly increasing the overhead. However, SemiCLIP can be trained with only a small amount of labeled data, achieving significant improvements in model performance in semi-supervised settings.

Given the similarities between [1] and our method, we will cite [1] in the revised paper and include it in the related work section.

[1] Nlip: Noise-robust language-image pre-training

Weakness #2: Caption-level trapezoidal consistency builds incrementally on CyCLIP.

Response: While there is a formal similarity between SemiCLIP and CyCLIP, we argue that the real innovation is found in the deeper insights and practical applications of the method, rather than its form. Therefore, we summarize the innovation and contribution of caption-level trapezoidal consistency (CTC) below:

(1) CTC introduces CyCLIP into the semi-supervised learning scenario by utilizing unlabeled images and surrogate captions, thereby extending the applicability of CyCLIP beyond complete image-text pairs.

(2) We found that CyCLIP can partially alleviate the impact of noise in the captions on training and provided a geometric explanation. From experiments in Figure (2c), if we directly constrain the reduction of the lower base of trapezoid, the model's performance will decrease by an average of 4.07%. This indicates that the direct alignment of unlabeled images and surrogate captions suffers from performance degradation due to the noise present in surrogate captions, whereas trapezoidal consistency effectively mitigates this issue and significantly improves performance through interactions between samples from both image and text modalities.

(3) CTC offers new insights for future work addressing less-than-ideal alignment in image-text pairs, which is a widespread issue in practical settings.

Dear Reviewer WcrK,

We sincerely thank the reviewer for valuable comments. We have addressed them in our responses and updated the manuscript accordingly. If the reviewer has any further questions, we are always ready to provide additional clarifications.

Best regards,

The Authors

Question #5: The sizes of datasets in the experiments.

Response: For remote sensing datasets, the RSICD, UCM, and Sydney datasets contain 8734, 1680, and 497 image-text pairs, respectively. The three datasets constitute RS-ALL, and we randomly select 10% of the image-text pairs from the training set as labeled data, with the remaining pairs treated as unlabeled. During the inference, the sizes for classification datasets are below:

For fashion datasets, the Fashion200k, FashionGen, and Polyvore datasets contain 61753, 60147, and 71967 image-text pairs, respectively. The sizes for classification datasets are below:

For other datasets, SciCap contains 106934 image-text pairs and Simpsons only contains 720 pairs.

We will provide further details on the sizes of the datasets in the paper.

Question #6: Why do the proposed model suppress CLIP (fine-tuned) and the upper bound for performance.

Response: The poor performance of CLIP (fine-tuned) can be attributed to the fact that it was trained solely on labeled data. The possible upper bound of the performance is supervised learning on all training data, which we refer it as Oracle (fully supervised fine-tuned). We show the averaged oracle performance below:

Interestingly, our method even outperforms the oracle calculated here in zero-shot performance. We believe this is due to the severe imbalance in the proportions of the sub-datasets within the training sets, RS-ALL and Fashion. When training on the full dataset, sub-datasets with a smaller proportion, such as UCM, perform poorly on their corresponding zero-shot test set, UCM-CLS, which resulted in a slightly lower overall performance. We find this to be an interesting phenomenon, and it warrants further research into how to mitigate the issue of imbalance in the proportions of the sub-datasets.

The oracle outperforms SemiCLIP in retrieval performance, indicating that increasing the amount of labeled data will significantly improve retrieval performance. This also points to a potential direction for further improvement of SemiCLIP in the future. We will include the oracle results in the experimental tables of the paper, helping readers gain a clearer understanding of the task.

Question #7: Why is the lower base not used in Figure 1(b)?

Response: In caption-level trapezoidal consistency, since surrogate captions for unlabeled images may not be reliable, we avoid directly constraining their relationship and do not directly use the lower base. In fact, by imposing constraints on the upper base, legs, and diagonals, we enable the interaction between in-modal and cross-modal samples, ensuring coherence among samples within each modality and substantially enhancing the model's overall alignment ability.

In Figure (2c), if we directly constrain the reduction of the lower base, the model's performance will decrease by an average of 4.07%. This indicates that the direct alignment of unlabeled images and surrogate captions suffers from performance degradation due to the noise present in surrogate captions, whereas trapezoidal consistency effectively mitigates this issue and significantly improves performance through interactions between samples from both image and text modalities.

Dear Reviewer zoSv,

We sincerely appreciate the reviewer for thoughtful feedback. We are encouraged for comments like a new way of using unlabeled images. We address your concerns one by one.

Weakness #1: The contribution of each stage.

Response: In the first stage, the model is trained using the contrastive loss from CLIP with labeled data, along with the soft cross-entropy loss for the semantic linear classifier as defined in Eq. (2). During this stage, the model is adapted to a task-specific domain and trained with a classifier that aligns with the concepts in the images, which lays an important foundation for generating surrogate captions in the second stage. It is worth noting that in Figures (2a) and (2b) of the paper, we observed a significant improvement in model performance after the first stage (i.e., the SPT stage) compared to fine-tuning CLIP, indicating that the alignment of concepts with the images during this phase contributes to enhanced visual representation capabilities. In the second stage, SemiCLIP achieves further performance improvement by leveraging concept-level semantic consistency and caption-level trapezoidal consistency.

The performance variations across the two stages and comparisons with other methods are presented below:

CLIP (fine-tuned) refers to CLIP fine-tuned using only labeled data. From the table above, Stage 2 achieved an average performance increase of 3.6% over Stage 1. In addition, Stage 1 outperformed CLIP (fine-tuned) by an average of 2.8%, indicating that the alignment of concepts with images helps improve visual representation.

Weakness #2: The performance on common zero-shot classification and multimodal retrieval.

Response: Our method aims to adapt the model to a task-specific domain by leveraging a small amount of labeled data and a large number of unlabeled images and realize a notable enhancement in model performance within this domain. However, it is evident that after the adaptation, the model may lose some of its broader generalization capability [1,2] due to catastrophic forgetting, so the model's performance on common zero-shot classification and multimodal retrieval is unlikely to achieve promising results.

However, we believe that in semi-supervised settings, adapting to a task-specific domain while maintaining the pre-trained model's original generalization ability will be an intriguing research topic, and we may explore this area more deeply in the future.

In addition, to evaluate the performance of our method on common dataset, we conducted experiments on CoCo, and the averaged retrieval results are as follows:

I2T and T2I represent image→text retrieval and text→image retrieval respectively. The results indicate that SemiCLIP can achieve significant performance improvements on common datasets over CLIP (fine-tuned) and S-CLIP. It is worth noting that S-CLIP's performance shows an average decrease of 4.5% compared to CLIP (fine-tuned), aligning with the paper's claim [3] that S-CLIP experiences performance drops when trained on a small number of image-text pairs in common datasets like CoCo. However, the superior performance of our proposed SemiCLIP is unaffected by the different types of datasets, achieving significant improvements on both commonly used datasets and task-specific datasets.

[1] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

[2] An empirical study of catastrophic forgetting in large language models during continual fine-tuning

[3] S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions