Finetuning CLIP to Reason about Pairwise Differences

We thank the reviewer for their time in providing a thoughtful review. We appreciate that the reviewer found that our experiments are comprehensive and that our evaluations are interesting and insightful.

[T]he experiment only covers LLaMA2-13B-chat-hf as the LLM. I wonder if the authors have any insight into whether the training is based on LLaMA2 but the inference is using any smaller/larger models. Would the change in the distribution of the output impact the performance of the model?

We believe that using different LLMs during training (e.g., for generating synthetic data) and during inference (e.g., for creating comparative prompts or extended class prompts) will not largely impact the performance of the model. In fact, for our comparative prompts and extended class prompts, we use GPT4, and we still see our improved performance with this change in distribution. Our rationale is that there are many different ways to represent class names or text differences in natural language, and with the sufficient diversity in our synthetic training data for PC-CLIP as well as the original pretraining data, we believe that our PC-CLIP model can account for these distributional differences.

I'm curious if this approach could be combined with other CLIP improvement techniques. For example, NegCLIP [1] improves compositional reasoning by focusing more on the text encoder. This method feels like a nice complement to it.

Yes, we believe that other CLIP improvement techniques can be complimentary to PC-CLIP. For the case of NegCLIP, this seems to further improve compositional reasoning, which would pair nicely with incorporating notions of pairwise differences. In our new experiments, we also show that using PC-CLIP with image-captioning techniques is also complimentary, achieving strong performance coupled with existing pipelines for image-difference captioning. Thus, it seems that PC-CLIP can be complimentary to many other methods, to achieve even stronger performance.

Thank you for your time in providing a thoughtful review. We appreciate that the reviewer found that our work provides extensive experimental validation with reasonable baselines.

Scaling up: The experiment focuses only on a single CLIP model size (ViT-L/14)

In response to your request, we provide an experiment where we train a ViT-H/14 model and show the results, compared to standard CLIP features with this model. We observe the following results:

This demonstrates that PC-CLIP is indeed effective at larger CLIP model scales and can scale up.

While the method shows consistent improvements on standard classification tasks, many of the gains are relatively small (1-2% absolute improvement). It is unclear whether the cost is worth such small improvements.

We argue that performing a single round of finetuning is worth the cost given that it would generally improve performance on a wide variety of tasks (including classification, difference-based classification, image generation). We believe that this is a simple single-time cost for benefits that we later observe during every inference pass. Also, we believe that the consistent improvements over standard classification tasks, as well as other applications, such as image-difference classification and text-to-image generation make our overall finetuning (< 20 hours on a single A6000 GPU) certainly worth the cost.

The method's core dependency on LLM-generated comparisons introduces risks from hallucination and noise in the training signal, with limited analysis of how comparison quality impacts downstream performance or how to ensure robust comparison generation.

We agree that depending on LLMs for generating synthetic data introduces complications; however, this is a standard practice done in the field. Furthermore, our work benefits from the improvements of better LLMs as well as better sampling techniques from these LLMs to detect and reduce hallucinations [1, 2]. Thus, we don’t see this as a limitation of our work, and it is certainly a common practice in the recent literature. Please also see our experiment regarding adding the more noisy examples that were previously filtered — we see only a slight drop in performance, but still stronger performance than the baselines. Thus, our method is robust to noise in the LLM generations.

What is the difference between Table 2 and Table 3? The numbers in Table 3 are much lower than in Table 2. Also, the PC-CLIP baseline numbers are much lower than CLIP (e.g., CUB 59.43 vs 52.57). Also, Table 3 is never referenced in the main text.

The numbers in Table 3 reflect the performance on the most highly confused 3 pairs of classes in standard zeroshot classification. The numbers for both PC-CLIP and CLIP on these tasks being lower than CLIP in Table 2 make sense than as they are the hardest classes for these models (Table 3), as opposed to over all classes (Table 2). Table 3 not being referenced is a typo on our part; thank you for pointing this out! We have updated this in our revision, we had meant to say "highly confused in Table 3" instead of "highly confused in Section 4.3".

What is the difference between Table 2 and Table 3? The numbers in Table 3 are much lower than in Table 2. Also, the PC-CLIP baseline numbers are much lower than CLIP (e.g., CUB 59.43 vs 52.57). Also, Table 3 is never referenced in the main text.

The numbers in Table 3 reflect the performance on the most highly confused 3 pairs of classes in standard zeroshot classification. The numbers for both PC-CLIP being lower than CLIP in Table 2 make sense than as they are the hardest classes for these models (Table 3), as opposed to over all classes (Table 2). The focus of Table 3 is to look at the relative improvement of comparative prompting with PC-CLIP or CLIP, in the (+ comp) column. The main takeaway is that comparative prompting seems to always benefit PC-CLIP, while this is not the case for CLIP.

Table 3 not being referenced is a typo on our part; thank you for pointing this out! We have updated this in our revision, we had meant to say "highly confused in Table 3" instead of "highly confused in Section 4.3".

[1] Farquhar, et. al. Detecting hallucinations in large language models using semantic entropy.

[2] Ji, et. al. Towards mitigating LLM hallucination via self reflection.

Could you perform similar experiments to demonstrate some main results conclusions still hold for this model, like a comparison prompt?

In response to your request, we have provided results with the larger CLIP model built off of the ViT-H when using comparative prompting and with extended class prompts. We still observe better performance at this larger model scale with both of these approaches on a majority of the considered tasks.

Comparative Prompting

Extended Prompts

We thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have.

Dear Reviewer Juc5,

In our response above, we have tried to address all your comments and concerns. To summarize, we have done:

• Added new experiments with finetuning larger CLIP (ViT-H) models, demonstrating better performance with ZS classification at scale
• Added results in terms of comparative prompting and using extended class descriptions, which show PC-CLIP's benefits over the CLIP baseline.
• Addressed a question on robustness to hallucinations in the LLM generations -- with new experiments that demonstrate leaving in the poor quality generations still leads to improved performance over CLIP *Addressed a question on the differences between Table 2 and 3

Thank you again for taking the time to review our work and we hope to hear back from you soon. Please let us know if you have any additional questions!

We thank the reviewer for their detailed review and for their constructive feedback. We hope to answer your questions and address your concerns below with some of our responses and new experiments.

lacks a good motivation on why CLIP models should exhibit the structure of purely-language based text embeddings... CLIP pretraining never encourages such structure to emerge so it is not unusual to observe such behavior.

While it isn’t unusual to observe such behavior in CLIP models as it has not been encouraged, we believe that it is a desirable property. For instance, the embedding space is much more interpretable and reflective of how we understand concepts with arithmetic and analogies. Furthermore, we argue that this is not directly encouraged via text pretraining objectives, although this is still a result of text-based training. Finally, we argue that the benefits across our various different experimental settings also support that CLIP benefits of reflecting this improved embedding space that better captures notions of differences.

does not seem relevant in the context of large multimodal models

While this is true that other large multimodal models would have stronger reasoning capabilities due to their LLM components, the main focus of this paper is to teach CLIP this ability. While CLIP is indeed an older model, it still gets plenty of use in things such as text-to-image generation via Stable Diffusion. We would also like to highlight that it is a component in many of these large multimodal models (including LLaVA). Thus, improving this foundation model can have large benefits in many pipelines that build off of it (though we feel this is better left for future work), making it an important model to study and improve, with many potential downstream impacts.

quite detached from the prior work on difference based image captioning [1,2]

Indeed, thank you for pointing out this related work. We highlight that the main focus of our work is different — improving CLIP’s embedding space to reflect these differences, which is the reason why our paper may seem detached from prior work in difference-based image captioning. In these papers, which we already discuss in our related work section, the goal is to generate text descriptions of the difference between two images, often using the CLIP model. We focus on the reverse — with a large synthetically generated set of these text descriptions of differences, can we improve CLIP embeddings (in terms of zeroshot and difference-based classification) by finetuning on these differences. This is a key distinction from these prior works. We have made this clearer in our revision, and we believe that our inclusion of new experiments in image-difference captioning better connect our work with this subfield of research.

In reality, we would need PC-CLIP for such tasks more than difference-based classification. There is no evaluation on such tasks… There is prior work that aims to address a similar problem [3,4]. The paper neither discusses nor compares with any of these methods.

In response to your request, we provide additional experiments on Image-Difference captioning and retrieval. To perform evaluations on such tasks, we use PC-CLIP paired with [3] for image-difference captioning (as they use the original vanilla CLIP weights). This allows for a fair comparison of the usability of PC-CLIP versus the standard approach in [3] that uses CLIP. We restate the experimental results from the overall response here for clarity:

Spot-the-diff retrieval

Spot-the-diff caption

We observe that using PC-CLIP for such a task is indeed helpful, as we achieve stronger performance than simply using the CLIP weights in the pipeline of [3] on text to image-pair retrieval and on 3 of the 4 metrics for image-pair captioning. PC-CLIP matches performance on the last metric. Thus, this demonstrates the utility of PC-CLIP for such image-difference captioning and retrieval tasks.

We remark that the dataset Clevr-Change is no longer downloadable via the links in [3], so we cannot provide results on that dataset. Furthermore, [4] does not provide any code, so we cannot replicate their experiments. We have added these new experimental results and references into our revision.

Comparative prompting is not well-grounded... In Table 2, we observe that comparative prompting leads to worse results than non-comparative prompting for CIFAR100, CUB, and almost similar results for SUN397. Additionally, comparative prompting requires a user to create difference captions and tune the alpha hyperparameter. It is not clear whether such efforts are worth the gains on completely unseen tasks and scenarios in comparison to a CLIP model which does not require any of these.

Yes, we agree that comparative prompting indeed does not always help performance. However, we argue that boosting performance on 2/5 tasks, while maintaing performance on 1 task shows that it is a viable alternative — showing large benefits in some cases (e.g., EuroSAT).

We believe that providing a natural way to incorporate prior domain knowledge into the zeroshot classifier in the form of difference captions is important and beneficial to performing classification with CLIP-style VLMs. This makes image classifiers via CLIP more controllable and usable by specific domain experts via these comparative prompts.

Apart from zero-shot classification, CLIP model achieved high-level robustness to natural distribution shifts (ImageNetv2/R/A/S). It is unclear how PC-CLIP would perform in such cases.

As per your request, we provide new results on a natural distribution shift benchmark of ImageNet-R and ImageNet-A.

We observe that PC-CLIP also achieves slightly stronger performance on these natural distribution shifts as well as the other downstream tasks with likely to be stronger distribution shifts from the CLIP pretraining data. That being said, we also believe that our datasets, such as EuroSAT, include data with very significant distribution shifts (e.g., satellite images) that are even more difficult than the natural distribution shifts mentioned by the reviewer. Thus, with our original datasets and these new natural distribution shift datasets, our experiments show the benefits of PC-CLIP over a variety of distribution shifts in practice.

Additionally, there are no linear probing results to understand the quality of the learned vision embeddings.

In response to your request, we provide additional results in terms of linear probing PC-CLIP embeddings. We train linear probes on top of the vision embeddings produced by CLIP and PC-CLIP (when doing full finetuning). We observe the following results:

PC-CLIP achieves better linear probing performance on a majority of tasks. We remark that while there are slight improvements on the vision encoder, there are much larger benefits for the text encoder. As mentioned in our paper in Table 6, we have results showing that the text embeddings are significantly improved in terms of localizing the differences between class names on various downstream tasks. This aligns with prior work that primarily highlights issues with CLIP’s text embedding space [1].

[1] Yuksekgonul, et. al. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?

We thank the reviewer for their constructive feedback. We hope to answer your questions with our new experiment and address your concerns below.

The main contribution of the paper is limited…. While Table 1 results on difference-based classification are impressive, the rest of the results show very marginal improvements over the baselines.

We respectfully disagree with the reviewer that our contribution is limited. As you point out, our finetuning on pairwise differences is indeed novel, and the process to generate this data at scale with LLMs is also a novel contribution. While in some experiments the improvements are slight, we argue that performing this finetuning and then observing gains on the majority of downstream tasks is significant. In fact, the generality of our experimental results is one of the main appealing factors of PC-CLIP.

We have shown gains in terms of zeroshot classification (with various different techniques, including different LLM-generated class prompts), improved text embeddings (Table 6), improved arithmetic in embedding space for text-to-image generation (Table 5 and Figure 4), and new experiments that show improvements in image-difference retrieval and captioning (Tables 7 and 8). The fact that we achieve widespread benefits in the variety of considered experimental settings shows the significance of our approach.

lot of repetition in the form of summarizing the same set of results/observations over and again

This is a good observation. The repetitive nature of the dataset often comes from the fact that we have generated pairs from 1000 examples, and individual images will show up in a large number of pairs. This can naturally be reduced by considering a larger dataset over more than 1000 examples, and only allowing for examples to show up in a small number of pairs. This is a useful suggestion, and we hope to include this in future work. We believe that our results are strong even with these images appearing in multiple repetitive pairs.

In Sec 4.1, it is mentioned that the 1000 randomly sampled images were paired and a filtering was applied to remove poor quality generations -- are there any ablations without such a filtering?

In response to your comment, we performed an additional ablation where we finetune PC-CLIP on the full unfiltered set of data (where 200k examples have been filtered from the total 990k). We observe the following results (bolding the best-performing method and italicizing the second-best-performing method).

Zeroshot classification

Difference-based classification

We observe that while PC-CLIP with filtering achieves the strongest performance on a majority of tasks, PC-CLIP Unfiltered outperforms vanilla CLIP weights on 4 of the 5 tasks, and even outperforms PC-CLIP with filtering on one task. This supports that our finetuning method is robust to noise in the LLM generations.

"highly confused in Table 3" instead of "highly confused in Section 4.3"

Thanks for pointing that out! We have fixed this in our revision.

Dear Reviewer i1CP,

In our response above, we have tried to address all your comments and concerns. To summarize, we have done:

• Addressed a new experiment that demonstrate the robustness of PC-CLIP to the unfiltered LLM generations
• This experiment still shows improved performance in terms of ZS classification and difference-based classification *Addressed a question on repetition in the generations, due to images being reused in multiple pairs

Thank you again for taking the time to review our work and we hope to hear back from you soon. Please let us know if you have any additional questions!