OC-CLIP : Object-centric binding in Contrastive Language-Image Pretraining

We thank the reviewer for their time and their extensive feedback. We believe we improved our work considerably and address the concerns in the following.

We made the following additions that we further detail below :

• Additional Benchmarks (VL-Checklist + full ARO)
• Additional Baselines (SVLC, DAC, DCI)
• Extensive modules + loss ablation study
• Ablation of different parser families
• Training from scratch of CLIP and OC-CLIP on CC12M and zero-shot classification evaluation.

Additional Experimental Comparisons

Our goal is to evaluate CLIP-like models' compositional understanding in plausible and grammatically correct cases. The SugarCrepe benchmark has identified exploitable textual biases in previous mainstream procedurally-generated hard negatives benchmarks like the COCO and Flickr set of ARO and VL-checklist. Specifically, $1$ shows that procedurally generated hard negatives are either highly grammatically incorrect and can be identified by a blind model or by a good language model. SugarCrepe proposes to follow the same fine-grained taxonomy on attributes, objects, relationships as VL-checklist but ensures that the hard-negative are not distinguishable by a blind model. However, as requested by the reviewer we added the results on the full ARO suite and VL-Checklist in Table 5. We also added DAC, DCI, SVLC as suggested as baselines in sugarcrepe and ARO/VL-checklist. As expected these methods still fall behind in splits that require specific object-attribute binding :

Ablation Study

We appreciate the reviewer's feedback and have addressed the concerns regarding ablation studies. In response to points 3 and 4, we have added an extensive ablation section (Appendix A.1) that thoroughly examines the specific object-centric design choices related to OC-CLIP.

In particular, our analysis includes an ablation study of the loss function and we investigate the impact of the following key components: Local loss, competitive attention, default token, relation scoring module modularity.

These ablations provide valuable insights into the importance of each component in the overall performance of our model. Most notably removing the local loss (as shown in Table 6) effectively impacts downstream relational understanding on SugarCrepe with a swap obj accuracy decreasing from 80.7 to 73.1 and a replace rel accuracy decreasing from 80.6 to 74.7.

We believe that this additional analysis strengthens our paper and provides a more comprehensive understanding of the OC-CLIP architecture. We hope that this revised version addresses the reviewer's concerns.

Scene Graph Accuracy

We want to stress the fact that the specific LLM instantiation is not a claimed contribution but rather a tool used in the proposed method. In particular, we consider an LLM-based parser as the best automated option currently available for obtaining scene graphs.

To support this claim, we have added a quantitative and qualitative ablation study (Appendix A.3) that compares three different parser families:

• LLM-based parsers (llama3)
• Automatic parsing based on Spacy
• Supervised scene graph parsers (T5-based)

This qualitative and quantitative ablation study provides valuable insights into the strengths and weaknesses of each parser family, illustrating some important failure modes (see added Table 9 in the Appendix) of automatic and supervised scene graph parsers that impacts downstream compositional performance when OC-CLIP is trained on those scene graphs (see added Figure 6 in the Appendix). We notice the strength of the parser impacts the relational understanding performance with a swap obj (llm) : 80.7 (supervised) :77.5, (automatic) : 71.4 and replace rel (llm) 80.6, (supervised) 74.3, automatic (66.5) which is consistent with the qualitative failure modes identified in Table 9.

[1] SugarCrepe : Fixing Hackable Benchmarks for Vision-Language Compositionality

Analysis of OC-CLIP’s general understanding capabilities

There is a misunderstanding on the fine-tuning point highlighted by the reviewer. There is no fine-tuning of the binding module involved in our method. We are training the binding module from scratch on top of a pre-trained CLIP backbone and comparing it to methods that do finetuning and do not add any additional parameters. Regarding the performances of the binding module itself, it is strictly bound by the vocabulary seen during its training (eg. COCO/VG/GQA) and cannot be expected to generalize further. The only way to assess correctly the differences between CLIP and OC-CLIP in both general and compositional image understanding abilities is to train CLIP and OC-CLIP from scratch on the same training data. To that end, we added CC12M experiments where both backbones are trained from scratch. Notably here the scene graphs are obtained from a noisy and non-human curated dataset. We added those results in Table 4 and 7 and noticed improvements in general understanding with +9.2% on Imagenet zero-shot classification performance while maintaining a significant gap in zero-shot sugarcrepe swap attribute (+15.9%) and swap obj (+14.3%) splits. We added the extensive zero-shot classification results on the ELEVATER suite in Table 7 [1].

Analysis of Loss Function Components

The proposed loss function includes multiple components, including the local graph contrastive loss, which is designed to prevent the relational module from collapsing and ignoring visual slots. We provide a detailed motivation for adding this component in Appendix A.1. (paragraph Local Graph contrastive loss)

Regarding the weighting components, specifically α and β, we clarify that these are learned parameters, not hyperparameters. This means that they are automatically adjusted during training to optimize the model's performance. We clarified the choice of initial value in Appendix A.1 (paragraph Similarity Score).

To further demonstrate the importance of the local loss component, we provide an additional experiment (Table 6) where we remove the local loss and evaluate the impact on relational understanding performance. The results show that removing the local loss drastically impacts relational understanding performance (-7.5% and -5.9% in the swap obj and replace rel sugarcrepe splits) highlighting its effectiveness in the overall model.

We hope we have addressed the reviewer concerns with those additional experiments and are happy to answer to any further questions the reviewer might have.

[1] ELEVATER : ELEVATER : https://computer-vision-in-the-wild.github.io/ELEVATER/

We thank the reviewer for all those precise comments/questions and address them in the following.

We made the following additions that we further detail below :

• Clarification of the goal of the synthetic data
• Training from scratch of both CLIP and OC-CLIP on cc12m and evaluation on zero-shot classification from ELEVATER
• Additional baselines (svlc, dac, dci)
• Extensive ablation study on the model + loss design choice
• Ablation on different parser families

Synthetic Data Experiments

We thank the reviewer for this suggestion. It’s true that the synthetic experiments have taken too much space in the first paper version in contrast to real world experiments. To have a better balance between synthetic and real experiments, we moved part of the synthetic experiments now to the Appendix A.5 and added new experiments on the CC12M dataset instead (see later comments).

The primary goal of the synthetic data experiments was to isolate the root cause of the bag-of-words behavior in CLIP models and evaluate the effectiveness of different approaches in addressing this issue. Those experiments allowed us to make the following observations about the use of hard-negatives:

• (1) simply adding more hard-negatives plateaus in terms of model accuracy, and
• (2) it is not sample-efficient, even in an easy synthetic environment.
  Notably, the attribute-binding results on synthetic data (Table 1) are consistent with the gap in swap attribute performance on SugarCrepe of OC-CLIP and both small-scale (NegCLIP, CE-CLIP ..) and large-scale (DAC, DCI, CLIP-SVLC that we added as requested) hard-negative methods. Our results show that all methods plateau and do not see any improvement, with OC-CLIP achieving over 88%.

General Understanding Capabilities

There is a misunderstanding on the fine-tuning point highlighted by the reviewer. There is no fine-tuning of the binding module involved in our method. We are training the binding module from scratch on top of a pre-trained CLIP backbone and comparing it to methods that do finetuning and do not add any additional parameters. Regarding the performances of the binding module itself, it is strictly bound by the vocabulary seen during its training (eg. COCO/VG/GQA) and cannot be expected to generalize beyond the seen vocabulary. The only way to assess correctly the differences between CLIP and OC-CLIP in both general and compositional image understanding abilities is to train CLIP and OC-CLIP from scratch on the same training data. To that end, we added CC12M experiments where both backbones are trained from scratch. Notably here the scene graphs are obtained from a noisy and non-human curated dataset. We added these results in Table 4 and 7 and noticed improvements in general understanding with +9.2% on Imagenet zero-shot classification performance while maintaining a significant gap in zero-shot sugarcrepe swap attribute (+15.9%) and swap obj (+14.3%) splits. We added the extensive zero-shot classification results on the ELEVATER suite in Table 7.

Additional Baselines

In our study, we focused on comparing to hard-negative-based baselines that generate hard negatives in-distribution, specifically from the COCO/VG/Flickr domain. This is why we included methods such as NegCLIP, CE-CLIP, and CounterCurate, which also focus on generating hard negatives within these specific domains. However, we acknowledge the importance of broader comparisons and have now included additional methods such as DAC, DCI and CLIP-SVLC to our analysis (see Table 2 and Table 5). These methods are different in that they incorporate both hard negative mining on a larger scale and dense captioning/recaptioning, which is not the primary focus of our work. Nonetheless, their inclusion provides a more comprehensive comparison and allows us to better position our approach within the broader landscape of compositional reasoning benchmarks. Notably, we see that on the sugarcrepe/ARO splits that assess for precise object-attribute binding (swap attribute/ARO-A) all these methods still fall behind :

Which confirms the results we presented with the synthetic experiments that the hard-negative approach, even at scale, is not sufficient to solve the bag-of-words behaviour of CLIP and justifies the inclusion of more structured inductive biases within the architecture of the model like we do for OC-CLIP

We hope that this expanded comparison addresses the reviewer's concern and provides a clearer understanding of our approach and its context within the field.

We designed OC-CLIP to tackle the object-binding problem, most illustrated by the swap attribute of SugarCrepe. All the baselines we compare to start from a an already strong backbone. When OC-CLIP is trained form scratch on cc12m it rather outperforms all the considered baselines even those fine-tuned in domain with targeted hard-negatives. Could the reviewer explain why they think those results are not in favor of OC-CLIP inductive biases?

See the swap attribute performance below :

NegCLIP and CE-CLIP finetune OpenCLIP on in-domain data (COCO/VG) and DAC on large-scale high-quality data (through recaptioning+ hard negative LLM-based generation), OC-CLIP here is trained from scratch on noisy data and still outperforms them.

We thank the reviewer for the answer. We would like to ask for some clarifications of the requests :

We think that there is a misunderstanding in expecting our model not losing their zero-shot capabilities. Our model does not rely on the initial image-text alignment (from both CLS tokens of the backbones, those are not used anymore) but rather learns a new aligned embedding space from scratch and cannot be expected to retain any zero-shot capabilities. Our model is not finetuned but also trained from scratch for the alignment part (binding + relationship module) from which the similarity score is derived. Hard-negative based methods to which we compare only finetune an existing powerful backbone and can be expected to retain some of the original zero-shot capacities. Comparing our OC-CLIP to the originally aligned CLIP is not fair setting. Could the reviewer explain why they think it is an informative comparison?

We put however the requested results here :

We thank the reviewer for their valuable feedback as they allowed us to considerably improve our work. In the following we tackle the raised points.

We made the following additions that we detail below :

• Extensive ablations of the model design choices + losses
• Qualitative and quantitive ablation of different parsers families
• Parsing method cost detail
• Comparison with TagAlign

Contribution Clarification

We would like to emphasize that our contribution resides in the design of a binding module that learns to extract visual slots conditioned on open-ended object queries and the modeling of the relationship component of the similarity score and not the use of the LLM as a parser. Our binding module is designed to learn a shared embedding space between open-ended object queries and visual parts. This allows us to extract visual slots that are relevant to the object query, enabling more accurate compositional understanding. The exact design choices are not straightforward. We added extensive ablations about them in Appendix A.1. Importantly, without the local contrastive loss that we propose, the relational understanding performance drops significantly as seen in the newly added Table 6.

LLM-Based Parser as a Tool, not a Contribution

We want to emphasize that the specific LLM instantiation used in our parser is not a part of our contribution, but rather a tool that we utilize to obtain scene graphs. We consider an LLM-based parser as the best automated option currently available for this task.

We added a quantitative and qualitative ablation study (Appendix A.3) that compares three different parser families:

• LLM-based parsers (llama3)
• Automatic parsing based on Spacy
• Supervised scene graph parsers (T5-based)

This ablation study provides valuable insights into the strengths and weaknesses of each parser family, illustrating some important failure modes (see added Table 9 in the Appendix) of automatic and supervised scene graph parsers that impacts downstream compositional performance (see added Figure 6 in the Appendix).

Comparison with TagAlign

We were not aware of this work and thank the reviewer for pointing us to it. We added a comparison in the added Scene Graph Discussion section (Appendix A.3). TagAlign also uses an LLM-based text parser, but there are significant differences between both approaches :

• Finite vs open-ended objects: TagAlign adds a fine-grained classification objective to the image representation by extracting a finite set of tags for objects and attributes. In contrast, our approach focuses on capturing open-ended objects and does not require the extraction of a finite set of tags. The fine-grained alignment is learned with a contrastive objective.
• Relationships: Moreover, TagAlign does not handle relationships, which is a crucial aspect of our approach. Enumerating a finite set of triplets (predicate, object, subject) does not seem reasonable, as it would lead to an explosion of possible combinations.

We additionally hypothesize that the classification objective in TagAlign might actually amplify the bag-of-words behavior when representing multiple objects. For example, "red apple and green banana" and "green apple and red banana" would be represented by the same classification labels measuring the presence of “red” and “green” attributes as well as “apple” and” banana” objects without any binding distinction. Unfortunately, the weights are not available, so we were unable to add this to the comparison.

In conclusion, while TagAlign shares some similarities with our approach, there are significant differences in both methods. We believe that our approach provides a more comprehensive understanding of relationships between objects and is better suited for open-ended compositional reasoning tasks.

Parsing Cost

We performed the parsing by serving N instances of Llama3-8b on v100 machines. Each dataset is then chunked into K processes that do not require any GPUs and send requests to the served LLM parsers through vllm to maximize the throughput of the parallelized requests.

For instance we parsed the COCO dataset (∼ 500k captions) parallelizing N=10 instances of the parser, with K=128 chunks in 3.5 hours. Similarly, we parsed the Visual Genome dataset (∼ 200k captions) with N=8 instances, K=64 chunks in 1.7 hours. We added the details in Appendix A.3.

We hope that this analysis addresses the reviewer's concern and provides a clearer understanding of the computational cost associated with using an LLM for parsing

$1$ vllm: https://github.com/vllm-project/vllm

We thank the reviewer for their feedback, it allowed us to improve our work considerably. In the following we tackle the raised points.

We have made those following additions based on the reviewers concerns and detail them further below :

• we trained both OC-CLIP and CLIP from scratch on CC12M and compared their zero-shot classification and compositional understanding.
• we added a computational analysis of our binding module (in terms of FLOPs) as suggested.

Additional Downstream Tasks

During the rebuttal period, we scaled up our experiments to train both CLIP and OC-CLIP from scratch on CC12M dataset to demonstrate the scaling potential of our approach. We train both models from scratch on 20 epochs of CC12M and report zero-shot downstream classification performance on the ELEVATER [1] suite in the newly added Table 7 and Table 4. Our results show that OC-CLIP achieves significant performance gains in zero-shot classification (e.g. +9.2% in ImageNet) compared to CLIP while maintaining a significant gap in compositional understanding in the zero-shot sugarcrepe swap attribute (+15.9%) and swap obj (+14.3%) splits.

We hope that these additional results address the reviewer's concern on the additional downstream performance of OC-CLIP and the usefulness of the introduced inductive biases to mitigate the bag-of-word behaviour previously identified in CLIP-like models.

Computational analysis

We also addressed the concerns regarding the computational overhead introduced by the binding module in OC-CLIP in the Appendix Table 8. We acknowledge that the binding module introduces additional complexity to the CLIP model, which can result in increased computational overhead due to the cross-modal attention operations. But, unlike CLIP’s text encoder, OC-CLIP only needs to encode information about single open-ended objects and relationships and thus requires much less capacity than CLIP that needs to encode whole sentences composed of multiple objects and relations between them. With this observation, we employed several strategies the in the experiments that we added on CC12M (Section A.2 of the Appendix):

• We used a smaller embedding size (256 vs 512) and number of layers (6 vs 12) in the text encoder.
• We operated on a reduced embedding space (256) for the binding module and projected the ViT-B-16 patches from a 768 to a 256 embedding space before computing the nodes to patch cross-attention logits.

These optimizations allowed us to reduce the computational overhead while maintaining the performance gains achieved by our approach.

To additionally quantify the resulting computational overhead, we performed an analysis of the binding module's impact on the overall computational cost of OC-CLIP in the added Table 8. Our results show that the binding module introduces an overhead when using a base architecture (2.2x FLOPS), but this overhead is reduced when scaling the ViT backbone to a L backbone (1.3x FLOPS) because the cost of the binding module is not the bottleneck anymore.

We hope that this analysis addresses the reviewer's concern and provides a clearer understanding of the trade-offs between computational overhead and performance gains in our approach. We are happy to answer any further questions the reviewer may have.

[1] ELEVATER : ELEVATER : https://computer-vision-in-the-wild.github.io/ELEVATER/

Zero-shot Classification

Several reviewers had concerns regarding whether OC-CLIP retains its original image understanding capabilities after fine-tuning with the scene graph. In our compositional understanding experiments, we compare our approach with data-centric fine-tuning methods that do not add any additional parameters. These methods are expected to retain some of the general capabilities of the initial backbone. However, our binding and relationship modules are trained from scratch on a smaller scale dataset, which means they may not generalize as well to unseen data and can only be expected to work well within the vocabulary domain they have been exposed to (e.g., COCO/VG/GQA in our experiment setting).

In order to assess how training on scene-graph affects general understanding capabilities, we added experiments where both CLIP and OC-CLIP architectures are trained from scratch on a noisy and non-human curated dataset , CC12M $1$ . This allows us to compare both models' general understanding and compositional downstream performance in the newly added Table 4 and 7. Notably we notice an improvement in general understanding with +9.2% on Imagenet zero-shot classification performance while maintaining a significant gap in zero-shot sugarcrepe swap attribute (+15.9%) and swap obj (+14.3%) splits. We added the extensive zero-shot classification results on the ELEVATER $2$ suite in Table 7 and we leave further scaling experiments for future work.

Computational Analysis

Some reviewers asked to comment on the trade-offs of the computational overhead and performance gains of our method. We acknowledge that the binding module introduces additional complexity to the CLIP model, which can result in increased computational overhead due to the cross-modal attention operations. However we were able to mitigate it in the scaling experiments because of the way OC-CLIP binds objects to visual parts. Importantly, unlike CLIP’s text encoder, OC-CLIP only needs to encode information about single open-ended objects and relationships and thus requires much less capacity than CLIP that needs to encode whole sentences composed of multiple objects and relations between them. With this observation, we employed several strategies the in the scaling experiments that we added on CC12M (Section A.2 of the Appendix). We added in Table 8 a quantitative analysis of the FLOPS comparison of CLIP and OC-CLIP and show that our binding module is not a bottleneck any more when scaling the ViT backbone.

We hope that this analysis addresses the reviewer's concern and provides a clearer understanding of the trade-offs between computational overhead and performance gains in our approach.

Scene Graph Discussion

Finally we want to stress the fact that the specific LLM instantiation is not a claimed contribution but rather a tool used in the proposed method. In particular, we consider an LLM-based parser as the best automated option currently available for obtaining scene graphs.

To support this claim, we have added a quantitative and qualitative ablation study (Appendix A.3) that compares three different parser families:

• LLM-based parsers (llama3)
• Automatic parsing based on Spacy
• Supervised scene graph parsers (T5-based)

Most notably we notice in Figure 5 the strength of the parser impacts the relational understanding performance with a swap obj (llm) : 80.7 (supervised) :77.5, (automatic) : 71.4 and replace rel (llm) 80.6, (supervised) 74.3, automatic (66.5) which is consistent with the qualitative failure modes identified in Table 9.