Object-centric binding in Contrastive Language-Image Pretraining

We thank the reviewer for their insightful feedback and take the opportunity to address the listed concerns below :

Q1 - Missing Text-only Vera baseline

We acknowledge the importance of including the Vera baseline in our analysis and will incorporate it into our updated manuscript for both Table 1 and Table 2. Our results demonstrate that our model's enhancements are robust and extend beyond the capabilities of text-only baselines, which strengthens our contribution.

We would like to emphasize that the Vera text-only was initially used by the Sugarcrepe paper as a motivation to develop their benchmark and to challenge models beyond mere plausibility and grammar correctness In fact, the Vera baseline has a random chance performance in all the Sugarcrepe splits.

We will update our results tables with the Vera scores for the sake of completeness of baselines coverage. We thank the reviewer for this suggestion!

Q2 - Hard-negative usage clarification

We appreciate the reviewer's concern regarding our statement about not using hard negatives during training. Upon reflection, we believe this statement may have been misleading as our proposed method does utilize a form (although not finegrained) of negatives derived from the scene graph structure.

Therefore, we'd like to contrast the negatives used in OC-CLIP with the hard-negatives leveraged by other approaches. Our method employs "negative" scene graphs, as defined in Eq. 5, which are derived from the scene graph at a coarse level (manipulating relationship directions only, needed to learn a non symmetric relation score). However, other hard-negative methods operate at a finer-grained level, generating new negatives by altering specific attributes or parts of speech tags in the ground-truth sentence (including both nodes and edges content as well as relationship directions). Our approach does not employ such fine-grained modifications.

We in fact refer to other methods as employing “finegrained hard negatives” as specified in lines 6, 204, 214 and 373.

We believe this distinction in granularity is essential to consider when comparing our approach to other hard-negative-based methods, especially in our controlled experiments (Section 4.1) focused on attribute-binding where fine-grained hard negatives consist of swapping the attributes of two objects in the scene. In those controlled experiments, no hard negatives are used for OC-CLIP (the controlled scene graphs do not have any edges).

In order to avoid any confusion, we will update our manuscript with a dedicated paragraph clearly stating the difference between the graph-based relationship negatives (as defined in Eq 5) used in OC-CLIP and the fine-grained negatives used by other approaches. We will clarify the claim accordingly.

Q3 - In-d vs architectural improvements

We would like to emphasize that the OpenCLIP-ft baseline is finetuned on the same data mixture as ours ( COCO, VG, GQA). Therefore, we believe this comparison isolates the effectiveness of the proposed modules vs the use of data from a similar distribution to ARO/SugarCrepe. When compared to this baseline, OC-CLIP shows improvements of +15.1% in the hard swap-attribute split and +23.2% on ARO-A..

Moreover the controlled experiments in section 4.1 are specifically designed to further isolate our model contribution effectiveness by comparing both OC-CLIP and CLIP trained on a 3D simulated domain with fixed vocabulary size.

The results of section 4.3 further confirms the source of improvements coming from our architectural biases. In this setting we train both CLIP and OC-CLIP from scratch on cc3m and cc12m and show that not only has better sample-efficiency but outperforms CLIP by a large margin in zero-shot Imagenet classification and compositional understanding (eg, SugarCrepe).

We will update our manuscript in the baselines description paragraph (l273) accordingly to avoid any confusion and clearly state that we finetune the Openclip-FT baseline on the same data as OC-CLIP.

Regarding the other baselines, we would like to emphasize that all of them except DAC are finetuned using data in the COCO/VG domain. The exception DAC recaptions 3m images coming from CC3M dataset along with dense and finegrained hard negatives.

We hope this clarifies our approach and addresses the reviewer's concerns. We are happy to answer any additional questions the reviewer might have.

Thank you for your new results and discussions. My concerns have been fully resolved. I tend to raise my score to 4 (Boarderline accept) before discussion with other reviewers.

We thank the reviewer for supporting our work! We are happy to answer any additional questions the reviewer might have during the discussion period.

We thank the reviewer for their feedback and appreciate the opportunity to clarify the limitations mentioned. We understand the importance of accurately capturing directional spatial relationships in our proposed algorithm. We would like to clarify how our structured similarity score is in fact designed to handle such directional nuances.

Explanation of Directional Handling

In our model, the functions ( $f_s$ ) and ( $f_o$ ) are designed on purpose with different learnt parameters ( $\\theta_s$, and $\\theta_o$) to capture the directionality of relationships and break the symmetry of the addition. This means that the embeddings for the subject and object are processed differently, allowing the model to distinguish between "A left of B" and "B left of A."

Simplified Example

To illustrate this, consider a simplified example where the slots S are 2d embeddings containing x,y coords of a visual object.

In that case $f_s$ and $f_o$ can easily model a simple ‘left of’ relationships by extracting the x coord with different signs (+ for $\\theta_s$ and - for $\\theta_s$) such that:

f_s(A, left) = f_{\\theta_s}(A, left) \= x_A

f_s(B, left) = f_{\\theta_s}(B, left) \= x_B

$f_o(A, left) = f_{\\theta_o}(A, left) = -x_A$

$f_o(b, left) = f_{\\theta_o}(B, left) = -x_B$

Score ( ‘A left of B’ ) $= f_s(A, left) +f_o(B, left) = x_A - x_B$

Score(‘B left of A’) $= f_s(B, left) \+f_o(A, left) = x_B - x_A =$ - Score ( ‘A left of B’ )

Hence capturing the directionality of the spatial relationship ‘left of’.

This approach is easily generalizable to higher dimensions. In a multi-dimensional space, the learnt functions ( $f_s$ ) and ( $f_o$ ) can be extended to operate on vector embeddings, allowing the model to capture complex spatial relationships by considering all relevant dimensions. The distinct parameterization ensures that directional relationships are accurately modeled regardless of dimensionality.

Learning Directionality with local loss ( $\\mathcal{L}_{\\text{rel}}$ )

To further enhance the model's ability to learn directional relationships, we employ a loss function ( $\\mathcal{L}_{\\text{rel}}$ ) (as described in Eq. 3.2 and Line 197). This involves swapping the subject and object indices to create a local negative example, effectively enabling our module to learn a non-symmetric relation score. We have ablated this local loss in Appendix 3.2 to demonstrate its impact.

Empirical Validation

Our results on the GQA-spatial and VG-spatial from the What’s up benchmark (Table 2), which are designed to exclusively test for spatial relationship understanding (eg. ‘A chair to the left of the table’), further supports the model's capability. We achieve accuracy improvements from random chance to over 90% when compared to CLIP, indicating the model's effectiveness in learning non-symmetric relation scores.

Conclusion

By employing distinct parameter sets for $f_{\\theta_s}$ and $f_{\\theta_o}$, and leveraging ( $\\mathcal{L}_{\\text{rel}}$ ), our model is capable of learning and distinguishing directional spatial relationships. This design choice ensures that the model does not treat "A left of B" and "B left of A" as equivalent, thus addressing the concern raised. We will emphasize this asymmetrical parametrization in the main method description to avoid any confusion.

We hope this explanation clarifies how our score module can handle directional spatial relationships. We are open to answer any additional questions the reviewer might have.

We thank the reviewer for their answer and for supporting our work! We are happy to answer any additional question the reviewer might have during the remaining of the discussion period.

We thank the reviewer for their insightful feedback and supporting our work! We take the opportunity to answer the listed questions below :

Q1 - Additional strategies to further optimize the computational cost

Since the bottleneck of our module for the binding module is the number of KQ comparison in the binding module (where each query node from the scene needs the be compared to each key candidate from the visual backbone), any method that reduces the number of candidate keys could drastically improve the computational efficiency during training. In that spirit combining our binding module with token merging methods such as $1$ could constitute a good avenue for further computational cost improvement.

We would also like to emphasize the fact that this 2.2x overhead is present at training time only (because each element of the batch needs to perform the binding KQ attention with all the other elements of the batch to compute the CLIP loss), At inference time the overhead is minor.

Moreover we would like to emphasize that while our model introduces a computational overhead, our training from scratch results in section 4.3 shows better sample efficiency on downstream zero-shot performance when compared to CLIP. For that reason we believe the computational overhead introduced by the binding module to be a reasonable trade off for better general performance and compositional understanding.

$1$ Token Merging: Your ViT but Faster

Q2 - Discussion on Robustness of the Parser

The quality of the parser definitely constitutes a bottleneck in terms of performance. We also comment on the quantitative impact of the parser quality in Figure 6 where the parser ability to extract good relationship structure greatly impacts performance on the relational tasks (12 to 15% performance drop when using a Spacy-based parser vs an llm-based parser).

We anticipate that the following mitigations strategies could greatly improve the robustness the parsing :

• Finetuning an llm-based parser on noisy captions to extract good scene graphs would be straightforward (although costly),
• Including more noisy and hard parsing examples given in the context prompt of the llm-based parser,
• Allowing the llm-based parser to filter out bad data points by prompting it to output an empty graph (thus ignored) when unsure about the scene graph parsing .

We believe those directions constitute great avenues for future extensive research on the parsing method itself and are orthogonal to our approach.

Q3a - Number of slots analysis

We would like to clarify that our approach in fact handles a dynamic number of slots as opposed to fixed (slot attention like) methods. The extraction of the slots in our binding module are conditioned on the input scene graph, such that the number of visual slots will be equal to the number of query nodes in the input graph (corresponding to the number of parsed textual objects) . We believe this design is key to the inherently ill-defined notion of an “object” in a visual scene that can be understood at different levels of granularities (as emphasized by the reviewer). In our case the visual patches are grouped conditioned on the objects (=query nodes) mentioned in the input graph. The number of resulting slots will be equal to the number of nodes in the scene graph.

To give a more concrete example, let us consider an image containing several colored chairs. For that same image different scene graphs will lead to different slots decomposition depending on the granularity of caption:

• (1) “a group of chairs” will be parsed as a single node graph “a group of chairs” that will group together the visual information coming from all the chairs into one slot

• (2)“a red chair and a blue chair” will be decomposed into a graph with 2 separate query nodes ( ‘red chair’ and ‘blue chair)’ and each one of the nodes will group the visual patches that correspond to a specific chair into a separate slot, resulting in 2 slots.

We believe this conditioning mechanism is key to the dynamic handling of plausible scene decomposition and hope we have clarified this point.

Q3b - Default Query Token role

Since the perceptual grouping is done using inverted cross attention ( eg. by applying the softmax along the query dimension), this inductive bias pushes all the visual key candidates to be softly matched along at least one query (as explained by $2$ ). However, natural captions are far from being an extensive description of the visual content and can arbitrarily focus on a single part of the image only. Since we only want the relevant visual information ( as captured by the input scene graph) to be extracted from the candidate patches, we need to re-route the “un-necessary” visual information towards other tokens, that we coin default query tokens. Those tokens participate in the competitive grouping attention but are not taken into account when computing the score. Their role is thus to absorb any remaining visual information that is not mentioned in the input scene graph.

$2$ Object-centric learning with Slot Attention

Q4 - Extension to MLLMs general framework

While a direct extension to the OC-CLIP to a more general MLLM framework is not straightforward, a good avenue for future work could be to study the impact on the representation learnt by the vision backbone of OC-CLIP when trained from scratch (on larger 400m or 2b scale, like its CLIP counterpart) on a downstream Llava-style MLLM (using OC-CLIP backbone stitched to an LLM by an MLP connector). Indeed, without any specific patch tokens constraints, CLIP-like training does not force any locality on the visual patches representation, which might in turn lead to poor feature binding at the visual representation level of the MLLM. Techniques like masked image modeling, or local iBot losses (as used in Siglip2 and Dinov2) aim to improve the locality and density of the visual patches.Therefore, it could be an interesting future work to study how OC-CLIP local graph matching constraints might similarly impact the locality of the visual patches that can be used by a downstream MLLM. As this question requires full scale training from scratch, we leave that for future work.

We hope we have addressed the reviewer concerns and are happy to answer any additional questions the reviewer might have.

We would like to thank again the reviewer for their time and for supporting our work! We hope our rebuttal answer addressed the reviewer's main concern. We are happy to answer any additional questions the reviewer might have during the remaining of the discussion period.

We thank the reviewer for their feedback and appreciate the opportunity to clarify the limitations mentioned. Below are our responses to the specific points raised :

Q1-a :Compatibility with vLLM:

vLLM is a framework that optimizes inference and training of llm-based models. We do in fact use vLLM to optimize the scene graph parsing efficiency (Appendix A.4). However vLLM is generally not considered useful for training and inference of embedding models like CLIP. This is because the primary optimization vLLM implements (e.g. chunked prefill, automatic prefix caching, etc..) focuses on generation efficiency.

We would like however to clairify that our model is an embedding model and is in fact end-to-end. It can therefore benefits from any additional opitimizations applicable to CLIP-like models.

We also would like to also emphasize that the scene graph parsing step is not part of the model but a data preprocessing step done offline as specified in l136. The model itself is end-to-end and can benefit from further wrapping and optimizations the same way as CLIP. We will update the main Figure to clarify that aspect.

Q1-b: Computational Efficiency:

Our module introduces a trade-off in computational efficiency, which we discuss in detail in the "Computational Analysis" section of Appendix A.3 (l752). We quantitatively measure the computational overhead of our modules and demonstrate that for larger-scale ViT models (e.g., ViT-L), the binding module does not become a bottleneck, as it remains fixed when scaling the ViT backbone. To mitigate overheads when training our model from scratch, we propose:

• Using a smaller text encoder (fewer layers and reduced width), since the OC-CLIP text encoder only needs to encode information about single objects and relationships.
• Operating the binding module on a reduced embedding space (256 vs. 512 for the original CLIP).

Q2 -Drop in Performance on some Tasks:

The binding module's relatively low training vocabulary size (eg. COCO domain), due to being trained from scratch, is the primary reason it does not outperform (and is not expected to) all baselines on "replace" and "add" tasks of Table 1. We anticipate this performance gap would close if trained on the original OpenCLIP pretraining set (Laion400m), as indicated by results in Section 4.3 at the 15m scale.

However we want to highlight that on tasks that require finegrained object and attribute binding (spatial, and swap splits) our model outperform all the other baselines, even when they are trained on order of magnitude more data (eg. DAC finetuned on CC3M augmented with hard negatives).

Specifically, the "add obj" (91.1% vs. 93.4% for CLIP counterpart) and "replace obj" (94.6% vs. 95.4% for its CLIP counterpart) splits in SugarCrepe show slightly lower performance for our model. While both models achieve high accuracy, this difference stems from two factors:

1. Vocabulary Generalization: Our binding module and multimodal alignment scores are trained from scratch, limiting their generalization to the vocabulary they were exposed to during training.
2. Out-of-Distribution Negatives: The "replace" and "add" negatives in SugarCrepe are generated using an LLM (GPT-4). This process does not guarantee that the objects/attributes introduced are within the distribution of our training data (COCO/VG). These LLM-generated negatives may include objects/attributes outside the COCO domain but within OpenCLIP's broader pretraining domain, leading to a slight, albeit minimal, drop in our model's accuracy. Training on the original OpenCLIP pretraining set (Laion400m) is expected to mitigate this issue.

Q3 - Lack of Parser Impact Discussion:

We would like to clarify that we already extensively discuss the impact of scene graph parsing in Section A.4 of the Appendix (l777) as referred in line 136 and line 382 of the main text, where we:

• Compare different parsing methods (automatic, fine-tuned lightweight parser, and LLM-based parser) and describe their qualitative and quantitative performance impacts, along with the error types each can introduce.
• Comment on the limitations of LLM-based parsing, especially when ambiguous parsing requires visual grounding, providing interesting directions for future work.
• Discuss and reference the scene graph parsing cost in the preprocessing step.

L3 -Clarity of section 4.1 figures

Figure 2(b) and 2(c) aim at comparing CLIP and OC-CLIP sample efficiency in a controlled setting where we vary the training data size (y axis) and the amount of hard negatives (x-axis). It shows that without any hard negatives and only 30% of the possible animal pairs combinations OC-CLIP outperforms CLIP even when trained on 70% of the colored animals pairs combination and 70% of all the possible swap hard negatives. We will update the figure titles of part 4.1 to emphasize that the numbers correspond to performance accuracy and not correlation. We apologize for the confusion!

We hope this clarifies our approach and addresses the reviewer's concerns. We are happy to answer any additional questions the reviewer might have.