Compositional Entailment Learning for Hyperbolic Vision-Language Models

We thank the reviewer for their praise of the idea and their positive feedback regarding the presentation and the experiments.

Improved qualitative results. We thank the reviewer for their suggestion and have now revised Appendix I by comparing HyCoCLIP traversals with both MERU and CLIP with plans to enrich the section with further illustrations. For CLIP, it should be noted that leveraging the Euclidean latent space, does not allow for a straightforward definition of the [ROOT] node. Therefore, for CLIP, we define this node as the centroid of the embeddings of the GRIT training samples, following the definition provided by MERU's authors.

Comparison to hierarchical understanding of pretrained foundation VLMs. We thank the reviewer for the reference [1], which we will include in Sec. 5. We zero-shot evaluate our method on the introduced Hierarcaps test set of the reference paper on their proposed metrics. In the following table, we report the results along with baseline numbers from the reference (marked **). We note that our models trained from scratch on a significantly lower volume of data perform comparably to the OpenCLIP and ALIGN baselines. Additionally, HyCoCLIP outperforms CLIP and MERU indicating better hierarchical understanding.

We have added further details on this experiment in our revised manuscript Appendix H.

Does the use of objects induce a bias towards nouns and concrete concepts? Our scene understanding experiment, described in detail in Sec. 3.2 and Appendix E, includes the VL Checklist Object and VG Attribute benchmarks, which feature words and expressions distinct from standard noun concepts. We performed this zero-shot evaluation test to assess our model's ability to comprehend attributes like color (e.g., white, brown, orange), verbs (e.g., crouched, sitting, burnt), materials (e.g., wood, metal, mesh), adjectives (e.g., empty, young), as well as object size and spatial positioning within the image. The outcomes show that our proposed method improves on the MERU's technique [2] also in this challenging task.

Filtering nouns. The abstract nouns are part of a predefined list that is excluded when extracting noun chunks using spaCy's English language model. The extracted nouns are largely open-vocabulary, as spaCy’s part-of-speech vocabulary is quite extensive. For grounding, the bounding boxes predicted by the GLIP model are filtered when their confidence score (measured by the dot product with the corresponding noun chunk) is below 0.65, ensuring that only high-quality boxes are considered. We have now included Fig. 7 in Appendix A which provides qualitative examples.

[1] Alper and Averbuch-Elor. "Emergent Visual-Semantic Hierarchies in Image-Text Representations", ECCV 2024.

We are grateful to the reviewer for acknowledging the contributions made by our method towards enhancing hierarchical understanding in VLMs. While existing works mainly evaluate the hierarchical nature of representation space, we go further toward enforcing it through our novel loss functions while demonstrating improved performance on several downstream tasks. In the following, we address the questions raised by the reviewer.

Highlighting best performances. We thank the reviewer for this suggestion. Following both their and reviewer SpcR's recommendation, we revised Table 1 to include the results of HycCoCLIP on the RedCaps dataset and underline the best results among competitors sharing the same backbone.

Gradient accumulation for simulating larger batch size. We agree with the reviewer that gradient accumulation could effectively simulate larger batch sizes. However, in the final paragraph of Sec. 3.3, we address the impact of the batch size on the performance. Our findings indicate no noticeable benefit in using very large values for the batch size (cf. Table 6 in the main manuscript). This saturation with batch size in contrastive loss has also been discussed by [1] with the additional entailment loss also playing a factor. Thus, we did not focus on scaling to greater batch sizes.

[1]: Zhai et al., "Sigmoid loss for language image pre-training", ICCV 2023.

We thank the reviewer for their positive feedback regarding the organization of the paper, the method, and the experiments.

Comparison with recently proposed VLMs. We compare our method with the recent SOTA hyperbolic VLM, i.e. MERU in addition to Euclidean CLIP. While there are indeed other recent Euclidean VLMs that excel at various tasks, these models are typically trained at a much larger scale, often requiring vast amounts of data and computational resources. As a result, a direct comparison between these models and our approach, which operates with a limited training setup and a relatively limited amount of data, would be impossible.

Details and sensitivity of hyperbolic space parameters. The key geometric parameters in hyperbolic space are the curvature and dimensionality. We allow the curvature ($\kappa$) to be a learnable parameter (initialized at $\kappa=1$), consistent with prior work [1], and clamped the parameter at 0.1 value. In response to the reviewer’s suggestion, we have included Table 7 in Appendix B to demonstrate HyCoCLIP's sensitivity to this parameter. We find that enabling $\kappa$ to be learned while training empirically yields the best results.

As for the dimensionality, we use the Lorentz model to represent the hyperbolic space $\mathbb{L}^n$ with $n=512$. In addition to these parameters, we experimented to determine the optimal aperture for entailment cones to maximize performance. The results of this analysis are detailed in Appendix B. While some of these details were already provided in Appendix B, we have expanded and clarified this section to include these further insights.

[1]: Desai et al., "Hyperbolic Image-Text Representations", ICML 2023.

We appreciate the reviewer’s positive feedback on our method and the results achieved. Below, we address the questions raised.

Training CLIP on additional image-text boxes shows no improvement. We agree with the reviewer that image-text boxes do not offer additional information when naively added as extra samples since they are directly extracted from the full image-text pairs. This is precisely where our approach excels, by leveraging the hierarchical alignment of additional boxes within the hyperbolic space. Additionally, we conducted the suggested experiment of training on a half-split of the training set on the ViT-S backbone. We present the results in the following tables and also include results of HyCoCLIP and CLIP on the entire GRIT as reported in Table 1 of our manuscript.

Evaluation of the models on zero-shot image classification,

$~$

Evaluation of the models on Flickr image/text retrieval,

$~$

We find similar trends to those presented originally in Table 1 of the manuscript. HyCoCLIP benefits the most from such localized boxes. Additionally, from the first table, we also find that HyCoCLIP scales best with the additional (nearly double) data.

(continued)