Natural Language Inference Improves Compositionality in Vision-Language Models

We thank Reviewer G1JW for all the feedback. We are glad you found our paper clear and easy to follow. We address the weaknesses and questions below.

[W1] Why contradictions help. Natural Language Inference (NLI) tasks inherently require models to leverage common-sense knowledge to determine semantic sub-sets and relations. Our experiments show that CECE provides a strong prior for caption expansion and exclusion by considering both entailments and contradictions. Specifically, we instruct the LLM to exclude negations; without this instruction, the LLM would output: “the banana is not in two or more pieces” instead of “the banana is in one piece” (example from Figure 2), or “no water is coming out of the spraying device” instead of “the spraying device is dry” (example from Figure 5).

NLI-based expansion produces semantically equivalent statements that contain different vocabulary and sentence structures; thus, the resulting subset of captions (both entailments and contradictions) aids the VLM in covering cases where the caption is too complex or ambiguous.

Contradictions contribute uniquely to this process by offering an additional mechanism for validation. If a given image-caption pair is correct, its contradiction represents an untrue statement of the image. For example, in Figure 2, the correct caption is “there is a split banana”. The generated contradiction, “the banana is in one piece”, is a description inconsistent with the image. When the VLM correctly identifies the contradiction as untrue, this adds further evidence supporting the correctness of the original image-caption pair. Thus, contradictions enhance compositional reasoning by providing a complementary layer of validation that enriches the semantic understanding of the image-text pairs. Contradictions help the model disambiguate nuanced relationships, such as in the example of “split banana” vs. “banana split.” This interplay between entailments and contradictions ensures robust coverage of the semantic space, improving the model's performance on complex tasks.

[W2] Include contradictions alone in Table 6. We include the results when using only the contradictions below and in the revised version. As expected, contradictions alone do not yield state-of-the-art results; however, the result is competitive against prior SDS approaches.

[W3] Impact of the number of generated captions and semantic diversity. We take the number of entailments and contradictions for Winoground, and compute some statistics:

Broadly, the LLM generally follows the instruction of 2 entailments and 2 contradictions; however, sometimes it outputs up to 10. In our experiments, we take all outputs from the LLM to compute the final values. We found that, in some cases where the caption is correctly structured and contains more than one action, the LLM tends to provide shorter and more nuanced entailments and contradictions, resulting in more captions.

For semantic diversity, we show the Jaccard Similarity to assess the lexical diversity of the generated captions (L428-L431). CECE captions and the original captions show a low similarity score; this supports the diversity statement. We also included human evaluations on captions expansions (L444-L457). Using a Likert scale, the entailment captions received an average score of 4.7, while the contradiction captions received an average score of 1.7. This indicates that the entailment and contradictions are both accurately generated.

Each set of entailments and contradictions provides evidence about the relationship between the image and caption, with some being more discriminative than others in specific cases. This does not necessarily imply a bias but reflects the complementary nature of these expansions in covering diverse semantic dimensions. Entailments often affirm subtle consistencies, while contradictions highlight mismatches that are crucial for distinguishing between closely related visual-text pairs.

We hope our response answers your questions. Please let us know if you would like to ask for any further clarifications.

[W2] Why the Text Score is better in Both_L, Symbolic_V baselines. Thank you for raising this question! Upon further investigation, we identified an error in the reported results. The baseline values for LLaVA-1.5 were slightly overestimated, while the CECE values with LLaVA-1.6 included only the entailment scores, leading to a slight underestimation. We updated Table 4 in the manuscript with the correct scores and attach the table below for reference:

The text score measures if the model can select the correct caption given an image. In our manual evaluation, we observed that the LLM outputs tend to be noisy when the original captions contain multiple objects, attributes, and actions. These captions are generally tagged as Both_L (the subset of samples where both objects and relations are swapped, or a single swap that changes parts of speech).

We thank Reviewer ZcB9 for all the feedback. We are glad you found our method simple yet effective, as it does not require fine-tuning since it is performed in inference time. We also hope our work could inspire further inference-time strategies, building on our findings. We address the weaknesses and questions below.

[W1] Ablations of the LLM. We include additional experiments using Llama 3.1-8B. This was also pointed out by reviewer Xyex, and we thank both for bringing it to our attention. We include these experiments in the revised version of the manuscript. We also show a fraction of the table below for reference.

Our results show that even with a significantly smaller model, our CECE approach consistently outperforms the end-to-end approaches, including the GPT4-Vision model (which leverages the GPT4V-Eval method proposed in Zhang et al., 2023).

As we mentioned in the Reviewer’s Xyex response, it is also worth noting that Llama3.1 8B outputs hurt other Sentence Decomposition via Semantics (SDS) methods. Our experiments show that a smaller LLM tends to output unrelated or truncated outputs with these approaches. We hypothesize that this behavior is due to the complexity of the prompt and required task decomposition in these methods (e.g., dividing the sentence as a subset of tasks that require separate processing, along with the larger textual context due to the multi-shot example demonstrations).

[W3] Ensembling VQAScore Llava1.5+Llava1.6: Table 4. We show results when ensembling VQAScores with LLaVA-1.5 and LLaVA-1.6. In this case, only the Symbolic_V tag subset surpasses our CECE without LLaVA-1.5 and LLaVA-1.6 ensembling. Symbolic_V tag includes samples with non-natural images such as drawings or illustrations. While the LLM outputs are aligned with common-sense knowledge, drawings are not necessarily grounded in reality. For the samples tagged as Pragmatics_V, the images need to be interpreted non-literally due to idiomatic uses of language in a caption.

In addition, we add qualitative samples in the appendix (Figures 10 and 11). These observations align with the semantic drift issue we mitigate by balancing CECE with the whole caption score. We include additional examples in the appendix (please also take a look at Figures 8 and 9, which are closely related).

[Q1] Why CECE performs better Image Score. The text score evaluates the model's ability to distinguish between captions for a given image, and the image score evaluates its ability to distinguish between images for a given caption. Prior work has pointed out that some visual features may be subtle or difficult for the models to capture at their input resolutions, making the image score task harder [2]. This is particularly true for almost all end-to-end models (e.g., in [1], all models shown in Table 3 perform worse under the image score metric).

A key advantage of the CECE and SDS approaches is their interpretability. By expanding or decomposing captions into smaller semantic units, these methods provide models with more fine-grained information to match against images. With CECE, noise or errors in the expansion process for one caption apply equally to both images, maintaining consistency in the scoring process for the image score case. In contrast, the text score involves comparing two captions with separate sets of entailments and contradictions against one image, which introduces variance and increases the likelihood of errors or inconsistencies affecting the result. This asymmetry makes the image score less sensitive to erroneous expansions, justifying CECE’s stronger performance in this metric.

[Q2] Including CECE with InstructBLIP in Tables 2 and 3: We run experiments with CECE using InstrucBLIP as the VLM backbone. Our results show that CECE outperforms the base model, and particularly in Table 3, CECE outperforms both InstructBLIP and LLaVA-1.5. We include both updated tables in the revised version. We also add Table 3 below for reference.

[Q3] Number of entailments and contradictions. We take the number of entailments and contradictions for Winoground, and compute some statistics:

Broadly, the LLM follows the instruction of 2 entailments and 2 contradictions; however, sometimes it outputs up to 10. In our experiments, we take all outputs from the LLM to compute the final values. We found that, in some cases where the caption is correctly structured and contains more than one action, the LLM tends to provide shorter and more nuanced entailments and contradictions, resulting in more captions.

We hope our response answers your questions. Please let us know if you would like to ask for any further clarifications.

[1] Thrush, Tristan, et al. "Winoground: Probing vision and language models for visio-linguistic compositionality." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Diwan, Anuj et al. “Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality.” Conference on Empirical Methods in Natural Language Processing (2022).

We thank Reviewer 1DuW for all the feedback. We are glad you found our core idea intuitive, pointing out CECE as a novel solution that enriches semantic understanding and our experimental results competitive, with clear improvements over existing methods. We address the weaknesses and questions below.

[W1] Semantic drift with CECE. We add detailed explanations for cases where CECE captions drift away from the original semantic meaning in the appendix. Particularly, in Figures 8 and 9 (in the Appendix), we add examples where the LLM outputs imprecise entailments, or the caption is not decomposed appropriately. We also add detailed descriptions in Figures 5-6 to explain cases where CECE fails in the scoring process.

[Q1] Minor experimental detail. System prompt: we do not use a particular instruction for the system (i.e., an empty string). Chat template: we do not modify the chat template (i.e., tokenizer.apply_chat_template has the default functionality from the huggingface model card).

We hope our response answers your questions. Please let us know if you would like to ask for any further clarifications.

[W2] More datasets. We show results using CECE variants with SugarCrepe [1]. It is worth noting that prior work that reports strong results in this benchmark tends to perform poorly on Winoground and EqBen. CF2C uses GPT-4V, where the input is an image and the pair of captions, and the model outputs which caption is more likely to describe the image. We include this number in the table below, even though we evaluate using only one image and one caption at a time.

These results provide strong evidence that supports the effectiveness of our work. We expect future research directions that leverage caption expansion techniques (i.e., entailments and contradictions) to develop inference-time methods and training strategies that improve compositional reasoning in vision-language models. We hope our response answers your questions. Please let us know if you would like to ask for any further clarifications.

[1] SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. NeurIPS: Datasets and Benchmarks Track, 2023.

[2] https://github.com/RAIVNLab/sugar-crepe/tree/main/gpt-4v-results

We thank Reviewer Xyex for all the provided feedback. We are glad you find our paper well written, with a clear motivation and method. We are also glad you acknowledge CECE’s strong performance, especially since it’s architecture-agnostic and does not require any fine-tuning. We address the weaknesses and questions below.

[W1] Inference cost. We include results using a smaller open-sourced model, i.e., Llama 3.1-8B in the revised version. For a fair comparison, we also run DSG using this same LLM. We show a fraction of the table below for reference.

Our results show that even with a significantly smaller model, our CECE approach consistently outperforms the end-to-end approaches, including the GPT4-Vision model (which leverages the GPT4V-Eval method proposed in Zhang et al., 2023). It is also worth noting that Llama3.1 8B outputs hurt other Sentence Decomposition via Semantics (SDS) methods.

Our experiments show that a smaller LLM tends to output unrelated or truncated outputs with these approaches. We hypothesize that this behavior is due to the complexity of the prompt and required task decomposition in these methods (e.g., dividing the sentence as a subset of tasks that require separate processing, along with the larger textual context due to the multi-shot example demonstrations).