MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

Thanks for your constructive suggestions. Your endorsement of our dataset and experiments gives us significant encouragement.

Q1: Thank you for your suggestion! We will analyze the patterns of super hard questions for shed light on how to design difficult questions for the competent VLMs in the revised paper.

Q2: We conducted experiments to compare different formats of input images, including combining multiple images into a single 'super image' and feeding the models with a list of images. The results, shown in the table, demonstrate that these formats have varying impacts on different models' performance. Specifically, providing a list of images improved the performance of Qwen2-VL-72B, but led to a decrease in performance for InternVL2-40B.

Q3: We follow the method proposed in FineMatch [1], which adopts the criteria of replacing attribute, relation, and object phrases while maintaining the Part of Speech (POS) tags unchanged. This approach keeps the mismatched captions as similar as possible at the character level to the original correct captions.

[1] "FineMatch: Aspect-Based Fine-Grained Image and Text Mismatch Detection and Correction." ECCV, 2024.

Thank you for your time, thorough comments, and valuable suggestions. We are pleased that you acknowledged our dataset as an improvement upon existing compositional datasets and recognized our experiments for their in-depth model comparison and component analysis.

Q1: Thank you for your suggestion. We have added the experiments results under the ICL setting. From the table we can observe that while introducing context examples into the prompt, models’ performance decreased in different degrees.

Q2: We have computed the proportion of multi-hop QA pairs in our benchmark, which is 2,459 out of 4,342, amounting to 56.63%. We compared the models' performance on multi-hop versus non-multi-hop questions, and the results are shown in the table. From these results, we observe that models struggle with multi-hop reasoning tasks. We also provide examples of the multi-hop questions in the revised paper; please see Figure 16.

Q3: We have addressed the questions in points 1 and 2, the results are shown in the corresponding tables.

We sincerely appreciate your invaluable feedback and the opportunity to address your queries regarding our benchmark.

Q1: As described in lines 132–133 of our paper, our proposed benchmark is a new human-annotated dataset for evaluating VLMs’ compositionally. All the question-answer pairs are human-annotated (see lines 270–272). We use various datasets with the potential to construct VL compositional QA pairs as our seed data (lines 212–213). Importantly, we only use the images from these seed datasets and their initial annotations as prompts to construct new QA pairs. Therefore, we have introduced new data in this work.

Q2: The goal of our work is to provide a comprehensive diagnostic analysis of current VLMs regarding their capability for VL compositional perception and reasoning, serving as a complement to earlier comprehensive benchmarks such as MMBench, MMStar, and MME. Therefore, the significance of our work should not be overlooked. We believe that our contribution is non-trivial and will benefit the community involved in VLM design and training.

Q3: We have updated the human subjects in the revised paper, please see Table 2 and Reviewer 33Vo Q3.

Q4: We provide the quantitative results of our error analysis in the appendix; please see Figure 13 and Section A.4.

Q5: As described in lines 132–133 of our paper, our dataset is fully human-annotated. All the QA pairs are human-annotated (lines 270-272). Therefore, the problem you mentioned is not apply for our paper.

Q6: Please refer to Q1 and Q2, we believe our contribution is non trivial and will benefit the community of VLM design and training.

In conclusion, we introduce a new dataset and all the QA pairs in our data are created by human annotators, as clarified in multiple sections of our paper. We have carefully verified the properties that distinguish our dataset from previous work. We believe that our contribution is significant and will benefit the VLM community.

Thank you for your feedback. We are pleased to address your remaining concerns.

Q1. There are no restrictions against using existing data as seed data for dataset construction. Many new datasets, such as RefCoCo, Visual Genome, LLaVA Bench, and MMVP, leverage existing data sources in this way and are well-accepted as significant tools for VLM evaluation and diagnosis. Additionally, you recognize our dataset as "a new benchmark" in the Strengths section. We believe this point is unrelated to the core contributions of our paper and seems to focus excessively on minor or irrelevant details.

Q2. We have included the main differences between our work and existing benchmarks. Table 1 and Section 1 clearly outline the novelty and distinct aspects of MMComposition compared to existing works.  Reviewer 1 has recognized the novelty and distinct aspects of our work, please also refer to R1 Q1, where we have successfully addressed this concern for them.

Q3. The resolution gap is a significant factor that may influence a model's capabilities. It is reasonable that models limited to processing low-resolution images exhibit poorer compositionality. Therefore, your statement that "it might be biased towards the weak models" is consistent with intuition. We have analyzed the impact of encoding resolution on models' performance in Section A.2 for the top three models (k=3). Thus, your concern regarding ‘the same for strongest-k models? ideally maybe k=1 or k=2’ has been addressed in our paper. In addition, our paper includes an in-depth analysis of the strongest models in terms of different expected human capabilities—these models include GPT-4o, Qwen2-VL, and InternVL. Furthermore, our work provides an in-depth analysis of the relationship between the design of VLMs and their compositionality. As the first study to comprehensively examine complex compositional perception and reasoning, it distinguishes itself from previous works such as ARO, Crepe, and SugarCrepe. We want to emphasize that this contribution should not be overlooked.

Q4. Our paper clearly "predicts the community needs to focus on". In Section 1 (lines 106–126), we have explicitly addressed this by highlighting: "(1) Visual Encoder Design: While a mixture-of-encoder architecture can enhance compositionality, adding more encoders does not necessarily improve performance." ... "we find that for relatively simple QA tasks, only a small portion of its language capabilities are utilized ... Once the language decoder size reaches a certain threshold (e.g., 34B, 70B), the visual encoder has a more significant impact on the model’s compositionality. "). We believe these points provide clear guidance on where the community's efforts could be most effectively directed.

In contrast, the LlaVA team's analysis primarily identifies the phenomenon but does not delve into the underlying reasons. Furthermore, prior works lack a comprehensive analysis of why models fail at fine-grained visual compositional perception and reasoning. In our paper, we thoroughly examine these underlying reasons in Sections 1, 5, and A.2. Therefore, we believe this concern has already been addressed in our work.

In conclusion, we believe that conclusions should be based on evidence, not merely on ” reasoning.” The concerns you mentioned have been thoroughly addressed in our work, and some of the "issues" you raised have, in fact, been recognized as strengths by other reviewers. Therefore, we respectfully request a reconsideration of the evaluation of our paper.

Dear Reviewer wuw2,

We believe we have addressed your concerns. If our careful and respectful responses continue to be ignored, we will report this to the ACs and/or PC.

Q6: Good point. We provide experimental results comparing models before and after tuning with extra instruction tuning data. Please refer to the Table below. The results indicate that models benefit from fine-tuning with extra related datasets for their compositional perception capability, and we have updated this finding to the revised version. In addition, Table 5 in our paper also compares the performance of the models with and without more data fine-tuning.

In conclusion, our work aims to provide a comprehensive diagnostic analysis of current VLMs regarding their capability for VL compositional perception and reasoning, serving as a complement to earlier comprehensive benchmarks such as MMBench, MMStar, and MME.

Thank you for the time, thorough comments, and nice suggestions. We hope our response can adequately address your concerns.

Q1: We have included a comparison of our benchmark with other general benchmarks, as shown in the table. Our benchmark stands out from well-known benchmarks due to its multi-hop QA pairs, its specific capability assessment -- compositionality -- and its challenging nature. This comparison has also been included in the revised paper (see Table 8).

Q2: To verify the challenging nature of our dataset and demonstrate the indispensable role of images, we conducted experiments comparing the models' performance between the standard setting and an image-blind setting. As shown in the table below, without image input, the models' performance decreases significantly, indicating that they must rely on image compositional information to obtain the correct answers. This result has also been included in the revised paper (see Table 10). Image-Blind Setting:

Q3: The data was initially annotated by student workers and then verified by another group of workers; finally, the dataset was refined and finalized by the authors. We followed the method proposed in FineMatch [1], which involves replacing attribute, relation, and object phrases while maintaining the Part of Speech (POS) tags unchanged. This approach ensures that mismatched captions remain as similar as possible at the character level to the original correct captions. We have updated the human performance metrics for each task in Table 2 of the revised paper. [1] "FineMatch: Aspect-Based Fine-Grained Image and Text Mismatch Detection and Correction." ECCV, 2024.

Q4: We have clarified in our paper (lines 122–125) that the visual encoder plays a more significant role in the compositionality of VLMs. Models with enhanced capabilities for perceiving fine-grained compositional image information can provide more detailed inputs to language models. Moreover, we have added a comparison of different visual connectors, including Q-Formers and MLP models, with the results shown in the Table below. From these results, we conclude that the Q-Former architecture cannot provide detailed visual references to language models for fine-grained compositional image understanding. This result has also been included in the revised paper (see Table 9).

Q5: Thank you for highlighting this issue. We have addressed it in the revised version.

From the authors' response, the work appears to be improved, particularly in its comparison with previous QA benchmarks, its analysis of the benchmark's challenges (e.g., the image-blind setting), and the additional exploration of connector design and fine-tuning approach.

Regarding the image-blind setting, I am curious whether pure language-based LLMs, such as LLaMA-3.1, GPT-3.5, or others, can perform well on the proposed benchmark in the absence of the provided image.

Thank you for your endorsement and valuable suggestions! In response, we have included results for pure language models, specifically GPT-3.5, Qwen2.5-72B-Instruct, and LLaMA-3.1-70B. Please refer to the tables for detailed comparisons and insights. From the tables, we observe that the performance of the pure language models is close to random guessing (30.15%), which underscores the indispensable role of visual information in our dataset.

My initial concerns have been addressed. I would be happy to adjust my score recommendation to 6 if the evaluation code release for MMComposition includes procedures for testing the models and settings reported in the paper, including image-blind settings with both VLMs and LLMs.

Thank you for your thoughtful feedback and for considering adjusting the score recommendation. We are pleased that our response has successfully addressed your initial concerns, and your suggestions have significantly helped us improve the quality of our work!

We want to confirm that the example evaluation code for MMComposition is now complete and has been made available through the supplementary materials. This code provides comprehensive procedures for testing the models and settings discussed in our work, including the image-blind settings for both VLMs and LLMs. We believe this fully addresses your concerns regarding reproducibility and transparency. Furthermore, we are actively working on refining and formatting all evaluation codes to support a more comprehensive and robust evaluation framework for MMcomposition. All the resources will be released soon.

Thank you again for your valuable time and thoughtful review!

Thank you for the comment! I have updated the score accordingly.