VisMin: Visual Minimal-Change Understanding

“which contradicts the authors' claim of improved text-image alignment in MLLMs”

We do not claim that we improve general image-text alignment for MLLMs. We claim that we improve CLIP’s general image-text alignment and substantiate that claim by showing improvements on the standard COCO image-text retrieval.

Also, we would like to note that improvement in fine-grained understanding tasks does not necessarily translate to an improvement in standard image-text understanding tasks as also observed in another recent work [1]. This is due to differences in data distributions of these tasks. For example, standard tasks have more diversity in object classes whereas fine-grained tasks have more diversity in relations, attributes etc. So the best we can hope for is that the performance on standard tasks does not drop much. And we show that this is the case for our method.

[1] Bugliarello et al. Weakly-supervised learning of visual relations in multimodal pretraining. EMNLP 2023.

Results on additional benchmarks

First, we would like to clarify that the benchmarks suggested by the reviewer require certain specific skills (in addition to general image-text understanding) that our minimal change data does not cover. For instance, TextVQA requires Optical Character Recognition (OCR), MathVista requires math understanding, MMBench requires OCR, structured data understanding, and celebrity recognition. The base model (Idefics2) has been trained on datasets containing such specific skills (e.g., Idefics2’s pretraining data consists of numerous math understanding datasets [2, 3, 4]). However, our minimal change data focuses on general scene understanding. Thus, finetuning Idefics2 on our minimal change data is expected to cause catastrophic forgetting [5] of such specific skills leading to a degradation in model performance on such benchmarks. To mitigate this, one needs to continue training the base model on datasets containing such specific skills along with finetuning it on our dataset, however experimenting with that is out of the scope of this paper.

As per the reviewer’s request we evaluated the models on the suggested benchmarks. In addition, we also evaluated the models on the VQA v2 benchmark which tests for general scene understanding. See Table 2 of the rebuttal PDF for the results. All results are reported on the validation split (test splits are not publicly available or require submitting to evaluation servers), except for MathVista, whose test split is publicly available. As expected, we see that finetuning on our data leads to a drop in the performance on all benchmarks except VQAv2 where we see improvements in the performance of the base Idefics2 model, thus demonstrating that our data is indeed helpful for improving general multimodal scene understanding.

[2] Kazemi et al. GeomVerse. arXiv 2023.

[3] Lindström et al. CLEVR-Math. arXiv 2022.

[4] Lu et al. Inter-GPS. ACL 2021.

[5] Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 2017.

“the authors could reduce the model size, such as using Qwen-1.8B, to validate the method's effectiveness”

We’ve actively researched smaller models for our MLLM experiments. To the best of our knowledge, Idefics2 is the best-performing open-source model capable of processing multiple images. We’d like to clarify that the Qwen-1.8B suggested by the reviewer is a language model; not an MLLM. Indeed, there’s an MLLM that’s built on top of Qwen LLM called Qwen-VL [6]. However, Qwen-VL is a 7B parameter model and cannot process multiple images. We’ve again reviewed recent papers (during this rebuttal period) and we’ve not found any smaller model capable of processing multiple images. For instance, although MiniCPM [7] has models with 3B params; however none of them can process multiple images.

[6] Bai et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv 2023.

[7] Hu et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv 2024.

Zero-shot image classification results

We’ve incorporated the zero-shot image classification results in the last column of table 2 of the rebuttal PDF. The results show that after fine-tuning with a hard negative dataset, both NegCLIP and VisMin-CLIP under-perform the base CLIP model on zero-shot image classification task. However, VisMin-CLIP shows a smaller performance drop compared to NegCLIP, thus demonstrating that our method learns more robust representations.

Lastly, as mentioned in our previous response, the drop in the performance on standard tasks is expected when finetuned on fine-grained tasks. We agree that an ideal model should work well for both types of tasks, however, given the large improvements of our method for fine-grained tasks and also for standard COCO image-text retrieval and VQA v2 tasks, we believe that the contribution of our method is still quite positive.

“There are also some typo”

We apologize for these errors and thank the reviewer for pointing them out. We will fix them in the camera-ready version.

We thank the reviewer for getting back on the rebuttal. We would appreciate it if the reviewer could elaborate a bit more on what they think about the additional results and the discussions we provided in the rebuttal.

Best, Authors

Some notations are confusing

We apologize for this. The captions C_0 and C_1 are minimal-change pairs as described in Fig 1 of the review version. The C in L247 is exactly the same as T in L294. We will clarify these and make the notations consistent in the camera-ready version.

Details about Instruct-IT dataset

In the VisMin dataset, a sample consists of a source image-caption pair (I0,C0) and its minimally edited version (I1, C1). To construct the VisMin Instruct-IT dataset, we create four instruction samples from each sample in VisMin:

• two samples for the Image-Task (selecting the best matching image given a caption): 1) given C0, select between I0, I1, 2) given C1, select between I0, I1
• two samples for the Text-Task (selecting the best matching caption given an image): 1) given I0, select between C0, C1, 2) given I1, select between C0, C1

Please refer to Table 3 in the rebuttal PDF for the exact set of instruction templates we used. We randomly sample one template for each task type when creating the instruction samples.

In total, we create 65k instruction samples for training. Additionally, we include 16k randomly selected image-caption pairs (this could be either I0,C0 or I1,C1) to retain the general image captioning ability of the model.

We will clarify these in the camera-ready version.

Possibility of noisy data being included in the proposed dataset, even after human filtering.

We believe the reviewer is referring to L376-378. These lines refer to our training set, not the human-filtered benchmark set. We will clarify this in the camera-ready version.

Human evaluations and statistics on the noisy data.

We did conduct a comprehensive four-stage human evaluation on the entire benchmark set as described in L174-188 of the review version (RV). The benchmark set is obtained after filtering out the samples that do not pass any of the four stages of human evaluation. We refer the reviewer to the RV for the evaluation criteria in each stage and the associated acceptance rates. All stages except the first stage have high acceptances rates (80-95%) indicating a low level of noise in our automatically generated data. For the first stage, the acceptance rate is 26%, due to a high rate of rejection (63%) of images that don’t look natural. We want to avoid unnatural looking images in the benchmark set to prevent models from easily recognizing such unnatural samples [1]. However, such noisy samples are acceptable in the training set as the goal is to teach minimal change understanding to the models. This is further validated by our experimental results where we see that finetuning on our noisy data improves model performance for both IID and OOD benchmarks.

[1] Hsieh et al. SugarCrepe. NeurIPS Datasets and Benchmarks Track 2023.

Visualizations of these noisy data samples.

We have provided random samples from both training and benchmark splits in Fig 16 and 17 in the supplementary material.

Motivation behind designing text scores and image scores for MLLMs

Text and image scores are needed to disentangle a model’s capability of “distinguishing between two captions given an image” from “distinguishing between two images given a caption”. We use the same metric for both foundation VLMs and MLLMs to ensure consistent evaluation.

Drop from text scores to image scores in MLLMs due to lack of instruction fine-tuning with two images.

We agree with the concern about image scores and for this very reason, we chose Idefics2 for our finetuning experiments as it is capable of processing multiple images (it’s trained with multiple images). GPT4, InternVL and Gemini are also capable of processing multiple images. Thus, low image scores for these models suggest an inability to detect minimal changes. Also, if the low image score for a model was solely due to its inability to process multiple images, the model should perform poorly in all categories (Object, Attribute, S.Relation, Count), however we observe particularly low scores on some categories such as S.Relation and Count.

How could VisMin benefit video understanding

Our finetuned models excel in discerning semantic differences, but neighboring video frames often exhibit low-level rather than semantic changes. Abrupt changes between consecutive frames are rare, such as a school bus changing to a car, a shirt changing color, the number of people changing or the objects swapping their positions. Moreover, our dataset does not cover action changes which are very common in videos. Therefore, we expect finetuning on our dataset to result in limited improvements for nearby video frames. We quantified the extent of improvement by reporting model performance on three video subsets of the EQBen dataset that were created by sampling nearby frames (see Fig 1 of the rebuttal PDF for some examples). Results are presented in Table 2 of the rebuttal PDF. For both CLIP and Idefics2, finetuning on our minimal change data leads to significant improvements on EQ-YouCook2 (group score), and similar performance on the other two splits (group score). From our qualitative inspection of the samples, we find that EQ-YouCook2 has more significant object changes compared to other splits (which mainly involve action changes). Thus our dataset can benefit video understanding when it has the kinds of changes that our dataset covers.

Will the model and dataset be released?

Yes we will release all model weights, generated dataset as well as generation code.

GPT-4V version

We use GPT-4-turbo-2024-04-09.

The social impact should be discussed in this paper

Thanks, we will include this in the camera-ready version of the paper. We provide a brief discussion of this in the comments below.

The positive societal implications of our work are the same as those of any vision-langauge research (as we aim to make VLMs more accurate), such as aiding the visually impaired in understanding their surroundings, teaching children through interactive demos, and interacting with in-home physical robots. However, like most other technology, accurate vision-language systems could also be used for potentially harmful applications such as extracting private information from CCTV cameras deployed at public places, and leaking personal data like credit card details when used to assist visually impaired users. Robust privacy safeguards are crucial to mitigate these risks.

Thank you for the response. My concern has been addressed in the discussion.

After reviewing the feedback from other reviewers and the corresponding rebuttals, I believe this paper makes good contributions, including the data generation pipeline, the generated training and testing sets, and the further fine-tuned MLLM models. Additionally, the reviewer recognize that the task addressed in this paper has the potential for high impact, particularly given the current trend of extending MLLMs to handle multi-image and video data. In this context, the ability to recognize minimal changes is a fundamental capability for further perception and cognition tasks. Thus, I am glad to raise my score to "Accept," while remaining open to discussions from other reviewers. Lastly, I would like to remind the authors to include the results and discussion in the final version and to fulfill their commitment to releasing the data.

Moreover, for the images we generated from scratch (counting and spatial relations), as well as the images edited on top of COCO, we avoid using Personally Identifiable Information (PII) in our text prompts used for generation (e.g., use common nouns such "a girl" instead of names entities such as "taylor swift"). We also filter all the edited images using an Not Safe For Work (NSFW) filter.

Some basic metrics like accuracy are also needed.

The T (Text), I (Image), and G (Group) metrics we use indeed measure accuracy, i.e., the proportion of examples for which the model produces the correct output [1]. The correctness criteria is different for each of T, I, G, as stated in L244-255. To obtain the aggregate T, I, G scores for the entire test set, we average the per-sample T, I, G scores across all samples in the test set. Thus, the aggregate T, I, G scores represent the accuracies on the test set. We will clarify this in the camera-ready version.

Moreover, these T, I, G metrics we use are standard metrics for evaluating fine-grained understanding, originally proposed in Winoground [2] and adopted in subsequent research efforts such as EQBEN [3] and SPEC [4].

[1] Goodfellow et al. Deep Learning, Chapter 5. MIT Press 2016.

[2] Thrush et al. Winoground. CVPR 2022.

[3] Wang et al. Equivariant similarity for vision-language foundation models. ICCV 2023.

[4] Peng et al. Synthesize, diagnose, and optimize. CVPR 2024.

The models being benchmarked are not sufficient.

We disagree with this point. Compared to the models suggested by the reviewer, such as Flava (2022) and CogVLM (2023), our selection already includes contemporary or more up-to-date models models like LLaVa1.6 (2023), Idefics2 (2024), InternVL1.5 (2023), GPT4V Turbo (2023), and Gemini1.5 Pro (2024). We chose representative models from various families – foundation VLMs and MLLMs, open-source and closed-source, covering 9 different models: CLIP (2021), SigLip (2023), BLIP (2022), Coca (2022), LLaVa1.6 (2023), Idefics2 (2024), InternVL1.5 (2023), GPT4V Turbo (2023 - closed source), and Gemini1.5 Pro (2024 - closed source). We believe our evaluation is extensive and up-to-date.

As per R1’s request, we benchmarked Flava and CogVLM (results in Table 1 of the rebuttal PDF). CogVLM outperforms other MLLMs in understanding object and attribute changes, likely due to the incorporation of grounding tasks and high-quality human-annotated datasets like Flickr30K Entities [5], RefCOCO [6], and VisualGenome [7] in its pretraining. Flava performs similarly to other foundational VLMs, except for spatial relations where both FLAVA and BLIP outperform significantly. This is likely due to the incorporation of COCO [8] in the pretraining datasets of these two models, which contains a significantly higher proportion of spatial reasoning data (10.74%), compared to web-scraped datasets such Conceptual Captions (5.72%) and LAION (2.32%) used to train other models benchmarked in the paper (see Figure 1 (right) in the PDF). To obtain the percentages, we compute the proportion of captions in each dataset that contain at least one of the following spatial relation words: left, right, top, bottom, below, above, under (these are all the spatial relation words in the VisMin spatial relation split).

[5] Plummer et al. Flickr30K Entities. IJCV 2017.

[6] Yu et al. Modeling Context in Referring Expressions. ECCV 2016.

[7] Krishna et al. Visual Genome. IJCV 2017.

[8] Lin et al. Microsoft COCO. ECCV 2014.

Reasons behind finetuned models not demonstrating improvements in all aspects.

We believe the reviewer is referring to the cases (listed below) where our finetuned models do not outperform the baselines. We have already provided the underlying reasons for each of these cases in the review version (RV), but we explain them again below:

• For spatial relation tasks in the IID setting (Table 3 in RV), we noted a slight performance degradation for CLIP upon fine-tuning on our minimal change data (L308-310). This is likely due to CLIP's contrastive loss design which often overlooks spatial prepositions, as also discussed in [9].
• For OOD settings (Table 4 in RV), VisMin-CLIP consistently outperforms the base CLIP model but does not outperform NegCLIP on Sugarcrepe, Valse and ImageCode. For Sugarcrepe and Valse, this is likely due to the NegCLIP’s method of generating hard negatives, which uses linguistic variations similar to these benchmarks, making the evaluation more favorable for NegCLIP (L317-318). For ImageCode, this is likely because ImageCode requires understanding various challenging phenomena (L325-326), such as temporal markers, actions, negation, and visibility/occlusion (source: Table 2 of the ImageCode paper [10]) which our minimal change dataset does not cover.

Additionally, for Idefics2, although finetuning on our data demonstrates overall improvements in both OOD and IID settings, there is a 2.8% decline in the performance on the VALSE benchmark. This is likely because VALSE tests for specific kinds of skills such as pronominal coreference resolution and understanding actions that our minimal change data does not cover.

We would appreciate it if the reviewer could share their thoughts on these justifications we provided above.

[9] Kamath et al. What's "up" with vision-language models?. EMNLP 2023.

[10] Krojer et al. Image Retrieval from Contextual Descriptions. ACL 2022.

“propose a new baseline method alongside the benchmark.”

Our paper aims to benchmark existing models and investigate if fine-tuning on minimally changed data alone can boost performance, without introducing new methods or losses. We propose a model-agnostic recipe to improve fine-grained understanding capabilities of both foundational VLMs and multimodal LLMs, which we believe would be of sufficient value to the community.

Potential ethical issues in the dataset

Since we only make minimal edits (object, attribute, counting, spatial relations) on top of published publicly available datasets such as COCO, we do not anticipate our edited images to have any data privacy, copyright, and consent issues following the definitions in the NeurIPS Ethics Guidelines. See more details in the comments below.