List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

Thank you for the appreciation of our work! Below are the response to your question.

Q1 The application area of this work and expanding this methodology to foundational models.

A1 Thanks for your suggestion. In our experiments, we have shown that our proposed dataset and learning paradigm is useful not only for empowering MLLMs with SoM prompting, but also for enhancing general visual reasoning and reducing hallucinations on popular benchmarks. It can be added as a complementary data source to existing visual instruction tuning datasets (see our additional results provided to other reviewers) for any visual LLM training.

Nevertheless, it would certainly be interesting to explore how this methodology can apply to other types of foundational models, e.g., enhancing image/video generation for diffusion models.

Thank you for reviewing our paper and the appreciation of it in terms of strengths. Below are our responses.

Q1 Contribution of the work.

A1 We would like to clarify our contributions here. First of all, data is perhaps one of the most critical aspects for today’s foundational models. Though we start with discussing SoM prompting and our work is motivated by it, we find “listing items one by one” is also effective for general MLLM training: if a model learns to list items with a specific order (in our case, it is determined by the visual numeric tags), it gains a fine-grained understanding of images. Verified by our experiments, we believe this is an important implication and opens up an interesting discussion on how to better align visual and language.

Q2 Details about the approach

A2 Thanks for the suggestion. Our data creation pipeline is simple, we add numeric tags to the images using semantic-SAM (section 4.1), and generate listing-style captions with GPT-4V (section 4.3). Data examples can be found in Figure.9-10 and Table.4 in appendix, we will add constructed examples to the main paper. We will also add a figure for readers to better understand the pipeline.

Q3 Comparison with direct referring

A3 In LLaVA-665k dataset we used, there is a category of visual referring data, ref-coco and visual genome, with 134k data. Hence our results in table.2 show that SoM data complements this type of data. Most visual referring LLMs [1,2] put coordinates of 1-2 objects into text, while we add visual tags on images to encode spatial information. We will cite more papers along this line, for example, [2, 3, 4]. Let us know if there are any specific papers to be included.

Q4 Details about the binary classifier

A4 We use the ViT from [5]. We use all raw images from coco train-2017, and construct tagged images by randomly adding 1-10 numeric tags (numbers chosen from 0-99) to the images. We built a balanced dataset with 236K images in total. We train the binary classifier for 5 epochs with learning rate 5e-5 and batch size 128. We will add details to the paper.

[1] Improved Baselines with Visual Instruction Tuning

[2] VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

[3] Ferret: Refer and Ground Anything Anywhere at Any Granularity

[4] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic

[5] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Thank you for the appreciation of our work! Below are the responses to your questions.

Q1 improvements on POPE are not very significant.

A1 The results are significant if we compare with the performance from other models (average F1 score), for example, the recently released phi-3-vision model has a score of 85.8, LLaVA-next-LLaMa-3 has a score of 87.0. Most models fall into the 83 to 87 range, even with more advanced models and data. The performance of LLaVA-1.5 is already decently high on POPE, hence it may be hard to improve further. Our SoM data can boost LLaVA-1.5 from 85.9 to 87.0, which shows its efficiency. We also present results from LLaVA-1.5-7B, where the performance improvements are consistently observed (slightly higher on some tasks, such as SEED-I and MM-Vet, than the 13B model)

Q2 How to ensure the quality of the dataset.

A2 Thanks for the question. We explore two approaches, one is to modify system prompts, the other is to design high-quality demonstrations for few-shot prompting. We have an example in Table.4 in Appendix. We conduct qualitative comparison between our model and GPT-4V in Figure.9 and 10, showing our model can generate better listings than GPT-4V in a zero-shot setting. Overall, how to control the quality of any synthetic datasets is an interesting direction that requires further investigation.

Thank you for the appreciation of our work! Below are the responses to your questions.

Q1 More evaluation benchmarks

A1 Thanks for the suggestion. We will test on more benchmarks before the final version.

Q2 Results on 7B parameter model

A2 Thanks for pointing it out. We are limited by compute resources when running ablations. Results from LLaVA-1.5-7B are here:

Additionally, we also present results from another recent M-LLM (xgen-mm-phi3 on HuggingFace, with a different architecture and pretrain recipe). We follow the setting on LLaVA to finetune the base model, with or without SoM data:

We will add these results to show the performance gain of different models with our SoM data

Q3 Do we need GPT-4V to produce the descriptions for each of the tagged objects?

A3 Thanks for initiating an interesting discussion. For MS-COCO we do have abundant annotations to create synthetic data. We considered this method in the early stage of our project, but after checking samples from VG, we find that the captions are not really “open-vocabulary” as what GPT-4V is capable of. For example, regional captions such as “man wearing in a hat”, “tree near the water” missing fine-grained attributes about the color of the hat, type of the tree, man’s posture or action, etc.

A similar idea that may work for all image sources could be extracting local patches using object detectors, and use GPT-4V or other caption models to annotate the patches. In our work, we go with a simple and scalable method that relies on two foundation models, semantic-SAM and GPT-4V, to generate such data.

For fair comparison with LLaVA-150k and share-GPT4v data, we use the same images from COCO to make sure no extra information are introduced in the images, so we can faithfully verify this new learning paradigm (list items one by one) is indeed effective. In the future, one can scale up the data to different image sources, using our pipeline or the alternatives discussed above.

Thank you for the appreciation of our work! Below are the responses to your questions.

Q1 The proposed approach and dataset are only applied to one MLLM. It would be good to try other MLLMs to see if the results generalize.

A1 Thanks for your suggestion. We deliver additional results from two other MLLMs.

First, we used another recently open-sourced M-LLM with pretrained weights available (xgen-mm-phi3-mini-base-r-v1 on HuggingFace, with a different model architecture and pretrain data compared with LLaVA). We compare with two visual instruction tuning recipes as in our paper: LLaVA-665K and SoM-LLaVA-695k to finetune the same base model, and evaluate their performance on the five benchmarks, using raw images without tags during testing. The performance comparison is shown below:

We also tested our dataset with LLaVA-1.5-7B which has a smaller language model (vicuna-7B) compared to the one we reported in Table.2. The performance comparison is shown below:

Q2 It would be good to evaluate the in-context learning (without fine-tuning) performance using exemplars from the new dataset.

A2 Thanks, that is an interesting direction to explore, with the recent releases of open-sourced interleaved M-LLMs that are capable of performing in-context learning.

Thank you for the appreciation of our work! Below are the responses to your questions.

Q1 Effectiveness on other types or sizes of VLMs

A1 Thanks for your suggestion. The reason we choose LLaVA is, it is a fully open-sourced model (both data and model weights for both pretraining and SFT stages are released), so it makes the full diagnosis possible. And its model size and training recipe is what we can afford given the limited budget.

Here we deliver additional results from another recently open-sourced M-LLM (xgen-mm-phi3-mini-base on HuggingFace, with a different architecture and pretrain recipe). We use two datasets: LLaVA-665K and SoM-LLaVA-695k to finetune the same base model, and evaluate their performance:

The performance comparison is shown below:

We also tested our dataset with LLaVA-1.5-7B with a smaller LM (vicuna-7B), as shown below:

Q2 Additional overhead for visual instruction tuning.

A2 Our dataset is relatively small (30K in total), adding less than 5% training time for finetuning. Finetuning an already instruction-tuned model with SoM data is possible, though it may lead to forgetting of previous data. We follow the common practice to mix the training data.

Q3 listing data from 75k to 100k

A3 We generate the listing with coco annotations. The object classes are limited (80 classes) hence there could be some overfitting issue. Also, the annotation itself can be noisy, so a matching score of around 85% may be hard to improve. We will clarify this in the revision.

Q4 Adding tags to small objects

A4 We follow the tagging method in the SoM paper [1] where we slightly move the tag down from the center by a few pixels, so small objects are still visible.

Q5 Training time

A5 It takes around 20-22 hours to train LLaVA-1.5-13B on 8 A100s (80GB).

Q6 Typo and formatting.

A6 Thanks for your careful review, we will revise the typos.

[1] Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V