What matters when building vision-language models?

Thank you FVDr for your review. We appreciate your positive feedback regarding the strengths of our work.

Regarding the point mentioned about section 3.1, we can clarify our approach:

We wanted to start by showing some results with the cross-attention architecture that was the reference approach at that time. This is why the ablations in Section 3.1 were conducted under the cross-attention setup. However, we have verified that the two most significant findings (replacing Llama1 with Mistral and CLIP with SigLIP) also hold under the fully autoregressive approach. Given that these results were consistent with those obtained from the cross-attention architecture early in training, we did not extend these additional training to full completion to save computational resources. As a result, these ablations were not included in the final paper.

We will be pleased to add these results with comments in the final paper for clarity and readability. We also ensure that our conclusions are in line with recent competing work in the literature.

Thank you for your detailed review. We appreciate your feedback and will address each point individually. We unfortunately cannot cite your questions without exceeding the maximum length.

1. It is true that our comparisons primarily involve open-source models. This is because state-of-the-art commercial models tend to be significantly larger, making direct comparisons challenging. However, we did include MM1, which is not an open-source model. Additionally, in Table 15 of the Appendix, we provide comparisons with Gemini and Claude 3. While our model approaches their performance on benchmarks like MathVista and TextVQA, it is clear that larger scales are needed to contain extensive factual knowledge in the model’s weights, and perform well on benchmarks like MMMU.

2. You are raising a good point that this is a very active field, with many concurrent works, and there are surely potential areas for improvement in the proposed model. We compared ourselves to the best models we were aware of at the time of the submission. Besides InternLM-XComposer2-VL known after, we are not aware of any other model having potentially better scores for the same size at the time of the submission.
   Now, if we had to justify our work against InternLM, we would say that the scope of the study is different: when building our model, one major point was the efficiency at inference time. This involved using a perceiver resampler to reduce the number of visual tokens. While this approach works great to obtain even better performance on most tasks, a high number of tokens is still needed for OCR heavy tasks. We also do not have a lower score on the most tracked benchmark MMMU. Second, we only use open datasets, while they use proprietary data. Finally, we focused on learning from ablations for the researchers/community rather than just giving a model.

3. We evaluated our base model with 8 in-context examples, as we wanted to compare our model to MM1-base-7B, the best base model in its class size at that time, which was only evaluated using 0-shot (which doesn’t really give information since the model cannot know in which format it should format its answer) and 8-shot.
   We did see an expected improvement when going from 4-shot to 8-shot by 1-2 points.
   When evaluating the base model, we are very sensitive to the template expected by the benchmark, which was never seen during training. For example, answering “yes it is” while the growth truth is “yes” would be counted as false. Adding more in-context examples boosts the scores, but there is no evidence at first that it’s improving the model’s ability to solve the task. Potentially, it could only help it to better follow the template expected by the benchmark. A solution to discriminate that would be to use a LLM as judge approach, but it would start to be a bit complex and out of the scope of this current study, and also not comparable with any other work, so this is also a reason why we didn’t push the in-context learning evaluations.

4. We guaranty that all the datasets used for the training are publicly available. Moreover, we are working on cleaning our codebase and documenting it before opening it.

5. We agree on the importance of testing under real-world conditions. Our model has been publicly demoed and tested by many users, although we cannot share details here without compromising anonymity. Additionally, our model was tested by many users against many other models with real-world images and prompts, and obtained a strong ELO score.

6. We provide in Table 15 in Appendix a comparison with the Gemini and Claude 3 series of models.

7. For both the cross-attention and fully autoregressive architectures, unfreezing all parameters during pretraining led to unstable training and eventual loss divergence, motivating Finding 3 in our paper.

8. Thank you for this suggestion. We have added the number of trainable and total parameters for each model in Table 3.

9. The entire pretraining and ablations were done with LoRA. After DoRA was published, we tested it as a direct replacement for LoRA and found similar or slightly better performance. We decided to use DoRA for fine-tuning as it did not hurt performance, did not add more parameters, and was proven to be efficient in various cases by the community.

10. The model can handle an arbitrary number of images as input, with the maximum number limited by the sequence length it was trained with. In our case, the maximum sequence length was 2048, allowing for just over 30 images. Fine-tuning with a longer sequence length could enable inference with more images, such as for videos.

11. We have corrected this issue.

12. Besides the comparison to commercial VLMs mentioned earlier, we acknowledge that the model is not evaluated on a large number of benchmarks (6 in Table 15). This is due to the challenge of finding benchmarks that effectively distinguish VLM performance.
    Some open-ended benchmarks are unreliable as good scores often indicate adherence to expected answer templates rather than true capability. For example, we obtained 81.2 on VQAv2 while Gemini 1.5 Pro scored 73.2. It doesn’t mean that our model is better than Gemini 1.5 Pro. We kept benchmarks like MMMU for its comprehensive categories, MathVista for reasoning and math abilities, TextVQA and DocVQA for OCR and document understanding, and MMBench for general VQA evaluation.

Thank you again for your detailed comments. We welcome further discussion if you have any additional questions.

Thank you reviewer Gtma for your interesting remarks, we will try to cover them one by one.

While I fully appreciate the importance of the work in standardizing a recipe for training VLMs, I believe it lacks vigour to make such bold claims. For example, Finding 4 does contradict results in Table 9, TextVQA.

You raise a valid point.
Our ablation results show that for most tasks, particularly non-OCR tasks, the number of tokens to encode the image is not critically important.
This is significant for applications like robotics, which do not necessarily require strong OCR capabilities but need efficient inference.
This is supported by Tables 9 and 15, which show the image splitting strategy's utility primarily for OCR tasks.
This finding aligns with other works published on Arxiv after our submission.
We agree that Finding 4 should be stated more cautiously. We have rephrased it to: “Reducing the number of visual tokens with learned pooling significantly improves compute efficiency at training and inference while improving performance on non-OCR downstream tasks.”

Some results seem a bit too overarching. For example, while the authors make a generalised finding “the quality of the language model backbone has a higher impact on the performance of the final VLM than the quality of the vision backbone” (Finding 1), they do mention that this might be because the vision encoders might not be well trained (L108), which seems conflicting.

We understand that Finding 1 might have been confusing.
Our intent was to highlight that, given a fixed size for the vision encoder (around 400M for CLIP-H or SigLIP) and a fixed size for the language model (around 7B for Llama1 and Mistral), using the best available open-source models at the time, the most significant performance improvement was observed when upgrading the LLM.
We have clarified this by rephrasing the finding with: “Better pre-trained LLMs and vision encoders lead to better performance on multimodal tasks. However, with the best current models for both, the LLM has a higher impact.”

It would be nice to have most/all the findings validated for XXXXXX.

All the findings are validated for the model XXXXXX, except for Finding 1 which has been done for the cross-attention architecture. However, we have validated that it holds also for the fully-autoregressive architecture as XXXXXX. We provide a detailed explanation for this question specifically in the answer to reviewer FVDr.

Can the authors elaborate on how YYYYYY is different from existing VLM instruction tuning datasets? Is there any scope of test dataset leaks?

At the time of submission, VLM instruction tuning datasets were primarily of two types: synthetically generated Q/A pairs, mostly using ChatGPT (like Llava), and compilations of existing training sets from different benchmarks (like InstructBLIP).
We adopted the second approach, conducting an extensive literature review to compile a mixture of the 50 highest quality datasets we found.
To our knowledge, this scale and diversity had not been done before, with many interesting datasets previously missed.
We transformed all datasets into a consistent format, created images for benchmarks without them (like tables), merged Q/A pairs on the same images to create multi-turn dialogues, prepared non-prompted datasets with prompts, and removed images present in test sets of benchmarks we evaluate on. We have done a contamination analysis by checking that images present in the splits of the benchmarks we evaluate on are not present in our dataset. We guarantee that the dataset YYYYYY is released for the community.

How do the authors ensure that the superior performance of YYYYYY is not driven by the dataset alone and only because of their findings?

The dataset YYYYYY certainly plays a crucial role in the final performance, but it is introduced during the fine-tuning stage. The findings are primarily derived before this stage, with the exception of Finding 6, which is independent of the data. Thus, the ablations in Section 3 informed better model and training method choices, while the dataset YYYYYY provided an additional, independent boost.

Thank you again for your review, and we will be happy for a further discussion if you have any additional questions.

Thank you for your remarks that we’ll address one by one, unfortunately briefly and without citing your questions not to exceed the max length.

1. Optimization for chat scenarios depends on training data.
   We used YYYYYY, a compilation of 50 high-quality, mostly human-annotated datasets from literature.
   These datasets' short Q/A pairs can lead to brief model responses.
   To address this, we fine-tuned first on YYYYYY for improved performance (reasoning, OCR, …), then briefly fine-tuned on long conversations to adapt output length.
   LLaVA, trained directly on long conversations, doesn’t need this step.
   We avoided using current conversational datasets due to their synthetic nature (often ChatGPT-generated) and simplistic, not diverse questions.YYYYYY's creation and open-source release aims to improve VLMs by leveraging diverse, accurate datasets from literature.

2. We acknowledge the rapid evolution of multimodal vision-language models, with some overlap in findings expected given typical 6-12 month project timelines.

Regarding the very recent papers you mentioned:
-[c] BRAVE (April 10): Considered concurrent work per NeurIPS guidelines, given its date.
-[a] Prismatic VLMs: We’ve cited it extensively. While there are similarities, we've expanded upon their findings. For instance, while they find that unfreezing the vision encoder degrades significantly the performance, we demonstrated that a LoRA strategy can achieve stable training and improved performance when adding trainable weights to the vision encoder, as mentioned just before Finding 2 in our paper. We also highlight additional differences in section 4.1.
-[d] VILA: Also cited, VILA's scope on image resolution was more limited (224 to 336 pixels), while we extend it to 980 pixels and focused more on image splitting strategies rather than comparing the input resolutions for a fixed number of tokens.
-[b] "Eyes Wide Shut?": While this paper analyzed the impact of pre-trained vision encoders, it represents only a very small portion of our broader study.

While we appreciate the similarities you've noted (from our contributions from section 3.1), we believe our work offers several unique and significant contributions, particularly in sections 3.2, 3.3, 3.4, and 4:
Most important part for this answer, please read our general message "Author Rebuttal" to all reviewers for a list of our unique contributions, as we cannot put them here due to length constraints.

These contributions offer new insights to the field beyond the points initially highlighted in your review, that can be now seen as complementary points in our broader study.
We've put effort to be comprehensive in our literature review and comparisons against other works throughout the paper, as shown by our extensive bibliography.

We hope this clarification helps to illustrate our contributions. We will do our best to highlight even more these differences in our final version.

3. We agree that [b,c,d] all contribute to advancing VLMs in various ways.Our work, while complementary to these efforts, introduces several novel approaches and improvements that we believe are unique and significant (cf previous answer).
   We have provided comparisons with other models throughout our paper. For instance, a key aspect in our strategy is efficiency at inference. We have managed to do so with three points:
   -Using an efficient pooling strategy with a very small number of visual tokens. This is compared against MM1, SPHINX-2K, DeepSeek-VL, DePALM (some of the strongest VLMs at that time, with better performance than VILA or Prismatic VLMs).
   -Using the image aspect ratio preserving strategy. To our knowledge, there is no other VLM using this strategy, so it could be compared against all the other models.
   -Fine-tuning with variable visual token counts (with/without image splitting). Unlike models we compare to like Llava, MM1, and Monkey that fine-tune only with image splitting, our approach allows users to choose token count at inference. This flexibility wasn't the norm before.

4. Our findings remain relevant in the context of recent works. The field has shifted towards unfreezing language models when feasible or using LoRA for improved stability, aligning with our proposed methods.
   Most recent works also use image splitting with variable sub-image counts during training, enabling compute-performance trade-offs at inference (section 3.4).
   Our dataset YYYYYY demonstrated the potential of leveraging diverse existing academic datasets for instruction fine-tuning, previously implemented at a smaller scale with less diversity.
   The dataset Cambrian-10M is very inspired by our work, containing most of the public academic datasets we have selected, and the images of the tables we have rendered (we created these table images from datasets that originally lacked visual representations, employing various styles to enhance diversity).\  Notably, our XXXXXX-8B model outperforms Cambrian-1-8B on challenging benchmarks like MMMU and MathVista, while being significantly faster at inference due to fewer visual tokens (64 vs. 576).

5. To mitigate hallucinations, we suggest: (1) improving image-text alignment in data, (2) using DPO to reduce hallucination probabilities, and (3) incorporating negative prompt datasets.
   We've released all datasets used. Regarding "lack of openly available", prior works often used LLMs to create synthetic Q/A pairs, limiting diversity and encouraging hallucinations. Our approach involved compiling and processing/modifying diverse, high-quality publicly available datasets.
   RLHF methods could enhance VLM robustness against jailbreaking.

We hope that our responses have provided a clearer picture of our research's unique aspects and its position within the current landscape of vision-language models.
We have tried to show why our work could be impactful and how it's different from what has been done before.

Thank you for your re-evaluation.

We created a public collection with

• the datasets used (pre-training + fine-tuning)
• the models (base, instruct, instruct-chatty)
• 4-bit quantized versions of the models
• tutorial to fine-tune the models

easily downloadable.

We are working on the remaining parts.