Thank you for your comprehensive review and for highlighting the broader implications of our findings. We're particularly glad you recognized how our results on the 80% continued training approach and the 90-10 image-text ratio could impact future VLM development practices. Your historical perspective connecting our "from-scratch" findings to earlier vision-language models (2020-2022) adds valuable context.

Main Response:

Additional Evaluations [W1]

My main problem is that these insights are only demonstrated on QA and VQA tasks, which I dont find enough for deducing generalizable insights. Good VQA performance does not necessarily reflect general performance. There are a suit of benchmarks (e.g., those used in LLava) used to evaluate the vision-language models. At least include other vision-language tasks such as image captioning and its variations? Visual entailment? VQA with Textual Explanations (VQA-X) and visual entailment with explanations (e-SNLI-VE) can also be good dataset for testing.

This is a good point – we agree that using only VQA is a bit restrictive.

1. Our evaluations also include other metrics aside from general VQA in our aggregate metric (which we called “stable_score”). Specifically, we considered a variety of tasks such as TextVQA (document understanding and OCR), RefCOCO (object localization) and OCID-Ref (cluttered object localization). Our current evaluation contains at least one benchmark from each of the categories listed in the Qwen2-VL paper (except for video/multilingual). See Appendix E for a detailed list of our tasks.
2. Nonetheless, we agree that using a larger set of benchmarks would be better. To fix this, we add our scores on MMMU below. We are also evaluating on other datasets and will add them over the next few days as soon as the results come in. | Checkpoint | MMMU Accuracy | |------------|-----| | 90-10 text-image ratio, 100% checkpoint, 4 epochs | 0.3066 | | 90-10 text-image ratio, 80% checkpoint, 4 epochs | 0.32 | | 90-10 text-image ratio, 60% checkpoint, 4 epochs | 0.327 | | 90-10 text-image ratio, 40% checkpoint, 4 epochs | 0.3533 | | 90-10 text-image ratio, 20% checkpoint, 4 epochs | 0.24667 | | 90-10 text-image ratio, 0% checkpoint, 4 epochs | 0.29333 | |PaliGemma-3B-mix-448 | 0.336 | |Qwen2-VL-2B | 0.423 | |LLaVA-Next-Mistral-7B | 0.37 |

The results shown are for our model initialized from the different text pretraining checkpoints, then trained on image text with a mix of 10% image captioning, 90% text, then finetuned for 4 epochs. The compared models, PaliGemma, Qwen-VL have public results for MMMU (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) we recomputed them to make sure our evaluation pipeline does not make a difference and computed the results in the same way for LLaVA-Next-Mistral. Surprisingly, and as opposed to our previous results, they do not show the 80% checkpoint to be best but it is still stronger than the 100% checkpoint.

Performance of 0% image data model [W2]

Something in Figure 5 caught my attention. When the image ratio is 0% (as if no image data is added, and we continue training on text-only data, as in a normal text-only LLM), the VQA performance is already around 75%, even though the model has seen no images. This makes me wonder that the model is answering questions without really looking at the image (this is a well-known bias problem in VQA models, presented in [R1]). Therefore, i am afraid these results may reflect biased models. I wonder what the results are without using any images? What about other VQA reasoning datasets such as A-OK-VQA?

To clarify, the 0% here refers to the 0% images used for pre-training. There is always a fine-tuning stage for all models, so the 75% that you’re seeing in Figure 5 is after fine-tuning. For more details, see the green “Setup” boxes – we finetune for 4 epochs. In other words, we never use a model that is trained only on text.

Masking [W3]

The authors mention in L739 in the appendix: “This time, we also make sure to mask out any system prompt and human question, only keeping the model responses unmasked.” If no question is there in the image instruction tuning stage, then this is not instruction finetuning (this is just a different dataset of image-text pretraining, similar form to the DataCompDR-1B). What is the point of conducting instruction tuning then? Im not sure exactly what is the rationale behind masking out the question from the instruction?

Thank you for noticing this imprecision. We will clarify this point in the main paper. Here by “masked”, we meant to say masked in the loss computation. It is given without masking as a model input.

I thank the authors for their response and additional experiments. However, can the authors comment on the MMMU results? As from what I see, they do not agree with the hypothesis mention in the paper, as it seems that the 40% checkpoint works best (in the main paper, it was 80% that the authors claimed that worked best). Additionally, the results shown are for what model? or are they for the method in Figure 2? or PaliGemma? Or Qwen2-VL-2B? What ratio and what checkpoints are these other models (PaliGemma, Qwen2-VL-2B, LLaVA-Next-Mistral-7B) using? I am not sure why the authors paste this table with no discussion.

In general, i see that my doubt is correct, the hypothesis is only valid for VQA tasks, and it is not generizable to other tasks. I am not sure about the impact this paper will have in that case.

Thank you for your interest and this follow-up question. We should have explained and discussed these results further. We are also surprised that the MMMU results show better performance from the 40% checkpoint and did not have the time to explore the reason why further at this time.

A few points we want to raise regarding this:

1. While these new results do not support our claim that the 80% checkpoint is the best, it still shows better results than the 100% checkpoint (supporting our claim). Moreover, considering all benchmarks, the 80% checkpoint is still the best choice for a generalist model.

2. In Appendix F, there are also other tasks like VizWiz where the 80% checkpoint doesn't perform the best. This is why we report an aggregate metric such as stable score, individual benchmark results might depart from the average performance. There are some benchmarks (like MMMU as we're seeing) where it doesn't perform the best (and that's totally expected). The main advantage of using some aggregate metric like stable score is so that we don't overindex on a single task like this.

• Now this brings the question of how well-designed our aggregate metric is, and whether the current set of tasks we're considering is too narrow. We tried our best to consider a variety of tasks, but we agree that it would be a more representative metric if we added a more diverse set of tasks. We are currently working on adding more tasks such as captioning and entailment and will let you know once we get more results.

The results shown are for our model initialized from the different text pretraining checkpoints trained on image text with a mix of 10% image captioning, 90% text, then finetuned for 4 epochs. The compared models, PaliGemma, Qwen-VL have public results for MMMU (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) we recomputed them to make sure our evaluation pipeline does not make a difference and computed the results in the same way for LLaVA-Next-Mistral. These models use the fully trained initial LLM checkpoint.

We would also like to point out that we did include non-VQA benchmarks in our published evaluations such as OCID-ref and RefCOCO that are localization benchmarks evaluated on intersection over union of bounding boxes. These results also show better performance from the 80% checkpoint.

I thank the authors for their response. I think the authors present an interesting finding that will likely have an impact on VLMs. This finding basically says to introduce image pretraining for 10% of the time, earlier than expected as per the scaling law (at 11.2B and 22.4B tokens for 40% and 80%, respectively). The only reason I am not raising my score and giving this paper an accept (score of 8) is because 1) this finding may depend on the VLM and architecture (as the authors showed results on their own architecture), 2) (my initial concern) there is no clear rule of when to actually introduce the image tokens. Sometimes 40% is better for a set of tasks, sometimes 80% is better for another set of tasks. Although both lead to increased performance, i still find it necessary to have some guidelines for researchers to follow (although i understand that its hard to have one rule that works on all). Therefore, I wish to stay with my score as a borderline acceptance.

Thank you for your detailed review of our experimental work on VLM training optimization. We're pleased that you found our extensive ablation studies and empirical findings to be well-supported and meaningful. Your recognition of the practical significance of our results is greatly appreciated.

Main Response:

Scaling (Weakness #1):

This study is very limited to the parameter size of the LLM. It seems hard to generalize to different size of LLMs. It seems this paper adopted 1B LLM, commonly used 7B and 13B LLM should also be studied. How do authors expect the findings might change at different model scales?

We performed a similar set of experiments with an LLM of 154 million parameters and another with 411 million parameters and found consistent patterns, with the exception that smaller models’ downstream performance require more than 4 epochs for the performance to degrade. The trend also shows that the difference in downstream performance between the models initialized from the 80% text training and 100% is decreasing as the model size increases.

Because our findings mostly hold from 154M to 411M to 1B, we believe that it will continue in larger scales. We attach our scaling plots in this anonymous link: https://zenodo.org/records/14207627

We are currently running a set of experiments at the 7B scale. However, there’s a chance these experiments may not finish in time for this discussion period. We wanted to get this response out first so that we could begin author-reviewer discussions, but we will share the 7B results as soon as we get them.

Availability of checkpoints (Weakness #3):

Current LVLM starts with a fully pre-trained LLM. But it seems not many strong LLMs can release their weight at 80%. What are the practical implications of this paper's' findings given the limited availability of partially trained LLM checkpoints? Could the authors explore ways to approximate the approach using fully trained models?

We agree that requiring partially trained checkpoints may limit immediate applicability of this paper’s findings. However, we hope this work will encourage open-model contributors to release more than just their final checkpoints in the future. Intermediate checkpoints are likely stored by model developers and could be released if there is sufficient demand, furthering open research. Publishing papers such as ours which shows the value of intermediate checkpoints can help push the field towards this direction.

Moreover, while our findings are directly relevant to labs with access to internal checkpoints, they also provide broader insights into the training dynamics of language models, contributing to the field’s understanding. Importantly, our experiments at smaller scales demonstrate that these insights are applicable even to those working on smaller models, ensuring accessibility beyond frontier labs.

We also appreciate the suggestion to explore approximations using fully trained models, which is an interesting direction for future research.

Comparisons to existing models (Weakness #4):

Would like to see the final model trained by this paper compared with existing LVLMs such as mentioned in paper's Table 1 or well-recognized models such as LLaVA-Next, LLaVA-One-Vision etc.

While our aim is to study training recipes in this paper, we agree that grounding the results with prior SOTA VLM results comparison will strengthen our results and be more convincing about the validity of our results. Find below a comparison of the results reported by Prismatic VLM and PaliGemma. Note that these are larger models but we could not find similar sized models that report the same set of evaluations as ours. We will run evaluation on the Qwen-VL-2B-Instruct model and add it to this table.

Hi reviewer ans3,

7B experiments

We have trained a 7B model, and we are happy to share that our method indeed scales well to the 7B scale. This confirms the main finding of our paper, which is that pre-training with a mixture of image and text data allows models to perform better on downstream tasks.

More specifically, we compared two 7B models as follows:

1. Introduce images (at a 90:10 text:image ratio) ~50% of the way through pre-training for a DCLM-7B checkpoint (2.3T tokens at a 4.3T token schedule). This is done at 1x Chinchilla tokens (~140B tokens). Then fine-tune with Llava for 4 epochs.
2. Take the final cooled-down DCLM-7B checkpoint (2.3T tokens cooled-down for 270B more tokens). For a fair token-matched comparison, we continue training with just text at the same token-matched scale as the model above (i.e. 1x Chinchilla tokens). Then fine-tune with Llava for 4 epochs.

Here, model 1 is basically what we're proposing, and model 2 is very similar to how many VLMs are conventionally trained.

Our results are as follows:

We hope that this gives you a bit more confidence in the validity and scalability of our results. If time permits, we will add even more 7B results to the final version.

..

With this update, we hope we have addressed all of the issues that you have raised. We deeply appreciate your feedback and we’d like to explore whether there might be a chance to reevaluate the score in light of all the points we have addressed. If you have further questions or concerns that you feel is a major blocker for acceptance, please let us know.

Responses (cont.)

3. [W3] Thank you for raising this. We think it’s a very good point that will help strengthen our paper. We trained a new model using the maximum learning rate of the 80% checkpoint and the initial checkpoint at 100% of the text training and report the results below. It appeared that our choice of hyperparameters was not optimal since this one yields slightly better results. However, the results from the 80% checkpoints are still better, so these new results support our overall conclusion. | Model | Stable-Score Text | Stable-Score Vision | |--------------------------------|-------------------|---------------------| | 100% checkpoint | 13.87 | 43.35 | | 80% checkpoint | 15.44 | 46.07 | | 100% checkpoint trained with 80% parameters | 14.29 | 45.61 |

4. [W4] We agree that evaluating only on accuracy has significant drawbacks. We would like to point out that we included tasks such as RefCOCO and OCID, which are bounding box prediction tasks that are evaluated as a percentage of results greater than a threshold which is not an exact match or multiple choice metric. It measures the percentage of produced bounding boxes with an intersection over union greater than a threshold.

Minor points

5. [W5] This statement from our paper is wrong, thank you for noticing this mistake, it will be removed. See our answer to your point W2 for more details.

6. [W6] From the PaliGemma paper, paragraph 3.2.1, it is reported that unfreezing the image encoder is better but it requires some adaptations of the learning rate schedule with two different schedules, one for the image encoder and one for the LLM. We leave this study that would involve tuning some additional hyperparameters to future work.

Questions

The LLM is only trained for 28B tokens in this work, while it is originally trained for 4.3T tokens in the DCLM-1B work, a factor 150x more data. Authors argue with the scaling laws from the Chinchilla paper, which would suggest “20 times number of params” of tokens is enough. But then, why use a LLM that is trained on “3070 times number of params” tokens?

It is true that better performance would be achieved with more training. When saying that “20 times the number of parameters is enough”, we do not claim that it is enough to achieve the best performance, only that we believe it is enough to study trends in the different models performance. We cannot claim this number is optimal in any way in the context of image-text training, only that it is a reasonable training time to study model performance variations. We used the best LLM available that is widely overtrained as you noted because it does yield better performance and is closer to what practitioners would chose.

How many (approximate) GPU hours were used for each experiment, or at least, how expensive is one run of the 1.4B model on 28B tokens? Appendix A shows GPU hours for the text-only training but not for the image-text training.

For a 28B tokens image-text training of a 1.4B parameter model, we used 2 nodes of 8 H100 for 4h15min. For a 4-epoch fine-tuning, we used either a node of 8 40GB A-100 for 10h40min or a node of 8 H100 for 2h20min.

Thank you for your thorough and thoughtful review of our work. We appreciate your positive feedback on the presentation and clarity of our findings. Your comments highlight the importance of our research in advancing the understanding of VLM training procedures, particularly in studying the optimal timing and approach for incorporating image-text data. We're especially glad you found value in our key finding about injecting image-text data during the LLM cooldown phase rather than as a separate training stage.

Main Responses

1. [W1] While our aim is to study training recipes in this paper, we agree that grounding the results with prior SOTA VLM results comparison will strengthen our results and be more convincing about the validity of our results. Find below a comparison of the results reported by Prismatic VLM and PaliGemma. Note that these are larger models but we could not find similar sized models that report the same set of evaluations as ours. We will run evaluation on the Qwen-VL-2B-Instruct model and add it to this table.

2. [W2] The text performance that we report is indeed fairly low and we only discussed the trends, neglecting their absolute values in the paper. We agree that absolute values and comparison with SOTA models is important. Thus, we report our results in the table below and compare them with the performance of SOTA baselines: LLama 3.2 which performs worse than the OpenLM model we used and Qwen 2.5 which performs significantly better but was released after the beginning of this work. The text scores are lowered by the image training process (Ours no FT is our model from the base 80% trained on image captioning and text). The performance is reduced by 4.5% after image training and an additional 5.5% after LLaVA fine-tuning, as shown in the table below. (Interestingly, AGIEval is the only text benchmark that sees an increase in score after image training and again after LLaVA finetuning.) To our knowledge other works do not report the text performance of the models after tuning them for image understanding.

Thank you for your detailed responses. I have two follow-up questions:

"[...] we included tasks such as RefCOCO and OCID, which are bounding box prediction tasks [...]" - how exactly are you running object detection on your VLM, are you finetuning an object detector head on top of your frozen features, or does your model include object detection capabilities? I could not find this in the paper or supplementary. So far as I understood it after finetuning on the Llava Instruction Data there is no further finetuning on the target sets.

Statement "It does this while retaining performance within 2% of our text-only baseline on downstream text evaluations." "[...] will be removed". When are you planning to update the paper?

Good questions!

1. Actually the llava_v1_5_mix665k.json subset of https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K already contains bounding box prediction as one of its tasks, so we do not need a separate finetuning step to do bounding box prediction.

Specifically, Llava uses the prompt "Please provide the bounding box coordinate of the region this sentence describes:"

For further exploration, please check out this short Jupyter notebook (anonymous): https://colab.research.google.com/drive/1ePSJsvGM9fOcPv9x4VWPcx3feN8jWZfB

2. Regarding updating the paper, we will probably do it after the review process. We understand that we could do it now, but we prefer to keep the paper in its original state so that reviewers and AC can see and compare when doing these discussions. We feel it might get a bit confusing if the reviewers are discussing something and then suddenly the point in discussion is no longer in the PDF anymore or it gets changed in the PDF suddenly.

If you feel strongly towards us updating the paper now, please let us know, and we will gladly do so.

Hi reviewer 7Ue5, we just wanted to let you know that we have posted a new overall response regarding some new updates we have during the rebuttal period.

https://openreview.net/forum?id=Pj4Aid3XqL&noteId=5VHjy4YvN9

This is mainly about the scalability (to 7B) and generalizability (to other benchmarks) of our work.

We really appreciate the detailed feedback you gave to us. It is validating to hear you found our idea relevant, and we sincerely thank you for taking the time to share your knowledge and insights on our paper. We were wondering if there might be room to revisit the score, given that we feel our paper right now is much stronger than the paper we originally submitted. Ultimately we feel like we have shown enough results to scientifically prove our overall conclusion. If you have further questions or concerns that you feel is a major blocker for acceptance, please let us know.

Responses (cont.)

Additional evaluations (Weaknesses 2 and 3):

One concern is that the chosen VQA benchmark is a bit restrictive. The authors should chose a larger set of evaluation benchmark such as MMMU, OCR-Bench to name a few (Qwen2-VL has a good set of possible evaluation benchmarks [Ref1]). Related to previous point, the authors should also include some caption/description as well as OCR heavy datasets as well.

This is a good point – we agree that using only VQA is a bit restrictive.

1. Our evaluations also include other metrics than general VQA included in our aggregate metric (which we called “stable_score”). We considered a variety of tasks such as TextVQA (document understanding and OCR), RefCOCO (object localization) and OCID-Ref (cluttered object localization). Our evaluation contains at least one benchmark from each of the categories listed in the Qwen2-VL paper (except for video/math/multilingual). See Appendix E for a detailed list of our tasks.

2. Nonetheless, we agree that using a larger set of benchmarks would be better. To fix this, we add our scores on MMMU below. We are evaluating on other datasets and will add them over the next few days as soon as the results come in.

The results shown are for our model initialized from the different text pretraining checkpoints, then trained on image text with a mix of 10% image captioning, 90% text, then finetuned for 4 epochs. The compared models, PaliGemma, Qwen-VL have public results for MMMU (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) we recomputed them to make sure our evaluation pipeline does not make a difference and computed the results in the same way for LLaVA-Next-Mistral. Surprisingly, and as opposed to our previous results, they do not show the 80% checkpoint to be best but it is still stronger than the 100% checkpoint.

Minor Points:

The provided methodology is shown on 1B models but not on larger models such as 7B and 70B. While in general it is expected the findings will carry forward, an explicit experiment would be better. However, it might be prohibitively expensive.

We are currently running a set of experiments at the 7B scale. However, there’s a chance these experiments may not finish in time for this discussion period. We wanted to get this response out first so that we could begin author-reviewer discussions, but we will share the 7B results as soon as we get them.

One issue with the guidelines is that these can be only applied at training time by the authors of the original work. Thus, if someone else wants to improve an existing VLM, it requires the original authors to have open-sourced their checkpoints including the optimizers weights and schedule which may be difficult to obtain.

We agree that requiring partially trained checkpoints may limit immediate applicability of this paper’s findings. However, we hope this work will encourage open-model contributors to release more than just their final checkpoints in the future. Intermediate checkpoints are likely stored by model developers and could be released if there is sufficient demand, furthering open research. Publishing papers such as ours which shows the value of intermediate checkpoints can help push the field towards this direction.

Moreover, while our findings are directly relevant to labs with access to internal checkpoints, they also provide broader insights into the training dynamics of language models, contributing to the field’s understanding. Importantly, our experiments at smaller scales demonstrate that these insights are applicable even to those working on smaller models, ensuring accessibility beyond frontier labs.

Is there a particular reason for choosing OpenLM instead of other models such as LLaMa and Qwen adjacent models (those in Table 1)?

This is connected to the point above – we were able to get granted access to the OpenLM (DataComp-LM) checkpoints, whereas the checkpoints for LLaMa and Qwen are not publicly accessible. Also, DataComp-LM was the strongest model of its size when we started this work and is still competitive (stronger than Llama3-1B on many benchmarks).

Thank you for recognizing the empirical depth of our work and the practical value of our findings. We're glad you found our extensive experimentation (300+ runs) and ablation studies to be well-designed and sufficient to support our conclusions. Your acknowledgment of the significant improvements (~2% on VQA benchmarks) validates the practical impact of our training guidelines.

Main Response:

Qualitative analysis (Weakness 1):

The authors should provide some qualitative analysis, of where the model scoring lower misses compared to other models. Currently, it is not directly clear what kind of questions the 2% drop come from.

We apologize for the confusion, this 2% drop from the text baseline was a mistake that will be removed from the paper. The drop in performance from the initial 80% trained model is 10% and from the 100% trained model it is 14%.

To provide further analysis, we detail below the result per benchmark at different stages of our training process. Most text benchmarks see a decreasing score as the model is trained on images and then fine-tuned for visual question answering. Interestingly, AGIEval score actually increases with image training while PubMedQA sees a sharp decrease of the score.

To help provide a qualitative assessment of our model performance, we also produce some data samples from VQAv2 with different question types https://zenodo.org/records/14201746.

Thank you for your follow-up question. The percentages in the MMMU results shown above are not mixtures but the completion of the initial text-only LLM. The rest of the training pipeline remains unchanged: image-text training (mixture of 10% image captioning and 90% text) and LLaVA finetuning.

Here is the list of tasks that show better results with the 80% initial checkpoint: refCOCO, OCID-ref, POPE, textVQA, GQA, VQAv2.

VizWiz shows better results at 100% but with high variance. The model struggles to understand its prompt and often fails to answer appropriately.

MMMU shows better results at 40%. We did not have the time to estimate variance or look at the reason for this surprising result.

Hi, good news! We have some updates to share:

7B experiments

Our 7B experiments have finally finished, and we are happy to share that our method indeed scales well to the 7B scale. This confirms the main finding of our paper, which is that pre-training with a mixture of image and text data allows models to perform better on downstream tasks.

More specifically, we compared two 7B models as follows:

1. Introduce images (at a 90:10 text:image ratio) ~50% of the way through pre-training for a DCLM-7B checkpoint (2.3T tokens at a 4.3T token schedule). This is done at 1x Chinchilla tokens (~140B tokens). Then fine-tune with Llava for 4 epochs.
2. Take the final cooled-down DCLM-7B checkpoint (2.3T tokens cooled-down for 270B more tokens). For a fair token-matched comparison, we continue training with just text at the same token-matched scale as the model above (i.e. 1x Chinchilla tokens). Then fine-tune with Llava for 4 epochs.

Here, model 1 is basically what we're proposing, and model 2 is very similar to how many VLMs are conventionally trained.

Our results are as follows:

We hope that this gives you a bit more confidence in the validity and scalability of our results. If time permits, we will add even more 7B results to the final version.

More holistic evaluations

From the suggestions of multiple reviewers, we have also added new evaluations and made it clearer in our paper that we are not only evaluating on VQA.

Specifically, aside from the standard VQA tasks, our suite of benchmarks includes:

• RefCOCO (object localization)
• OCID-Ref (cluttered object localization)
• (NEW) MMMU
• (NEW) SNLI-VE Visual entailment

With these, we recompute our stable-scores for Section 3.1 and arrive at very similar overall trends:

While some individual benchmark-level results may vary (e.g. MMMU), the overall trends remain the same after averaging over the entire suite of benchmarks. More importantly, this once again validates our claim that introducing images into pre-training outperforms the 100%-checkpoint results.

..

We hope you can reconsider your score in light of these new promising results, and please don't hesitate to let us know if you have any other concerns.