Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight

Q1. Ruling out the impact of the scale of training data.

Thank you for pointing this out. We agree that the scale of training data is one of the most crucial contributing factor in the downstream performance of MLLMs.

However, in our humble opinion, it is hard to single out the impact of the scale of training data on the downstream performances, since the scale of data is entangled with many design choices and hyper-parameter settings in training large models, such as the data composition during supervised fine-tuning, model architecture (the visual backbone and the language model), training strategy (freezing or training the visual backbone and the language model), etc.

In order to compare with existing connector structures used in modern MLLMs (e.g., Qwen and VILA), we categorize them into two groups:

• linear projection based MLLMs (e.g., LLaVA, VILA, DeepSeek-VL, etc.),
• and resampler based MLLMs (e.g., Qwen and mPLUG-Owl2).

The downstream performances of these two types of connectors are compared with Chain-of-Sight in Table 2 and Table 3, with identical training configurations, including the training data in pre-training and fine-tuning, the visual backbone, the language model, and the training strategy.

Note 1: The number of VQA here is different from the ones in Table 4 as the VQA in Table 4 did not take ScienceQA into account.

Note 2: The average performances of Captioning, VQA, and Text, as well as the overall average performances are calculated based on the results in Table 2.

From the above tables, we can observe that under the same configurations, Chain-of-Sight outperforms both linear projection and resampler based methods.

Q2. Differences in the convergence speed of different compound scaling strategies.

Counter-intuitively, we find different compound scaling strategies lead to a similar convergence speeds during the fine-tuning process, despite the huge differences in the downstream performances. Nevertheless, the model with more visual tokens and higher resolutions generally achieve a lower loss throughout the fine-tuning process.

Q3. Interpolation of positional embeddings whtn increasing the input resolution.

Yes, the positional embeddings in the pre-trained CLIP backbone are interpolated when we increase the input resolution to 448.

Q4. Refering to Table 3 instead of Table 6 in Line 222.

Yes, we meant to refer to Table 3 in Line 222. We will correct this in our revisions.

Q1. Experimental results based on other models.

Thank you for the constructive comment. We provide experimental results based on different language models in the following table, which shows Chain-of-Sight benefits from stronger language models.

Note 1: The number of VQA here is different from the ones in Table 4 as the VQA in Table 4 did not take ScienceQA into account.

Note 2: The average performances of Captioning, VQA, and Text, as well as the overall average performances are calculated based on the results in Table 2.

To validate the effectiveness of Chain-of-Sight, we also performed ablation studies with Llama-3-8B, where various nubmers of tokens are used during supervised fine-tuning. The table below shows that the compound scaling strategy effectively improves the downstream performance. Since the pre-training takes long for resampler-based (336 tokens) and linear-based models (256 tokens), we only validate the effectiveness of the compound scaling here, and leave the baseline results to our revisions.

In addition, we also go through the whole training process with different language models, which includes the stage 1 pre-training, high resolution post pre-training, and multi-task supervised fine-tuning.

We will include results based on other language models and visual backbones in our revisions.

Q2. Explanation on how the numbers of tokens during pre-training and fine-tuning are determined.

Because of the size of a large model and the scale of training data, it would be infeasible to do a grid search on the number of tokens. Therefore, the model hyperparameters including the window sizes and the number of visual tokens are determined under several key rationals.

Pre-training.

In the pre-training stage, we employ the resolution of 224, which gives us the feature of size 16x16.

Window sizes. Since the main objective of Chain-of-Sight is to accelerate pre-training by reducing the number of tokens during pre-training, we limit the number of visual scale hierachies to two, i.e., one global view (window size 16) and one local view (window size 4).

Token counts. Features with larger window sizes are usually more informative w.r.t. image contents, thus requiring more visual tokens to represent. For efficiency, we intuitively selected 16 visual tokens for encoding the global view (instead of 32 as used in BLIP-2 or more). As for the token counts in the local view, we have experimented with 1, 2, and 4 per window, as in Table 2 and 3. Eventually, we use 4 tokens per window for the local view, which strikes a good balance between the training speed and downstream performance.

Supervised fine-tuning.

As mentioned in the manuscript, given a pre-trained Chain-of-Sight model, there are two ways of scaling up the number of tokens, i.e., increasing the resolution and reducing the window sizes, respectively. Both methods can increase the token counts by 4 times for a specific pre-trained window size, producing a 16x token count for each window size, where 80 tokens are scaled up to 1280 tokens. In addition, we keep a copy of 16 global tokens for providing a comprehensive overview of the input image, resulting in a total number of 1296 tokens.

We provide a detailed calculation of the number of tokens in both stages in the table below. Detailed explanation on the choice of model hyperparameters will be included in our revisions.

Q1. The method of hierarchical (multi-scale) visual input has already been explored in many former works, such as LLaVA-NeXT and InternLM-XComposer2-4KHD.

The multi-scale nature is the fundamental characteristic of images manifested by David Marr's pioneering work on vision perception back in 1970s. How to exploit the multi-scale nature effectively has been a key research topic in most vision tasks, if not in all tasks, for half a century, which is likely to continue to be an active research topic in future decades.

Thanks for raising this point and we are happy to discuss the differences in the multi-scale strategy between Chain-of-Sight and the methods used in LLaVA-NeXT and InternLM-XComposer-4KHD, in the following three aspects.

Conceptual idea

Both LLaVA-NeXT and InternLM-XComposer-4KHD process the high-resolution input image in two paths. One splits the high-resolution image into partitions of sub-images with a 'base resolution' supported by the backbone, which is 336 for both methods. The other resizes the high-resolution image to 'base resolution' to be processed by the backbone. The multi-scale idea is exploited on the input side before the visual backbone, which is similar to the multi-resolution/multiview strategy [1,2,3].

Differently, inspired by the pyramid structure in contemporary vision models [4,5,6], our Chain-of-Sight method constructs in-model multi-scale features, where multi-scale visual prompts are generated after the visual backbone based on the features of a single 'base-resolution' image. Hence, our method of leveraging multi-scale hierarchy is in parallel with the multi-view method used in LLaVA-NeXT and InternLM-XComposer-4KHD, and it is possible to combine both methods to achieve even stronger visual capabilities.

Motivation

The motivations of LLaVA-NeXT and InternLM-XComposer-4KHD are to enable MLLMs for capturing more details in the high-resolution images, while Chain-of-Sight is proposed to leverage the flexiblity of our multi-scale visual resampler such that the pre-training could be accelerated with lower nubmer of visual tokens without compromising performance.

Methodological details

In terms of the methodological details, both LLaVA-NeXT and InternLM-XComposer-4KHD leverages a two-level visual hierarchy, i.e., the global and local views. Though Chain-of-Sight splits the input image into global and local windows in the pre-training, it is able to be extended to three or more scale levels during fine-tuning thanks to the compound scaling strategy of Chain-of-Sight.

In addition, we would like to highlight that, as mentioned in Sec 2.3, LLaVA-NeXT and InternLM-XComposer-4KHD can be considered as a special case of our compound scaling, where only the resolution of the input image is scaled up, while the window size is kept the same.

More details will be provided in our revisions.

[1] Karpathy, Andrej, et al. "Large-scale video classification with convolutional neural networks." In CVPR 2014.

[2] Yan, Shen, et al. "Multiview transformers for video recognition." In CVPR 2022.

[3] Feichtenhofer, Christoph, et al. "Slowfast networks for video recognition." In ICCV 2019.

[4] Lin, Tsung-Yi, et al. "Feature pyramid networks for object detection." In CVPR 2017.

[5] Wang, Wenhai, et al. "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions." In ICCV 2021.

[6] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." In ICCV 2021.

Q2. Mistakes in the experiment section.

Thank you. We will re-examine the manuscript and correct the mistakes.

Q3. Why is "CoS, PT80, FT336" stronger than "CoS, PT336, FT336"?

It is indeed the case. Though the performances of the two variants are similar on referring expression comprehension, "CoS, PT336, FT336" underperforms "CoS, PT80, FT336" in almost all the other aspects.

Note 1: The number of VQA here is different from the ones in Table 4 as the VQA in Table 4 did not take ScienceQA into account.

Note 2: The average performances of different tasks and the overall average performances are calculated based on the results in Table 2.

We believe the reason behind this is that the low capacity of the "CoS, PT80" model during pre-training acts as a filtering mechamism for the noisy data, which allows it to learn the more commonly existing distributions in the pre-training data. Specifically, we find the pre-training loss of "CoS, PT336" lower than "CoS, PT80", while the fine-tuning loss of "CoS, PT336, FT336" is notably higher than "CoS, PT80, FT336". Given the higher level of noise in the pre-training data, we believe the higher capacity of "CoS, PT336" model makes it learn more low-quality data than the "CoS, PT80" model, leading to the worse downstream performance of the "CoS, PT336, FT336" model. In fact, similar phenomenon can be observed with resampler-based model. As in Table 2, the average performance of "Resampler, PT80, FT 400" is also 0.6 higher than "Resampler, PT336, FT336".

Q4. The results in Table 5 is different from that in Table 2&3&4.

Yes, the training settings are different between our final model and the models in the ablation studies. The model in Table 5 is trained with an additional post-pre-train high-resolution stage, while for ablations, we skip this stage for efficiency, as mentioned in L184. We will make this clearer in our revision.

Q5. Larger scale model with full-parameter training.

Limited by time, we are unable to finish training large-scale model with full-parameter training. We will include the results in our revisions.