POINTS: Improving Your Vision-language Model with Affordable Strategies

Disscussion about the performance of different models in Table 1

Here are some possible reasons for the fluctuations observed in the benchmarks:

1. Different LLMs were used.
2. Different pre-training data and instruction fine-tuning data were utilized.
3. Different experimental parameters were applied.

However, this does not affect the conclusions of our experiments or the effectiveness of our method, because during the experiments, the comparisons between our current model and the baseline were conducted with strictly controlled experimental parameters and configurations.

Why choose Yi-1.5-9B-Chat

1. The main reason for using Yi-1.5-9B-Chat is that our training framework is more compatible with models configured with the LLaMA model, and at that time, we also considered overall performance, so we chose Yi-1.5-9B-Chat.
2. After submitting the paper, we continued to conduct experiments using Qwen2-7B, and the results are as follows.

GPT-4o data introduces knowledge distillation

1. The data generated using GPT-4o accounts for less than 1% of the total instruction fine-tuning data.
2. When we removed the data generated by GPT-4o, the model's performance only decreased from 61.8 to 61.7. Thus, the introduction of GPT-4o data does not introduce knowledge distillation

Discussison about the different configuration of different methods

We compare from several aspects: pre-training dataset size, instruction tuning dataset size, LLM, vision encoder, and dynamic resolution.

What does it mean to not have IS?

"Not using IS" means that we entirely used the instruction fine-tuning data from [1] and did not use Individual Select to introduce any additional data.

[1] Rethinking Overlooked Aspects in Vision-Language Models

We sincerely appreciate your great efforts in reviewing this paper. Your constructive advice and valuable comments really help improve our paper. Considering the approaching deadline, please, let us know if you have follow-up concerns. We sincerely hope you can consider our reply in your assessment, and we can further address unclear explanations and remaining concerns if any.

Once more, we are appreciated for the time and effort you've dedicated to our paper.

Motivation is unclear

Our motivation is mainly divided into three points:

1. The leading models in the industry generally do not disclose their training details.
2. There has been no in-depth exploration of how pre-training data is processed in current open-source work.
3. During the instruction fine-tuning phase, most current work focuses on continuously adding new data, which becomes difficult to improve upon in the later stages.

In response to the above motivations, we further propose:

1. Building a stronger baseline for subsequent experiments: A stronger baseline often determines the final performance of the model. Many previous works have conducted experiments based on LLaVA, but with the development over the past year, many new improvements have emerged in this field, such as DHR and CapFusion. Using existing methods, we have obtained a stronger baseline based on LLaVA.
2. For processing pre-training data, we propose using perplexity to filter the data, thereby achieving better performance with less data.
3. During the instruction fine-tuning phase, to address the bottleneck encountered with newly added datasets, we propose using model soup with models trained on different data, which can further enhance the model's performance.

Why highlight OCR-related tasks?

1. OCR is a very important direction for MLLM applications. Many previous works have focused on enhancing the OCR performance of multimodal models, such as Vary[1]. However, enhancing OCR capability is not our primary goal. In our entire work, only the Dual Vision Encoder and CATTY are directly related to OCR. Our main objective is to improve the overall capability of the model, with enhancing OCR-related abilities being just one aspect.
2. LLaVA-Next's capability in OCR is still quite limited, as can be seen from its performance on OCRBench. LLaVA-Next scores 53.1 (according to OpenCompass results), while our model scores 70.6. In terms of approach, LLaVA: 1) did not significantly enhance the OCR capability of the vision encoder, and 2) applied padding to images in the use of DHR, which might affect the model's learning. To address these issues: 1) we introduced an additional OCR-enhanced vision encoder, and 2) we proposed CATTY, which segments images without padding and while maintaining the aspect ratio.

No mention of computation overhead

In Appendix I, we explain our training costs

How does POINTS perform when the downstream test tasks have very few training samples?

Additionally, we further tested on the recently released MEGABench, which covers over 500 real-world tasks, most of which do not have related samples in the training set. The specific results are as follows:

How does the inference time of CATTY change as the image size increases?

CATTY and DHR have similar scaling curves for image size versus inference time.

$$\mathrm{I n f e r e n c e T i m e} \propto \frac { H } { w } \times \frac { W } { w } \times w ^ { 4 } = H W w ^ { 2 }$$

Here, H and W are the height and width of the image, and w is the window size.

Lines 75-78 misses citation

This is the phenomenon we observed in our experiments. Please refer to the second row of Table 4 for the experiment.

[1] Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models

Why not structure the paper to be centered around OCR?

1. Thank you for your suggestion. But as we mentioned earlier, our main goal is to achieve performance comparable to industry-leading models with fewer resources. Our focus is on the general performance of the model, and OCR is just one aspect of it. In this work, most of our efforts are directed towards general performance, not just OCR. For example, we use perplexity to filter pre-training data and employ model soup.
2. Affordable means that all the methods adopted in this paper are quite lightweight, allowing us to achieve a model comparable to the state-of-the-art (SOTA) with fewer resources. For instance, our pre-training data consists of only 1 million samples, and using 32 x H800 GPUs, we can complete all model training within 10 hours (please refer to Appendix I). This approach is a good option for teams with limited resources.

Wall clock time of CATTY with varying image sizes?

We chose the following experimental configuration to test the wall clock time of CATTY and DHR:

Device: 1 x 80G H800
Model: POINTS-Qwen2-7B
Inference framework: HuggingFace transformers
Time calculation method: Measure the forward time for 10 images using teacher forcing mode.

The images we use are square-shaped, and the content fed into the language model includes an image and its caption. To highlight the impact of image sequence length on wall clock time, we have chosen relatively short captions (10 tokens).

Potential limitations of model soup

1. Currently, we have observed that although there are significant gains in the early stages, it becomes increasingly difficult to improve the model's performance as the process continues. This phenomenon has also been noted in a recent study[1]. The main reason is that as model soup progresses, the similarity between models increases, making it challenging for the model to break through the original local optimum to reach a better local optimum.
2. We believe that the distribution differences in the dataset do not have a significant negative impact on model soup; instead, they may actually provide benefits. For example, as shown in Figure 3 of the original paper, performing model soup with a model trained on the Base Set + MathV360K and a model trained on the Base Set + ScreenQA can significantly improve the model's performance. MathV360K is a dataset related to mathematics, while ScreenQA is a dataset related to screen-based question answering. There are significant differences between the two datasets in terms of both images and questions.
3. Model soup do not show overfit to a specific benchmark. We conducted model soup primarily based on the Overall Score metric in Table 1. However, in the final presentation of results, we also provided metrics for MME, RealWorldQA, and LLaVA-Wild, where we achieved good performance as well. Additionally, we further tested on the recently released MEGABench and obtained impressive results. The specific results are shown below.

Risks of reduced generalization ability due to a smaller pre-training dataset size

1. When selecting model data, we primarily referred to the Overall Score metric in Table 1. However, in the final presentation of results, we also provided metrics for MME, RealWorldQA, and LLaVA-Wild, on which we achieved good performance as well.
2. Additionally, we further tested on the recently released MEGABench[2], which covers over 500 real-world tasks. The specific results are as follows: | Method | Overall SI | | --- | --- | | InternVL2-8B | 29.21 | | POINTS-Qwen2-7B | 26.10 | | MiniCPM-V2.6-8B | 25.95 | | LLava-OneVision-7B | 25.69 | | Idefics3-8B-LLaMA3 | 12.06 | To maintain consistency with previous works, such as LLaVA-OneVision, we have newly added the POINTS-Qwen2-7B model, which uses Qwen2-7B as the LLM.
3. The current pre-training of MLLMs is essentially a process of aligning language and image signals. The learning of knowledge has already been accomplished separately during the training of the Vision Encoder and the LLM. Therefore, using less data during the alignment phase does not significantly impact the model's ability to acquire knowledge.

In summary, the reduction in pre-training data has not weakened the generalization capability of POINTS.

Have any areas been observed where CATTY is not as good as DHR?

1. In our response to the Reviewer eeM9, we mentioned that CATTY introduces additional computational overhead compared to DHR, but this overhead is negligible relative to the total computational cost.
2. As shown in Table 1, the main advantage of CATTY over DHR is its improved performance on OCR-related tasks, such as OCRBench. This is because OCR tasks are more sensitive to image pixels, a point that has been noted in previous work[3]. However, for some other tasks, the performance of CATTY is comparable to, or even slightly lower than, that of DHR, such as in MMVet. But compared to DHR, CATTY will bring overall improvement to the model.

[1] Exploring Model Kinship for Merging Large Language Models

[2] MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks

[3] Improved Baselines with Visual Instruction Tuning

We sincerely appreciate your great efforts in reviewing this paper. Your constructive advice and valuable comments really help improve our paper. Considering the approaching deadline, please, let us know if you have follow-up concerns. We sincerely hope you can consider our reply in your assessment, and we can further address unclear explanations and remaining concerns if any.

Once more, we are appreciated for the time and effort you've dedicated to our paper.

Missing analysis about CATTY

1. For the 1M pre-training data used, we calculated the number of tokens obtained using CATTY and DHR respectively. Compared to DHR, CATTY introduced approximately 10% more original vision tokens. This newly introduced number of tokens accounts for about 2% of the total number of tokens, so it essentially does not introduce significant computational overhead.
2. Regarding redundant information, we conducted statistics on the 1M pre-training data and found that the average overlap between every two adjacent windows is 10 pixels. Intuitively, we can infer that redundant information might affect the model's counting ability. Therefore, we selected two specific metrics: Object Localization from MMBench (which includes counting problems) and Count from MME. The performance of CATTY and DHR is as follows: | Method | Object Localization (MMBench) | Count (MME) | |-------|-------------------------------|-------------| | DHR | 77.1 | 168.3 | | CATTY | 77.2 | 167.8 |
3. Since CATTY introduced an additional 10% of vision tokens, and previous work, such as Idefics2[1], has shown that additional vision tokens can enhance model performance, we further enlarged the width and height of the images by a factor of 1.05 (sqrt(1.1)) before using DHR. We denote this set of experiments as DHR(large). The specific experimental results are as follows: | Method | Overall score | | --- | --- | | CATTY | 53.4 | | DHR | 52.6 | | DHR (large) | 52.4 |

How the baseline in Table 4 is obtained

The baseline in Table 4 is derived from the last row of Table 1, which utilizes the Base Set discussed in the Individual Selection section of 3.1. Further details about the Base Set are provided in Table 6 (marked in black) in the appendix.

VLM used in pre-training dataset filtering

Currently, the model we use for PPL filtering is the one obtained from the last experiment in Table 1. Additionally, we used another VLM (InternVL2-8B) to filter the data (as detailed in Appendix F). Pre-training on the data filtered by InternVL2-8B resulted in an overall score of 60.1.

[1] What matters when building vision-language models?

We sincerely appreciate your great efforts in reviewing this paper. Your constructive advice and valuable comments really help improve our paper. Considering the approaching deadline, please, let us know if you have follow-up concerns. We sincerely hope you can consider our reply in your assessment, and we can further address unclear explanations and remaining concerns if any.

Once more, we are appreciated for the time and effort you've dedicated to our paper.