InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Q1: The paper frequently...

A1: Thanks for your valuable comments.

1. Enabling LVLM to understand high-resolution images remains a challenging and open problem in the field. The primary goal of our paper is to explore a general and effective strategy for high-resolution image understanding LVLMs, and further study how the image resolution influences LVLM performance.
2. Efficiency is an important under-explored topic, but in most cases, there is a trade-off between performance and efficiency. In this work, we focus on pushing the performance upper bound without introducing additional variables, and we would leave the study on efficiency in future work.
3. Our proposed weighted multi-dataloader sampling strategy (Please see Appendix B.1) is designed to optimize the training process. By enabling each GPU to process batches with either high-resolution images only or low-resolution images only, we reduce the need for padding tokens to handle unbalanced sample lengths caused by varying image sizes. This approach speeds up the training process, making it more efficient and scalable

Q2: lack of analysis regarding the inference efficiency...

A2: The model inference process consists of two stages: encoding the prefix (model input) and autoregressively decoding new tokens (model output). Correspondingly, the inference efficiency considers two parts: time to encode the prefix and speed to decode each token. Here we report the prefix encoding time and per-token decoding speed under different HD settings. We test the speed with a Nvidia-A100 80G. With the results below, we have three observations

1. Prefix encoding time increases linearly with the number of prefix tokens.
2. Decoding speed remains relatively constant, regardless of prefix length, thanks to optimizations on transformers from research communities and companies, including kv-cache and flash-attention.
3. When generating 2048 tokens, total inference time usage is nearly identical across HD9 to HD55, as encoding time is much smaller.

Based on the above analysis, we believe the inference efficiency of our model is acceptable

We believe some targeted designs can further improve efficiency, but as we discussed in A1, our paper focuses on enabling LVLM to understand high-resolution images with a general and effective solution, and we would leave the efficiency exploration in future work.

Q3: The introduction repetitively emphasizes......

A3: Sorry for the confusion.

1. Regarding the variability in patch layouts, we illustrate this challenge with Fig.2 in the rebuttal PDF. Images a and b are distinct, yet when flattened into a one-dimensional input sequence, they become identical inputs (Fig. 2(c)) for the LLM. In practice, images also exhibit diverse aspect ratios, which can lead to confusion in object positioning. To address this, it is crucial for LVLM to accurately comprehend the 2D structure. We propose two core designs to achieve this: Global-Local Format and Patch Layout Indicator. The global image provides a concise overview, while the indicator precisely conveys the 2D structure to the model.

2. For 'Even when' in Line 56, this is an inference drawn from observations in Fig. 4 (model performance is not saturated even with HD55) and Table 6 (model performance improves with higher inference resolution) in our paper. We provide quantitative results on a challenging subset of InfoVQA to support this. Specifically, we selected images with resolutions exceeding 1680x3696 (HD55) pixels from the InfoVQA validation set, resulting in a 15.6% subset. We evaluated IXC2-4KHD's performance under HD55 and HD65 settings. The results, presented below, demonstrate that even though the model is trained with the 4KHD setting, it still benefits from larger inference resolutions when applied to images larger than the training cases. InfoVQA HD55 subset | 55 | 65 -|-|- HD55| 70.41 | 71.64

3. For the 'methods trained at 4K resolution might fail to gain additional performance at lower resolutions,' problem, we observe that when comparing MME (at most 512x512) and MMBench (at most 512x512) results between HD9 and HD25 in Fig. 4, the performance is nearly identical. This suggests that model performance saturates when the training resolution exceeds the image's original resolution, and increasing the training resolution yields no further gain. In contrast, when the training resolution is smaller than the image's original resolution, model performance continues to improve with increasing training resolution, as seen in DocVQA (at most 5367×7184) and InfoVQA (at most 2921×24978) in Fig. 4. These observations underscore the necessity of our goal: exploring a general and effective strategy to enable LVLM to understand images across a broad range of resolutions, from 336 pixels to 4KHD.

Q4: It prompts inquiries....

A4: We appreciate your insightful suggestion and have investigated this issue using the college-level benchmark MMMU. Specifically, we curated a subset of high-resolution images (with resolutions exceeding 1000x1000 pixels) from the MMMU validation set, which accounts for approximately 10.5% of the dataset. We then compared the performance of IXC2 and IXC2-4KHD on this subset. The results demonstrate a notable improvement in performance by IXC2-4KHD, indicating its capability to not only handle dense OCR tasks on large images but also tackle complex, college-level tasks that require high-resolution input. Additionally, we have included two exemplary cases in the rebuttal PDF Fig.3, showcasing our model's ability to solve intricate reasoning problems based on the visual information provided.

Q1: Limited novelty...

A1: Thanks, we would like to highlight the novelty of our work in the following aspects:

1. As discussed in the related work (Lines 95-101), our approach is the first to address the challenges and propose solutions for handling variability in image feature patch layouts, enabling effective training with dynamic high-resolution images.

2. Our method introduces a dynamic resolution and automatic path configuration strategy, allowing for seamless scaling from 336 pixels to 4K resolution - a range that exceeds previous approaches. In contrast, existing methods for high-resolution understanding are limited to either predefined high-resolution settings or a narrow range of resolutions. For instance, Monkey and LLaVA-Next employ sliding windows and image splitting, respectively, but are restricted to segmenting images into a maximum of 6 and 4 patches, respectively. This limitation hinders their ability to handle images with extreme aspect ratios, commonly found in infographics. Our IXC2-4KHD, on the other hand, supports up to 55 local image patches, making it capable of tackling these challenging cases.

3. Our work also presents a thorough analysis of high-resolution strategy design, examining the impact of global image, 'newline' token, and training/inference resolution - aspects that were underexplored in previous studies. Through extensive experimentation, we have gleaned valuable insights, including:

   • The LVLM struggles to comprehend the 2D structure of a simply flattened image, but incorporating a global image and 'newline' tokens could solve this problem with clear performance improvement.
   • For tasks requiring high-resolution images, our model demonstrates improved performance with larger images, showing no signs of saturation even at 4KHD resolution. This suggests potential for further scaling and improved results.
   • Conversely, for tasks insensitive to resolution, model performance plateaus as image size increases.

4. Finally, we note that InternVL-1.5 is a concurrent, unpublished work that has not undergone peer review or been accepted by any reputable conference or journal. Similarly, LLaVA-Next is a GitHub blog post, rather than a rigorously evaluated academic publication.

Q2: Experimental... More comparisons or ablations.

A2: Thanks for your valuable advice. For a fair comparison, we trained a new model with the identical architecture, training strategy, and dataset as our original model, but with one key modification: we adopted the high-resolution strategy from LLaVA-Next. We name it as IXC-LLaVA and compare it with 1) LLaVA-Next 0530 official results and 2) IXC2-4KHD under HD-9/16 setting. The results below demonstrate that IXC-LLaVA achieves promising performance across all six benchmarks, leveraging the benefits of additional training data and advanced IXC architecture design. However, it is outperformed by IXC2-4KHD HD-9, which utilizes fewer image tokens yet yields better results. This fair comparison underscores the efficiency and effectiveness of our proposed high-resolution strategy, highlighting its advantages over the LLaVA-Next approach.

Q3: Compare with other MLLMs, e.g., InternVL-1.5.

A3: Thank you for your comments.

1. As discussed in A1, InternVL-1.5 is not a peer-reviewed publication that has not been accepted by any conference or journal yet. Moreover, it is a concurrent work that is posted on Arxiv within one month to the NeurIPS submission deadline. Therefore, we do not feel obligated to compare our work with it at this time.
2. In Table 3, we have already reported the performance of InternVL-1.2, which was a previously released version.
3. In Table 4, as mentioned in line 214, we conducted a comprehensive comparison with open-source LVLMs of similar scale. However, InternVL-1.5 has a significantly larger model size (26B parameters) compared to our model (8B parameters), making a direct comparison unfair and meaningless.

Q4: The implementation details are not clear...

A4: Sorry for the confusion. We list the detailed data usage in the following. We use almost 3M data for the pre-training and 1.2M data for the SFT stage.

I appreciate authors' response, which partly addresses my concerns. However, I still think that the novelty and contribution of this paper are still below the acceptance bar of NeurIPS.

Expect for the patch layout, all designs of the paper exist in recent methods. For example, The global-local format and the patch division is proposed in MonKey. The Newline Indicator has been adopted in many works like LLaVA-NeXT and LLaVA-UHD. By the way, one thing that might be unacceptable is that the authors did not even mention these methods in their methods, which might mislead other reviewers.

Most importantly, the authors attribute the high-resolution advantages to their patch layout, which is not rigorous. IXC2-4KHD is able to handle such high-resolution input not because of its patch layout, but because the token merge strategy can reduce the number of tokens so that it can be trained with a maximum token window of 4096. This explain why Monkey and LLaVA-Next are restricted to 4-6 patches. Unfortunately, the token merge strategy is not firstly proposed in the paper, so I still doubt the novelty of the paper.

Based on this concern, I think InternVL-1.5, which also applies the same token merge strategy, is a good comparable method to prove the patch layout design. I also understand that InternVL-1.5 may be a concurrent work to this paper. So, maybe the authors should strictly control the same token merge strategy to prove its patch layout design.

Overall, I am still inclined to reject this paper.

Q1: More training details

A1: Thanks for your valuable comments, here we show the training loss in the Pretrain/SFT stage. Please refer to Figure 1 of the rebuttal pdf.

Q2: Is it reasonable to use so such many visual tokens for ONE image?

A2: This is a great and interesting question, and we address the reasonableness of using more tokens for HD images by considering the following:

1. Can an LLM understand an image with so many tokens? Our paper reveals that the LVLM model can effectively leverage a large number of image tokens. It captures image details from thousands of image tokens, leading to significant gains on various high-resolution benchmarks.
2. Is an image really worth thousands of tokens? The aim of our paper is to develop a universal solution for LVLMs to analyze images with varying resolutions, ranging from 336 pixels to 4K resolution. Further token compression may result in unavoidable information loss and impact our study. As discussed in Figure 4 of the main paper, we observed that more visual tokens lead to consistent performance gains on benchmarks requiring fine-detailed understanding, such as DocVQA and InfographicVQA. This implies the necessity of scaling up the image resolution for a single image. We leave the task of increasing image resolution with efficient image token numbers as a future research direction.

Q3: The influence of LLM parameters

A3: Thanks for your valuable idea! Due to the rebuttal period limitation, there is no sufficient time for us to train a 20B LVLM. Instead, we conduct the experiment on a smaller model InternLM2-1.8B and we found a clear performance improvement with LLM parameter increasing. We believe if we keep increasing the LLM size, the model will get better performance.

Q4: Performance on Video Benchmarks

A4: Thanks for your valuable comments, we tried to concatenate the video frames into a long image and finetuned it with the SFT data used in Video-ChatGPT [1]. Here we report the results on three widely used video benchmarks. As shown below, our model performs on par with GPT-4V on the challenging MME-Video, and significantly better than GPT-4V on the MVBench and MLVU.

[1] Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models