Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

---

Comment#1 ： The processing of both low-resolution and high-resolution images in the paper is mainly square-based, such as 448x448 and 1024x1024. Is there any adaptation mechanism for handling images with different aspect ratios? Would processing high-resolution images in a way that matches the input image's aspect ratio lead to better performance?

Response: We appreciate for this professional comment. In practice, we have already preserved the aspect ratio of the image and padded the short sides with zeros. Nevertheless, your advice also inspires us to combine MRA with existing dynamic high-resolution methods [A]. By doing so, MRA not only achieves adaptation to arbitrary aspect ratios, but also further increases the resolution to 3k. As you expected, the model performance is further improved on TVQA and PoPE.

[A] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a

---

Comment#2 ：For high-resolution image inputs, we are more focused on improvements in OCR-related tasks. The results for OCRVQA in Table 5 don’t seem to be the best. Additionally, Table 6 only presents results for LLaVA-HR+, but it lacks results for LLaVA-HR-7B, LLaVA-HR-13B, and LLaVA-HR-X with less training data. It would be helpful to include these results to better illustrate the impact of MRA on OCR-related tasks.

Response: Thanks for your careful review. In Table 5, Qwen-VL uses much more OCR-related data than LLaVA-HR, so it performs slightly better on OCRVQA, i.e., +1.5%. For this reason, we make a more fair comparison in Tab 6, where LLaVA-HR uses similar or fewer data than existing methods. In Tab 6, even with less model size and training data, LLaVA-HR still outperforms existing methods like DocOwl-1.5-Chat on all OCR-related tasks.

Moreover, we also fully agree that the LLaVA-HR with less data should also be compared with existing methods on these OCR-related data. Thus, we provide the apple-to-apple comparison between LLaVA-1.5 and LLaVA-HR in the table below, where the training data and the LLM are kept the same. From this table, the benefit of MRA can still be observed on all OCR-related tasks. We will update these results in our final revision.

---

Comment#3：Could the authors further explain why the MR-Adapter is inserted in the last 3 stages? What is the design principle behind this decision? Could it be inserted in the earlier stages instead?

Response: Great question! We think that MR-Adapter should not be inserted in earlier stages for two reasons:

1. The early stages of ViT usually aim to encode low-level visual information, which is inefficient for grasping high-level semantic and fine-grained information from the features of ConvneXt.
2. Since the early stage of ViT has not yet extracted high-level semantics, the early fusion of ConvneXt may hurt the original feature semantics of ViT.

In Tab 3, we have already conducted detailed ablations to validate the insert position of MR-Adapter, which also confirms that the last 3 stages is the optimal choice.

---

---

Comment#1 ：As demonstrated in Table 1, it seems that there is no significant gap between ‘Avg. Pooling’ and the proposed MRA for the VQAv2 task, which is perplexing. The paper should explain the experimental phenomenon.

Response: Thanks for this comment. We would like to explain this from two aspects:

1. In a fair comparison setting, performance gains of MRA are indeed noticeable on VQAv2, i.e., +1.3% over ‘Avg. Pooling’. In practice, improving VQAv2 performance to above 80 is quit challenging. To the best of our knowledge, PaLI-X (55B) achieves state-of-the-art VQA performance of 86.0, which is only 3.7% higher than our much smaller model (LLaVA-HR-X, 14B).
2. Most images of VQAv2 are low- and middle-resolution ones, so the high-resolution benefit of MRA cannot be fully reflected in VQAv2. This is also evidenced by the larger gains of MRA in more fine-grained benchmarks such as 7.5% on TextVQA.

Following your suggestion, we will add these discussions in our final version.

---

Comment#2 ： The paper should carry out a qualitative experiment between the proposed MRA and the model variant in Table 2.

Response: Thanks for this constructive suggestion. We fully agree your advice and have provided several visualization examples in our appendix. We believe that these comparisons do contribute to the understanding of our paper.

---

Comment#3： The paper fails to clarify the version of LLaVA-1.5 used in Figure 4.

Response: Thank you for your careful review. We apologize for the missing model details in Figure 4. In fact, we use LLaVA-1.5-13B from the official checkpoint for comparison, and its LLM is the same as LLaVA-HR-X. According to your advice, we will add more details in our final version.

---

---

Comment#1: LImited performance imprvement: The performance gains with MRA are modest. The low-resolution branch operates at 448×448, so the appropriate baseline is LLaVA-1.5 with 448-pixel resizing. Compared to this baseline, the improvements MRA achieves are minimal (e.g., +0.7 on VQA v2, +31 on MME, and +0.8 on POPE). Training cost and inference speed are also similar between MRA and LLaVA-1.5-448, reducing the practical benefit.

Response: Thanks for this comment. We fully agree that LLaVA-1.5-448 should be a suitable baseline for LLaVA-HR-1024. As you said, with similar costs, LLaVA-HR can already achieve varying degrees of gain on low-resolution or medium-resolution benchmarks such as VQAv2 and MME. But as a high-resolution method, more gains of LLaVA-HR should be observed on high-resolution benchmarks such as TextVQA and DocVQA. To help you better understand our contribution, we provide an apple-to-apple comparison on these benchmarks in the table below, which shows the clear gains of LLaVA-HR-1024 over LLaVA-1.5-448.

---

Comment#2: Limited novelty: the dual-pathway, high-and-low-resolution approach isn’t particularly new. Similar strategies have been explored in other works, such as Mini-Gemini and CogAgent, yet the authors do not compare their method with these models. Explicitly differentiating MRA from these approaches would help clarify its unique contributions.

Response: Thanks for this kindly suggestion. We would like to recognize that Mini-Gemini, as the concurrent work to LLaVA-HR, do have the similar idea in dual visual pathways. However, in terms of micro designs, LLaVA-HR is quit different and efficient against Mini-Gemini. From the comparison in Tab 4, we can see that LLaVA-HR with 1,024 visual tokens can outperform MiniGemini with 2880 visual tokens on 5 of 6 benchmarks.

Compared to CogAgent, LLaVA-HR still establishes advantage in simplicity and efficiency. For example, the high-resolution cross-module of CogAgent requires a large amount of data for pre-training, while our MRA does not. To further validate the benefit of LLaVA-HR, we would like to provide a relatively fair comparison on high-resolution benchmarks in the table below, where CogAgent uses much more training data. From this table, we also see the better performance of LLaVA-HR than CogAgent on 3 of 4 benchmarks.

Your advice is highly beneficial to our paper, and all comparisons will be added in our final version.

---

Comment#3: Limited generalizability: the authors apply MRA solely to LLaVA-1.5. Expanding the evaluation to other MLLMs, like Qwen-VL, would strengthen claims of the method’s generalizability across architectures.

Response: We fully respect to your concerns regarding the generalizability. However, LLaVA-based architecture has almost become the mainstream paradigm of existing MLLMs, thus LLaVA-1.5 may be the most representative and generalizable baseline. Based on your concerns, we have tried to combine MRA with LLaVA-NeXT, another reprehensive MLLM Architecture with dynamic high-resolution strategy. By adopting MRA to each dynamic patch for feature extraction, we observe additional gains on TVQA and PoPE.

---

Comment#4: Clarification on Visual Encoder Notation: In line 206, it states that $\mathcal{F_{I}}l$ and $\mathcal{F_{I}}h$ are visual encoders for high- and low-resolution images, which seems to be a typo. The correct notation should reflect that $\mathcal{F_{I}}h$ and $\mathcal{F_{I}}l$ correspond specifically to low- and high-resolution encoders, respectively.

Response: Thanks for your careful review and we will revise these typos in our final version. In addition to your mentioned typos, we will carefully revise our paper to improve the readability.

---

---

Comment#4：If MR-Adapter could integrate a wider variety of low- and high- resolution visual encoders, its contribution would be significantly enhanced.

Response: Thanks for this advice. We focus on high-resolution adaptation for MLLMs, thus two visual pathways can already efficiently achieve our target. However, MR-Adapter can also be directly extended to more visual encoders. To validate this, we conduct a toy experiment in the table below, which further fuses features of SigLip into the CLIP-ViT ones. From the table, experimental results also confirm the generalization ability of MR-Adapter.

---

Comment#5 ：In the Mixture-of-Resolution Adapter, the authors choose the addition operation to fuse features of different resolutions. (Deformable) Cross Attention is also an option. I wonder which method is better?

Response: Yes, cross attention is a viable choice for feature mixing. However, compared with MR-Adapter, cross attention requires longer training steps to converge and incurs more computational cost. Therefore, we adopt the simple yet effective design for MR-Adapter.

---

---

Comment#1： In Table 1, the MRA is compared to other high-resolution adaptation methods that use a single visual pathway. However, the introduction of a new visual encoder in the MRA raises concerns about the fairness of this comparison. Could the authors provide a baseline that uses dual visual pathways without the MR-Adapter?

Response: Thanks for this suggestion. We fully respect your concerns and think that our dual-pathway designs including the MR-Adapter and the multi-resolution pathway (rather than the additional encoder) play crucial roles in LLaVA-HR. Therefore, we conduct additional ablations in the table below, which shows that the gain of the additional visual encoder is minor if our designs are not used. We hope these comparisons can further eliminate your confusion.

---

Comment#2： The analyses of the MRA’s architecture and design details are insufficient, particularly regarding $\mathcal{F}_l$, $\mathcal{F}_h$ and the gate function. Could the authors provide ablation studies on these components?

Response: Thanks for your detailed review. As discussed above, our main focus and contribution are the macro designs of MRA (the MR-Adapter and the multi-resolution pathway) , whose motivations and ablations are detailed in Sec 4.2-4.3 and Tab 1-2, respectively. As for the micro design of MRA, we aim to explore its optimal choice through empirical studies, and part results are already listed in Tab 3 (including the impact of $\mathcal{F}_l$, $\mathcal{F}_h$, fusion direction and insert position).

To further address your concerns, we provide more ablations of the gate function in the table below. As shown in the table, their significance is far from the macro design of MRA, so we may lack detailed discussions due to page limitations. Based on your suggestion, we will add these results in our final version.

---

Comment#3：The main novelty of the paper appears to be the Mixture-of-Resolution Adapter. While the application of dual visual pathways for high-resolution adaptation in MLLMs is innovative, the overall contribution of the paper seems somewhat insufficient.

Response: Thanks for this comment. To help you better understand our innovations, we would like to highlight the contribution of our dual-pathway design from two aspects.

1. Design principle. As discussed in Comment#1, directly combing two visual pathways does not lead to obvious performance gains in MLLMs. Therefore, the main advantage of the dual-pathway comes from our design principle, which fully considers the visual complementarity of different encoders from the perspective of functionality and alignment, as described in Sec 4.2.
2. Technical details. Previous works we compared in Table 4 (such as Sphinx) also mix multiple visual features, but their motivations and technical details are quit different. For example, Sphinx mixes visual embeddings for better representation, but still requires a dynamic high-resolution method for high-resolution encoding. In contrast, we unify feature mixing and resolution enhancement into one dual-pathway design, greatly improving the efficiency.

Overall, we believe that the design principle and technical details of MRA will provide good hints for future work.

---

---

Comment#1： In section4.3 (line 258), the statement, global average pooling is confusion, is the features are pooled into 1 global token? If so, it seems to be not consistent with figures. Please clarify the exact dimensions of fv after global average pooling.

Response: We feel sorry for any confusion of Fig 3, which does not detail the pooling operation. Actually, global average pooling is only used to compute the gating vector $g\in d$. In this process, the concatenated visual features from dual pathways are globally pooled to a vector $f_v \in \mathbb{R}^{2d}$, and produce the gating vector via Eq. 4. We will add more details in Fig 3 to improve its readability.

---

Comment#2： In Table 1, resizing LLaVA-1.5 to 672 pix achieves close performance with 768pix version of LLaVA-HR, is there a direct comparison between 768-pix version of them?

Response: Yes, we are glad to provide the 756-pix version of LLaVA-1.5 for comparison. Note that 768 pixels does not evenly divide the stride of ViT ($14 \times 14$), so we choose 756 pixels for better performance.

As shown in the table, a higher resolution of LLaVA-1.5 does not necessarily lead to higher performance, but it does incur a greater computational overhead. In stark comparison, LLaVA-HR achieves 2$\times$ faster inference speed and higher performance. We hope that these results can better help you understand our contribution.

---

Comment#3： In table 2, there is an ablation of "tune vision" referring to finetune vision encoder. However, I think the vision encoder in LLaVA-1.5 is fixed, can you provide a detailed description about this. For example, implementation and aim of tuning vision encoder.

Response: Thanks for your detailed review. Actually, fine-tuning the vision encoder can usually improve performance, especially as image resolution increases. In this case, all of baselines in ablations (including LLaVA-1.5) adopt "tune vision" as the default setting, thus providing a fair and strong comparison with LLaVA-HR.

To further address your concerns, we provide the ablation of "tune vision" for LLaVA-1.5 in the table below, which shows that stronger baseline performance can be obtained via "tune vision".

---

Comment#4： LLaVA-HR is proposed to process input resolution of 1024, what if input images larger than 1024. Is there any extended experiments for even larger images such as 4K ones.

Response: Thanks for this professional comment. Based on your suggestion, we consider two ways to further improve resolution for LLaVA-HR:

• Resize: Directly resizing image to a larger resolution.
• DyRes: Combining with the dynamic high resolution from LLaVA-NeXT, and adopting our dual-pathway design to each dynamic patch.

As shown in the table below, the resolution of 1024 can already achieve promising results for most multimodal tasks. Moreover, LLaVA-HR can be seamlessly combined with the dynamic high-resolution strategy of LLaVA-NeXT to further boost performance on OCR-related tasks, i.e., +3% on TextVQA.

---