Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG

We would like to thank the reviewer for the constructive comments and suggestions.

Q1: Experiments on widely used benchmarks.

R: We appreciate the reviewer's suggestion. As suggested, we conduct additional experiments on five widely used benchmarks: DocVQA, ChartQA, TextVQA, AI2D, and MMStar. As shown in the table below, incorporating RAP led to performance of 1.8% and 2.1% on LLaVA-v1.5 7B and 13B, respectively. We also observed that RAP brings more notable improvements on higher-resolution images. For example, on DocVQA, which has an average resolution of $1599\times 1241$, RAP improved performance by 4.7% and 2.3% for LLaVA-v1.5 7B and 13B, respectively.

Q2: Experiments with more powerful MLLMs.

R: We thank the reviewer for the constructive advice. To further demonstrate the effectiveness and generalizability of our RAP, we conduct experiments on HR-Bench using several advanced MLLMs, including Oryx-1.5-7B, CogVLM-LLama3-19B, Cambrian-8B, LLaVA-ov-7B and 72B. As shown in the table below, our RAP consistently boosts performance across all models. These results underscore the robustness of our RAP across a wide range of model architectures, highlighting its potential as a universal enhancement for high-resolution image perception. We will include the corresponding results and discussions in the revised version.

Q3: Comparison with native high-res MLLMs.

R: We appreciate the reviewer for the comments. We'd like to clarify that due to limitations in visual encoders and LLMs' long-context handling, current MLLMs struggle with high-resolution (HR) image perception. Existing native high-res MLLMs mainly address this through two strategies:

• Splitting HR images into multiple image crops

  These methods split HR images into smaller crops, encode them separately, and concatenate the features for the LLM. Oryx uses OryxViT to handle arbitrary resolutions, but processing 8K images produces up to 300K visual tokens, heavily taxing the LLM's long-context capacity. In practice, this often led to erroneous outputs, so we downsampled to 2K resolution (as recommended), which unfortunately resulted in loss of important visual details.

• Using HR visual encoder

  Another line of work explores HR visual encoders designed to handle HR images. For instance, ConvNeXt-L supports inputs up to $768\times 768$ resolution. However, such encoders still downsample 8K images to align with pretraining resolutions, limiting their effectiveness on HR images. Even Cambrian, a multi-encoder framework, only supports resolutions up to $384\times 384$, necessitating downsampling for 8K inputs and resulting in significant loss of critical visual information.

Inspired by RAG's success in improving long-context capabilities in LLMs, we adapt it to MLLMs for HR image perception. RAP retrieves and preserves only the most relevant image crops, reducing input resolution while retaining essential information. Our RAP can be integrated into any MLLMs, significantly enhancing their perception of HR images. For example, LLaVA-v1.5 7B with RAP reaches 53.8% accuracy on HR-Bench 8K, outperforming Oryx-1.5-7B (49.5%) and Cambrian (37.9%), validating RAP's effectiveness.

Q4: The practical usefulness of our RAP.

R: We'd like to thank the reviewer for the comments. Here, to robustly demonstrate the practical usefulness of our RAP, we evaluate its impact on both throughput and accuracy. As shown in the table below, LLaVA-ov-0.5B equipped with RAP achieves performance comparable to the much larger LLaVA-ov-72B, yet operates with 2.3 times higher throughput and requires 22 times fewer parameters. These results highlight RAP's potential for efficient, high-performance deployment, particularly in resource-constrained environments such as edge devices.

We truly appreciate the reviewer for the thoughtful comments and suggestions, as well as the positive support.

Q1: The concern about the extra retriever increasing computation.

R: We appreciate the reviewer's comments. Although our RAP includes an additional retriever module, it still delivers a significant performance gain despite the overhead. As shown in the table below, our evaluation on HR-Bench 8K demonstrates that integrating RAP with LLaVA-ov-0.5B, using VisRAG as the retriever, results in a total model size only 3.3 billion parameters. Despite this lightweight configuration, it achieves an impressive 20.7% improvement in accuracy. Notably, LLaVA-ov-0.5B w/ RAP matches the performance of LLaVA-ov-72B while using approximately 22 times fewer parameters, highlighting RAP's efficiency, scalability, and strong potential for high-resolution visual understanding.

Q2: Replace confidence score calculation in RE-Search.

R: We appreciate the reviewer's point. To clarify, the OpenAI API [R1] currently offers a logprobs parameter, which returns the log probabilities of output tokens — an essential feature we leverage in our RE-Search module to compute confidence scores.

Beyond this, we also explore an alternative approach using generation-based confidence scoring. Specifically, we prompt the MLLM to self-assess whether the image provides sufficient information to answer the question. The constructed prompt is as follows:

Question: {}\nCould you answer the question based on the available visual information? Return only a JSON object with a numerical confidence score (0~10) of "Yes" like {"Yes": x}.

We normalize the final score to the range $[0,1]$ for consistency. As shown in the table below, incorporating this generation-based score into RAP resulted in a 7.4% performance gain over the baseline, demonstrating its effectiveness.

The relevant results and discussions will be incorporated into the revised version.

[R1] OpenAI API, https://platform.openai.com/docs/api-reference/completions/create

Q3: What is the fundamental difference between RAP and search-based methods?

R: We thank the reviewer for the constructive comments. We would like to clarify the main differences between our RAP and search-based methods as follows:

• Search-based methods yield high-resolution inputs with multiple crops, while RAP lowers resolution by keeping only key spatially-aware crops

  Search-based methods model high-resolution images using a tree structure and apply search algorithms to identify key image crops. However, when a question requires information from multiple crops, methods like $DC^2$ tend to select the LCA (Lowest Common Ancestor), which is an image containing all relevant areas. This results in extremely high-resolution inputs. In contrast, our RAP employs a Spatial-Awareness Layout to selectively retain only the key image crops, effectively reducing image resolution while maintaining their spatial relationships (see Figure 4 in our paper).

• Search-based methods typically follow a top-down strategy, while RAP starts directly from low-resolution image crops

  Search-based methods often treat the high-resolution image as the root node, splitting it into sub-images (such as a $2\times 2$ grid) to generate child nodes. However, high-resolution root images can mislead the MLLM by providing incorrect search cues at the outset, resulting in inefficient paths and potential errors. Our RAP mitigates this issue by starting directly with low-resolution image crops, enabling more efficient and accurate retrieval of relevant content (see Table 5 in our paper).

Q4: Essential References Not Discussed.

R: Thanks for pointing out the work on SV-RAG and MMed-RAG. SV-RAG applies RAG to multimodal long-document understanding, while MMed-RAG enhances the factuality of Med-MLLMs. In our revised version, we will include these two works in the discussion of Multimodal RAG in the Related Work section.

We truly appreciate Reviewer tave's constructive comments and positive support.

Q1: The concern of the comparison methods are comprehensive enough.

R: We'd like to thank the reviewer for the advice. In fact, building on established works such as $DC^2$ [R1] and ZoomEye [R2], we conduct experiments on $V^*$, HR-Bench, and the widely used benchmark MME-RealWorld. Our study includes a comprehensive comparison with 12 widely adopted MLLMs, and we further demonstrate the efficiency of our RAP method against leading state-of-the-art approaches (see Table 5 in our paper).

To echo the reviewer's concern, here we further strengthened our contributions by including additional experiments:

• We kindly refer the reviewer to our response to Reviewer h1HT (Q2) for additional experimental results on five widely used benchmarks: DocVQA, ChartQA, TextVQA, AI2D, and MMStar.
• We respectfully refer the reviewer to our response to Reviewer TfPd (Q2) for additional evaluations involving five state-of-the-art MLLMs, including LLaVA-ov-7B/72B, Oryx-1.5-7B, Cambrian-8B, and CogVLM-Llama3-19B.

[R1] Wang W, Ding L, Zeng M, et al. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models[C]. In AAAI 2025.

[R2] Shen H, Zhao K, Zhao T, et al. ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration[J]. arXiv preprint arXiv:2411.16044, 2024.

Q2: The effect of crop size.

R: We appreciate the reviewer for the valuable advice. We'd like to clarify that we use the default crop size of $448\times 448$ recommended by VisRAG in our RAP. To further investigate the impact of crop size, here we perform additional experiments on HR-Bench 8K using LLaVA-ov-0.5B and show the results below. Our findings reveal that while variations in crop size result in relatively minor differences, all configurations of our RAP method yield substantial performance gains over the baseline.

Q3: An analysis of failure cases is missing.

R: We'd like to thank the reviewer for pointing out this issue. Admittedly, while RAP demonstrates strong capability in accurately retrieving key visual information, its performance can degrade when the input question lacks critical object-specific details. A detailed failure case is available at the following anonymous link:

https://anonymous.4open.science/r/RAP_Case-1D48/Failure_Case_Study.md

In our revised version, we will incorporate a comprehensive analysis of this case to further clarify the limitations and potential directions for improvement.

We truly appreciate Reviewer h1HT's insightful comments and suggestions.

Q1: Replace confidence score calculation in RE-Search.

R: The reviewer's point is well taken. We clarify that currently, APIs like OpenAI's [R1] provide the logprobs parameter, which returns the log probabilities of output tokens and can be used as confidence scores in RE-Search.

Admittedly, there are also some closed-source models that do not support logprobs. Here, to echo the reviewer's concern, we explore a simple alternative approach using generation-based confidence scores. Specifically, we design a scoring prompt that asks the model to evaluate whether the given image contains sufficient information to answer the question. The constructed scoring prompt is as follows:

Question: {}\nCould you answer the question based on the available visual information? Return only a JSON object with a numerical confidence score (0~10) of "Yes" like {"Yes": x}.

In the implementation, we rescale the final score to the range $[0,1]$. We conduct additional experiments on HR-Bench using LLaVA-ov-0.5B. The experimental results are shown in the table below. Compared to the baseline, using RAP with generation-based confidence scores achieved an average improvement of 7.4%, demonstrating that generation-based confidence scores through the model can also lead to significant performance gains.

The corresponding results and discussions will be included in the revision.

[R1] OpenAI API, https://platform.openai.com/docs/api-reference/completions/create

Q2: Experimental results on general benchmarks.

R: We appreciate the reviewer's advice. As suggested, we extend our RAP to LLaVA-v1.5 7B and 13B, evaluating it on DocVQA, ChartQA, TextVQA, AI2D, and MMStar. These five benchmarks cover resolutions ranging from 390 to 1600. Across both benchmarks and model scales, RAP consistently yields notable performance gains, reinforcing the robustness and generalizability of our approach. We will include the corresponding results on these broader benchmarks, along with further analysis, in the revised version.

Q3: Why does the FCP task still underperform despite preserving spatial relationships?

R: We thank the reviewer for the comments. We would like to clarify that the experiments in Table 1 were conducted using a fixed value of $K$. Our findings indicate that the choice of $K$ significantly affects final performance—using a fixed $K$ leads to suboptimal results on FCP compared to the baseline. To address this, we propose RE-Search, a method for adaptively selecting an appropriate $K$. As shown in Table 4, with RE-Search , both FSP and FCP outperform the baseline, demonstrating the effectiveness of our approach.