FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering

We sincerely thank you for your questions and feedback on our paper.

Robustness w.r.t. hyperparameters

"The proposed method introduces many hand-crafted components such as ROI, non-maximum suppression which contains more hyper-parameters. This potentially makes the method less robust."

We understand your concern regarding robustness of FOCUS w.r.t. components such as region of interest (ROI) proposals and non-maximum suppression (NMS). We conduct additional experiments using both LLaVA-1.5 and LLaVA-OV on V*Bench to systematically examine the impact of key hyperparameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\text{dist}$), the maximum ROI size ($s_\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only marginal degradation of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating the stability of our method. During the robustness analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings.

In the following, we present the results of our hyperparameter robustness analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

For the minimum distance between ROI anchor points $s_{\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.

For the maximum ROI size $s_\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.

For the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

Moreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.

In addition to demonstrating robustness to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyperparameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.

Evaluation beyond fine-grained VQA

"The proposed method is only evaluated with limited datasets. Beyond the fine-grained VQA, I would like to see the evaluation on broader range of vision-language datasets."

In our work, we primarily focused on fine-grained VQA, as this research area remains underexplored and lacks efficient, high-performing methods. To assess the broader applicability of FOCUS, we conduct additional experiments on the A-OKVQA dataset, which features questions involving more commonly sized or large objects. Notably, this dataset is also used to evaluate ViCrop's performance in large-object scenarios [R1].

Below, we report the performance of LLaVA-1.5 and LLaVA-OV with and without FOCUS on A-OKVQA, employing the same hyperparameters as used for V*Bench:

We observe only marginal performance degradation, no more than 3.3% compared to the vanilla counterparts. This indicates that FOCUS remains compatible with VQA benchmarks involving larger objects, highlighting its adaptability across different vision-language datasets.

Comparison to ZoomEye

"While becoming more efficient, the proposed method does not outperform another cropping method, ZoomEye. It seems like the method trades performance for efficiency."

We thank you for highlighting this concern. We would like to take this opportunity to put your assessment into perspective. As shown in the table below, FOCUS consistently achieves substantial efficiency gain over ZoomEye, often by 2× to 5×, without compromising accuracy. In 5 out of 8 dataset-model combinations, such as LLaVA-1.5 on HRBench-4K and LLaVA-OV on V*Bench and HRBench datasets, FOCUS even outperforms ZoomEye in terms of accuracy.

In summary, FOCUS does not compromise performance for efficiency. Instead, it achieves a balanced and optimized trade-off, outperforming or roughly matching ZoomEye in accuracy while being significantly more efficient. At a time when the computational demands of AI models are steadily increasing, we see our work as a contribution toward addressing resource constraints and reducing the energy consumption associated with large-scale vision-language systems.

References:

R1: MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs, Zhang et al, ICLR 2025

We sincerely thank you for your constructive feedback and greatly appreciate the emphasis on real-world efficiency metrics, such as inference time, which aligns closely with the core motivation of our work.

Additional efficiency metrics, e.g., FLOPs

"While the number of forward passes is used as the primary metric for efficiency, it would be important to also report actual inference time. This is because efficiency is affected by various factors such as memory access patterns, hardware-level optimizations, and framework-specific implementation details. Although inference time is reported in the appendix, it should be discussed in the main text, and a plot showing accuracy versus number of steps (n_step) would help better illustrate the actual trade-off."

and

"It is unclear how reliable the number of forward passes is as an indicator of computational cost, as it can vary significantly depending on implementation. FLOPs or MACs may serve as more objective metrics for computational complexity."

and

"It is unclear how reliable the number of forward passes is as an indicator of computational cost, as it can vary significantly depending on implementation. FLOPs or MACs may serve as more objective metrics for computational complexity."

In our paper, we select the number of forward passes (FPs) as the primary indicator of computational cost due to its simplicity. To provide a more comprehensive efficiency analysis, we additionally report both inference time and FLOPs for FOCUS and ZoomEye. We calculate the FLOPs for LLaVA-1.5 based on the estimation formulas introduced in FastV [R1] and PyramidDrop [R2]. Following these prior works, we exclude the embedding, projection layers and the output head of the MLLM from our FLOPs estimation:

def flops_estimate(seq_length):
    """
    Estimate the FLOPs of one LLaVA-1.5 forward pass given the sequence length
    Using the estimation formula from FastV/PyramidDrop paper
    Per LLM layer: 4nd**2+2n**2*d+3ndm
    Per ViT layer: 4nd**2+2n**2*d+2ndm
    ViT-CLIP-L outputs 576 + 1 (CLS) = 577 visual tokens
    Number of visual tokens in the sequence is always 576 
    """
    llm_dim = 4096
    llm_num_layers = 32
    llm_ffn_dim = 11008
    vit_dim = 1024
    vit_num_layers = 24
    vit_ffn_dim = 4096
    flops_llm = llm_num_layers * ((4 * seq_length * (llm_dim**2)) + (2 * (seq_length**2) * llm_dim) + (3 * seq_length * llm_dim * llm_ffn_dim))
    flops_vit = vit_num_layers * ((4 * 577 * (vit_dim**2)) + (2 * (577**2) * vit_dim) + (2 * 577 * vit_dim * vit_ffn_dim))
    return flops_llm + flops_vit

Due to the new rebuttal guidelines, we are unable to include new plots. Therefore, we present the results in table format. For the revised version of the paper, we plan to include more detailed efficiency plots, including inference time, FLOPs, and number of steps.

We conduct our experiments for LLaVA-1.5 with FOCUS on V*Bench and report the average inference time and FLOPs per image across the full dataset:

As these are newly conducted experiments, the results for n_steps=8 slightly differ from those reported in the appendix. For comparison, we also provide the results for LLaVA-1.5 with ZoomEye on V*Bench:

In summary, the efficiency gains reported in terms of FPs are consistently reflected in real-world improvements. FOCUS achieves competitive or superior accuracy with significantly lower inference time and fewer FLOPs compared to ZoomEye. For instance, at n_steps=6, FOCUS reaches 70.68% accuracy in just 2.00 seconds and 51.26 TFLOPs, whereas ZoomEye (depth-2) requires nearly 4× more FLOPs and 5× longer inference time to reach a comparable accuracy. These results underscore the practical efficiency of FOCUS for fine-grained visual reasoning tasks.

Pseudocode

"The paper is generally easy to follow. However, providing the full algorithm (e.g., in pseudocode or step-by-step form) in the main text would improve clarity—particularly since Section 3.3 is somewhat difficult to follow."

We agree with you that presenting the full FOCUS algorithm in pseudocode would enhance clarity. Due to the updated rebuttal guidelines, we are unable to upload PDFs containing properly formatted LaTeX-style pseudocode. Nevertheless, we have made an effort to represent the core steps of Section 3.3 using simplified pseudocode within the rebuttal. A complete and properly formatted version of the pseudocode will be included in the revised paper.

"Python- or PyTorch-specific notations (e.g., reshape), which are not appropriate in mathematical equations. These should be replaced with standard mathematical notation."

We agree that replacing Python- or PyTorch-specific notations with standard mathematical notation would enhance clarity. We will revise the equations accordingly in the updated version of the paper.

References:

R1: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, Chen et al, ECCV 2024

R2: PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction, Xing et al, CVPR 2025

We are grateful for your feedback and pleased to hear that you see broad application possibilities for core concepts of FOCUS, not only for fine-grained VQA but a wide spectrum of visual reasoning tasks!

Design choices

"FOCUS has many hyperparameters, as it is quite a complex inference-time algorithm. Authors show the values for those hyperparameters in the appendices, but they don't specify how they set them. There are also many design decisions that I suppose they had to test, such as the prompts used, applying a Gaussian filter to the pseudo-attention map, normalizing confidence scores...

We thank you for pointing out the need for more clarity regarding hyperparameter selection and design choices. We first explain our design choices. Several components of FOCUS are based on best practices from prior work rather than iterative tuning:

• Gaussian filter on pseudo-attention: We apply a Gaussian filter with σ = 1 as a standard denoising step commonly used in image processing [R1]. Empirically, we observe that this results in cleaner and more coherent object relevance maps.
• Confidence normalization: This is adopted from ZoomEye and used without tuning the normalization function.
• Existence prompt: This is inspired by ZoomEye, with only minor wording adjustments to better fit our task context.

Development set and robustness w.r.t. hyperparameters

Do you use a development dataset to take all those decisions? Do you use one of the MLLMs only to find those hyperparameters and then apply them for the other also? Those details are very important in my opinion and I couldn't find much information about them."

and

"From the paper text, I understand that the ablation studies of Section 4.3 are done on the evaluation datasets (VBench and HRBench-4K). This is not a good practice, since ablation studies and development experiments can be connected. Any reader may suspect that ablation studies are actually used to develop the system, optimizing for the test datasets. This weakness is related with the previous. I think authors have to make those points clear."

We acknowledge that we did not use a dedicated development set. One practical challenge is that high-quality, fine-grained VQA data is still quite scarce. We select hyperparameters based on common values from literature, and did not systematically tune them to optimize the performance on evaluation datasets. Regarding the ablation studies in Section 4.3, we confirm that they were conducted strictly post-hoc, after the method was finalized and all hyperparameters were fixed. We exclusively use them to interpret and validate design choices, not to guide system development.

To directly address your concern regarding potential overfitting or reliance on hyperparameter tuning, we conduct a systematic robustness analysis using both LLaVA-1.5 and LLaVA-OV on V*Bench. We examine the impact of key hyperparameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\text{dist}$), the maximum ROI size ($s_\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only marginal degradation of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating the stability of our method. During the robustness analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings.

In the following, we present the results of our hyperparameter robustness analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

For the minimum distance between ROI anchor points $s_{\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.

For the maximum ROI size $s_\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.

For the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

Moreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.

In addition to demonstrating robustness to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyper-parameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.

Our analysis of FOCUS's robustness with respect to hyperparameter selection shows that the method's performance is stable across a range of settings and thus broadly applicable.

References:

R1: Digital Image Processing, Gonzalez and Woods, 2008

We sincerely appreciate your detailed assessment of our work and value the thoughtful questions, which provide meaningful direction for improvements of our paper.

Open-ended VQA

"The current experiments focus only on multiple-choice VQA tasks. The method has not been evaluated on generative or open-ended VQA, and it remains unclear how well FOCUS would generalize to these settings."

We agree with you that this is a valuable consideration, especially when assessing FOCUS for real-world usage. To our knowledge, we are not aware of fine-grained datasets focusing on open-ended VQA. Therefore, we reuse V*Bench for open-ended evaluation. While it primarily follows the multiple-choice format, it also provides the ground-truth answer in a natural language format, e.g., "The color of the dog is white.". To explore the open-ended VQA capabilities of FOCUS and baseline models, we provide them with VQA questions without answer options, e.g., "What is the color of the dog?". Then we compare their responses with the respective ground-truth sequence. We evaluate the responses of the MLLMs using an LLM-as-a-judge framework [R1], leveraging Qwen-2.5-7B. Moreover, we manually review Qwen-2.5’s judgments and correct any misclassifications. The results of this analysis for LLaVA-1.5 clearly show that FOCUS substantially improves the fine-grained open-ended VQA performance, increasing accuracy from 44.50% for the vanilla LLaVA-1.5 model to 65.97%:

Sensitivity to hyperparameters

"The paper uses several hyperparameters, like the minimum distance between ROI anchors and the ROI size threshold. Did you test how sensitive FOCUS is to these choices? Is there an ablation on these hyperparameters?"

We thank you for pointing out the need for more clarity regarding hyperparameter sensitivity. To address your question, we conduct a systematic sensitivity analysis using both LLaVA-1.5 and LLaVA-OV on V*Bench. We examine the impact of key hyper-parameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\text{dist}$), the maximum ROI size ($s_\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only marginal degradation of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating our method's low sensitivity to hyperparameters. During the sensitivity analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings. In the following, we present the results of our hyperparameter sensitivity analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

For the minimum distance between ROI anchor points $s_{\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.

For the maximum ROI size $s_\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.

For the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).

Moreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.

In addition to demonstrating insensitivity to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyperparameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.

Our analysis of FOCUS's robustness with respect to hyperparameter selection shows that the method's performance is stable across a range of settings and thus broadly applicable.

Limitations regarding extremely small objects

"In the Limitations section, you mention that FOCUS struggles with very small objects in high-res images. Can you show more failure cases or examples to help understand this problem?"

We would like to refer you to Figure 7 (IV) on page 25 of the appendix, which illustrates this issue. Due to the updated rebuttal guidelines, we are unable to include additional qualitative examples at this stage. In a revised version of the paper, we will incorporate further examples to more clearly demonstrate this limitation.

VQA regarding spatial concepts

"Can FOCUS handle questions that involve spatial relationships, such as "the red ball on the left," where token similarity maps may not uniquely identify the correct region?"

This question highlights an important limitation of current MLLMs. These models exhibit poor understanding of spatial concepts such as "on the left" and "on the right" in the image [R2-4]. While targeted fine-tuning can partially mitigate this issue [R3], it is not part of the standard training of MLLMs. Consequently, our FOCUS method inherits this limitation and may produce ambiguous object relevance maps when applied to off-the-shelf MLLMs. We will explicitly acknowledge this limitation in a revised version of the paper.

References:

R1: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al, NeurIPS 2023

R2: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, Tong et al, CVPR 2024

R3: SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities, Chen et al, CVPR 2024

R4: Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models, Qi et al, arXiv 2025