MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs

Thank you for reviewing our paper. We appreciate your valuable feedback and will try to address your concerns below:

W1:

Thanks for pointing out the work on Matryoshka Query Transformer. Training MLLMs with MQT allows them to have varying visual context size during inference, and this indeed can reduce computational cost. In our current results, we have shown that our methods can work with two different MLLMs with distinct visual context sizes, so it seems entirely possible that our method can still work with varying visual context size under MQT. We will add this discussion to the paper and will explore MQT more closely and integrate into our methods in future works.

W2:

Yes, but please note that most general VQA benchmarks are dominated by large visual objects, and therefore it is natural that a method that improves small object perception does not achieve significant boosts in the general VQA benchmarks. We report these results to show that our methods, while improving perception of small objects, do not come at the cost of perception of large visual objects, that is, they can maintain performance on general VQA benchmarks.

Q1:

In our early experiments we noticed that concatenation works better than addition. We think this is because, without any finetuning, adding the global and cropped image tokens results in shifted image tokens that the frozen LLM has no idea how to deal with. In contrast, concatenation works because the cropped image tokens are simply treated as additional context, and the LLM in the MLLM already knows how to use varying amounts of context.

Thank you for reviewing our paper. We appreciate your valuable feedback and will try to address your concerns below:

W1:

Regarding the draw-backs of MLLMs for small-sized objects, some recent works have noticed this limitation anecdotally (discussed in the opening of Section 5), but to our knowledge, we are the first to A) quantitatively study its existence across multiple SOTA MLLMs (Section 3), B) show that it is causally related to object-size (Section 3), and C) it is primarily a perception limitation rather than localization limitation (Section 4). We will update our related works to better place our work in the literature following your point.

Q1:

We conducted the experiment with weaker intervention per your suggestion for LLaVA-1.5 and InstructBLIP (on TextVQA). Specifically, we crop around the ground-truth object (GTO) in the small set such that its relative size becomes the same as the average relative size of GTOs in the medium set (denoted Align to Medium) and large set (denoted Align to Large). We also randomly move the crop around so that the GTO is not always in the center of the cropped image. We still observed a significant increase in the MLLMs’ perception accuracy as a result of cropping, suggesting that weaker cropping is still effective:

We also note that the accuracy surpasses the medium/large-set’s (similar to human-CROP in Table 1). This suggests that the small-set’s questions are easier than the medium/large-set’s questions, which in turn suggests that the limitation in seeing small objects is even stronger than we observed in Table 1 (because the small set seems to have the advantage of easier questions).

Lastly, we think the accuracy gains from our automatic visual cropping methods (Table 2) provide additional evidence that weaker/less-tight crops are still beneficial.

Q2:

The prior localization works mentioned in the related works (PNP-VQA and Img2LLM) are developed specifically for the BLIP model that has a dedicated image-text similarity computation neural network called the Image-Text Matching network, and are therefore not directly compatible with general MLLMs that do not explicitly train for text-image similarity (like LLaVA-1.5). In this work, we derived a more general way for localizing the attention of MLLMs to images from first principles (the product of answer to image-token attention and image-token to image patch attention). Regarding interpretability of attention in later layers, we observed in Figure 3 that middle to outer layers are more likely to correctly localize than earlier layers (i.e., have higher attention ratio).

Q3:

Our intuition is that the model can use surrounding information to identify where to look. For example, in the car example, it sees the overall appearance of a road next to buildings in the distance, but cannot really see enough details to assign high enough probability to the car tokens.

Q4:

Thanks for the suggestion, we have implemented a stronger baseline: we compared our internal visual cropping methods with external cropping methods under the same setting in Table 4 (SAM, YOLO, CLIP which are stronger than random cropping), and observed that our internal methods perform much better.

Q5:

It is the image resolution that the MLLM can receive inherently (specified in lines 421-425). For example, for LLaVA-1.5 that has an image resolution of 336x336, we choose windows with resolutions from 336x336 up to 672x672 when searching for the cropping bounding box, and then resize the discovered box down to 336*336 for input to the MLLM.

Q6:

The periodic patterns in Figure 3 (top) is due to the periodic definition of the layer number on the x-axis (in lines 249-271). Essentially, within each period (for example 24 for BLIP-2), the layer number L goes from the first layer of the backbone LLM to its outer layer (24 layers in BLIP-2) and computes attention with a specific layer in the connector. So, the number of connector layers (Lc) determines the number of periods, and the number of LLM layers (L) determines the length of each period. Consequently, for LLaVA and Qwen that have a single-layer connector (Lc=1), there is no periodic pattern.

Thank you for the clarification, it is clear to me now and makes sense. It is helpful to understand the average input image size.

Thank you, we appreciate your support of the paper’s contributions. We are happy to explain the bounding box selection further, might be more clear to present it in 2 stages:

1. Selecting the best location per bounding box size (Ln 357-359): For each box size (e.g., 2x the input resolution size), we find the box location on the original image that has the highest internal sum of importance. Note that the original image can be larger than 2x the input resolution of the MLLM. For example, the average input image size is 954x818 in TextVQA (we report dataset statistics in Table 6 in Appendix C).
2. Selecting the best bounding box size (Ln 359-362): Among the “best location per size” boxes (selected in the stage 1 above), we then select the box whose internal sum has the largest difference from the average internal sums of its adjacent positions. This latter step is a heuristic to avoid choosing too small or too large windows (notice that in both cases, moving the window slightly left/right or up/down will not change its internal sum significantly).

Coming back to your specific question, the 2nd step specially avoids selecting a box the size of the entire image, because such a window would have zero difference from its adjacent windows of the same size (since its adjacent window is itself).

I thank the authors for their response. My concerns/comments in the original review have been addressed, and I appreciate the authors' efforts. I am satisfied by the authors' response, and will keep my rating of accept.

It is not that important, but I still have some confusion on how the bounding box selection happens in Ln 356 (if the sliding window is larger than the image, and locations are selected to maximize sum of importance, how would small bounding boxes be selected vs. the whole image?). If the authors see this perhaps they can clarify my confusion. However, I am sure that the approach is fine and my accept rating does not depend on this clarification.

Thank you for reviewing our paper. We appreciate your valuable feedback and will try to address your concerns below:

W1/Q1:

Regarding traditional grounding algorithms, we compared our internal visual cropping methods with several SOTA grounding methods (SAM, YOLO, CLIP) under the same settings in Table 4. We observed that our attention/gradient-based methods (ViCrop) consistently outperform the use of these grounding methods. We think the reason is that ViCrop utilizes the MLLMs’ internal strong question understanding and reasoning capabilities to perform a question-dependent and context-aware grounding.

Regarding the V* method (SEAL), we did not compare with it because SEAL requires substantial training and finetuning of several neural networks, whereas our methods are completely training-free, so a comparison would not be fair. Nonetheless, to provide an idea of how our method compares to SEAL in an “as-is” fashion (i.e., if a user just wants to pick one method as-is off-the-shelf), we report the accuracy of SEAL compared to LLaVA-1.5+rel-att below. We observe that our method outperforms SEAL except on the V* benchmark (we think this might be because SEAL is designed and tuned specifically towards achieving high accuracy on the questions in its V* benchmark). We also note that the inference time of SEAL is significantly slower than our method (4.44s compared to 1.88s on average per question, tested on the same random 100 TextVQA samples with one A6000 GPU).

(We could not report on VQAv2 for the rebuttal because it contains 200K testing samples -- which is 20 times larger than GQA and 40 times larger than Textvqa -- and running SEAL on it will cost us more than 10 days on our available computing resources; we will certainly add it for the final version.)

W2:

Our approach achieves substantial improvements on benchmarks requiring the perception of small visual concepts. To better clarify the gains, we reiterate our rel-att method’s improvements from Table 2 below:

• TextVQA: LLaVA-1.5 goes from 47.80% to 55.17% (7.37 gain); InstructBLIP goes from 33.48% to 45.44% (11.96 gain)
• V*: LLaVA-1.5 goes from 42.41% to 62.30% (19.89 gain); InstructBLIP goes from 35.60% to 42.41% (6.81 gain)
• DocVQA: LLaVA-1.5 goes from 15.97% to 19.63% (3.66 gain); InstructBLIP goes from 9.20% to 9.95% (0.75 gain)
• POPE: LLaVA-1.5 goes from 85.27% to 87.25% (1.98 gain); InstructBLIP goes from 84.89% to 86.64% (1.75 gain)

Other general VQA benchmarks are dominated by large visual concepts, and therefore it is natural that a method that improves small object perception does not achieve significant boosts in the general VQA benchmarks. We report these results to show that our methods, while improving perception of small objects, do not come at the cost of perception of large visual objects, that is, they can maintain performance on general VQA benchmarks.

W3:

We fully agree that studying the architectural causes of this perception limitation is valuable, but we think it merits its own separate paper and therefore we leave it to future works. Our scope in this paper was to first show that the limitation in seeing small objects exists in SOTA MLLMs and is causal (per Table 1), that it is primarily a perception limitation rather than a localization limitation (i.e., MLLMs internally know where to look, per Figure 3), and lastly, that this can be utilized to improve their perception without any training (per Table 2).

Q2:

The method uses a multi-scale bounding box selection strategy (described in Section 5 lines 353-364) that allows the method to select the box that contains the visual subject as completely as possible. However, if the question requires seeing multiple visual subjects that are very far apart, and more than one of them are small, then our method can only help with the visual subject that the model thinks to be the most important. We have discussed this limitation under Limitations and Future Works in lines 497-499, and will try to address this in future works.

Thank the authors for all the comments, I'll keep my score.

Thank you for your question. We use the internal dynamic resolution of LLaVA-NeXT in both the baseline and our method in Table 5. We apply our method to LLaVA-NeXT mostly in the same way as LLaVA-1.5, except for one difference that we will elaborate next. LLaVA-NeXT, unlike LLaVA-1.5, internally divides the global input image into several patches, then independently processes these patches (and a down-sampled global image) by the vision encoder, and then concatenates the resulting features into a single long visual context for the LLM. When applying our method to LLaVA-NeXT, we compute the attention with respect to these patches (rather than the downsampled global image), and then stitch the patch attention maps back together to form an attention map over the original global image. We then use this global attention map for finding the region of interest (per lines 353-363), then crop it, process it with the vision encoder into features, and concatenate the features to the input visual context (which already contains features of patches and downsampled global image from the baseline LLaVA-NeXT). We will of course release the code after publication to further assist the use and adoption of our methods.

Thanks for your question. Following your suggestion, we have run both LLaVA-1.5 and InstructBLIP under our relative attention visual cropping method (rel-att) with (w/) and without (w/o) providing the global image tokens to the LLM on TextVQA:

We observe that the global information is indeed needed as discussed in lines 300-304.