Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Models

Novelty. The proposed method builds upon existing multi-resolution and multi-crop techniques without offering substantial novel contributions...

Thank you for raising this point. We agree that the mechanism here is indeed similar to any-resolution methods: like us, LLava-1.5-HD/Llava-UHD also featurize multiple crops and pass them all to the LM, potentially with compression to manage their computational cost.

The difference is that these prior methods use a small number of crops, because their only goal is to preserve the image at native-resolution and avoid destroying information via downsizing. If this was all we needed for a rich visual representation, there would be no more room for improvement by zooming in. Our results clearly show that this is not the case, and that zooming beyond native resolution leads to significant gains, especially for tasks like text and chart understanding that rely on fine-grained, localized image details (see the results of our controlled comparison in Table 1).

The impact of our work is not introducing the idea of multi-crop features, but in finding a new direction for improving visual encodings for VLMs. Our interpretation of the consistently strong results is that it exposes a fundamental weakness of current ViTs: they fail to capture fine-grained details within their context window, even when they are not affected by excessive downsizing. In the short term, this issue can be mitigated by techniques like ours to featurize multiple zoomed-in crops and ensure the LM receives high-quality embeddings. Long-term, we believe a more efficient but challenging solution is to revisit ViT pre-training and even post-training techniques and improve ViTs’ ability to capture more information from single crops (a point we’ve now added to the Conclusion).

To better clarify our motivation and contribution in relation to existing work, we have refined the Abstract, Introduction, and Conclusion sections of our paper. We have also included an additional ablation to further solidify the source of our improvements and differences from prior work in Section 4.4 of our updated submission.

Model Comparison: Dragonfly is developed using the more advanced Llama3 model, whereas comparable methods utilize less capable language models, such as Llama2 and Qwen2. This discrepancy raises concerns about the fairness of comparisons. How does Dragonfly's performance measure up when evaluated against these models?

Thanks for raising this important and valid concern. We would like to clarify that for a controlled and fair comparison, Table 1 shows results of several models implemented within our codebase (including the same dataset, compute resources, LLM, and ViT backbone), where the only difference is the visual encoding strategy. In response to the reviewer’s concern, we have also included new results implementing LLaVA-UHD within our experimental setup, a related work that aims to handle images at native resolution. Empirically, Dragonfly outperforms all these baseline methods across all evaluated tasks, demonstrating the strength of our approach. Notably, LLaVA-UHD’s image encoding aims to preserve native image resolution, theoretically maximizing raw visual information. The fact that Dragonfly surpasses LLaVA-UHD indicates that zooming in beyond the native resolution provides a meaningful performance boost, further validating our design choices.

Regarding the comparison to other models in Table 4, we wanted to test our model when scaled up to a larger dataset and show how it compares against other open-source models, but we acknowledge that these results are confounded by many differences. Most of these models are trained on significantly larger datasets than ours.

Data Influence: It is unclear whether the observed performance improvements with Dragonfly stem from the curated data or from the model's design. How does Dragonfly perform when tested with a commonly used dataset?

We thank the reviewer for raising this important point. Most open-source models are trained on slightly different data mixtures, making it difficult to control for variations in training data. Additionally, even with identical data mixtures, differences in model architecture and compute resources can influence performance. To ensure a fair comparison, we trained the most relevant baseline models (LLaVA-1.5-HD, LLaVA-UHD, and several implementations of our approach) using the same experimental codebase as Dragonfly, including compute, dataset, LM and ViT backbones. The results of this controlled comparison are provided in Table 1 of paper. Regarding the dataset, we use commonly used datasets in the public domain such as LLaVA-Pretrain and ShareGPT4V. Detailed descriptions of these datasets and our filtration process can be found in Section 4.1 and Appendix A.

The novelty is limited. The proposed strategy is only an extension of any-resolution technique. Compared to any-resolution which uses two levels of resolutions, they only resize the image, crop more patches and use three levels of resolutions of image.

Thank you for raising this point. We agree that the mechanism here is indeed similar to any-resolution methods: like us, LLava-1.5-HD/Llava-UHD also featurize multiple crops and pass them all to the LM, potentially with compression to manage their computational cost.

The difference is that these prior methods use a small number of crops, because their only goal is to preserve the image at native-resolution and avoid destroying information via downsizing. If this was all we needed for a rich visual representation, there would be no more room for improvement by zooming in. Our results clearly show that this is not the case, and that zooming beyond native resolution leads to significant gains, especially for tasks like text and chart understanding that rely on fine-grained, localized image details (see the results of our controlled comparison in Table 1).

The impact of our work is not introducing the idea of multi-crop features, but in finding a new direction for improving visual encodings for VLMs. Our interpretation of the consistently strong results is that it exposes a fundamental weakness of current ViTs: they fail to capture fine-grained details within their context window, even when they are not affected by excessive downsizing. In the short term, this issue can be mitigated by techniques like ours to featurize multiple zoomed-in crops and ensure the LM receives high-quality embeddings. Long-term, we believe a more efficient but challenging solution is to revisit ViT pre-training and even post-training techniques and improve ViTs’ ability to capture more information from single crops (a point we’ve now added to the Conclusion).

To better clarify our motivation and contribution in relation to existing work, we have refined the Abstract, Introduction, and Conclusion sections of our paper. We have also included an additional ablation to further solidify the source of our improvements and differences from prior work in Section 4.4 of our updated submission.

The visualizations only show the response of the proposed Dragonfly, other MLLM's responses are encouraged to be listed for better comparison.

Thank you for raising this point. As requested, we have added responses from other baseline models (LLaVA-1.5-HD, LLaVA-UHD) and a stronger model (GPT-4o) alongside Dragonfly for the same image and question. These comparisons can be found in Figure 3 of the appendix in the revised submission. While all models perform reasonably well on most tasks, Dragonfly and GPT-4o excel in tasks requiring fine-grained detail, such as reading a license plate, where both provide correct answers while others fail. On a more challenging task of interpreting the numbers in a chart, Dragonfly is the only model that gets it right.

We believe our quantitative results provide a better comparison between models and therefore moved this qualitative figure to the appendix, but we retained a similar set of QA pairs in Figure 2 to demonstrate the types of biomedical questions in our benchmarks.

Dear Reviewer,

Thank you for taking the time to review our paper. We are reaching out one final time to remind you of our rebuttal to your initial review. We would greatly appreciate it if you could take a moment to review it and let us know if we have adequately addressed your concerns.

Best regards, The Authors

Thank you for taking the time to review our paper and provide feedback. We respond to the concerns mentioned in your review below. We have also revised our submission and the revisions are all in blue.

The paper uses a very simple strategy that combines multi-crop and mean-pooling to enhance VLM

We are not sure if this is meant as a weakness. If so, we believe the simplicity of our approach is not an issue and perhaps even a positive point. In case the reviewer is concerned about the novelty of our approach, we kindly ask the reviewer to take a look at our responses to other reviewers who directly raised those concerns. Briefly, there is a distinction between previous methods that only aim to avoid excessive downsizing that would destroy image information, and our method that shows that zooming beyond native resolution helps capture additional, fine-grained details that ViTs otherwise overlook.

However, the motivation for choosing mean-pooling instead of other compression techniques is not clearly stated. So why does mean-pooling outperform other strategies?

We would like to thank the reviewer for making this important point and we have addressed the reviewer's concern below.

The main idea we introduce in this work is zooming in beyond native resolution; compression is necessary to make this practical, and we take an empirical approach to finding the technique that works best. We tested multiple techniques in Section 4.2 (Table 1), and we choose mean pooling because it matches or outperforms other methods across our suite of benchmarks.

Alternative approaches, such as the perceiver resampler or using a lower-capacity ViT, present certain trade-offs. A perceiver resampler is capable of compressing image tokens into a smaller number, but it’s a parametric approach that requires learning the mapping; it could work better with more training data, but our experiments operate in a relatively low-data regime compared to works like Qwen-VL, and that may be why a learning-free approach like mean pooling proves more robust and efficient. Based on the reviewer's comment, we ensured that this point is discussed in the paper.

Using a less powerful ViT with fewer tokens addresses the token count issue, but at the expense of high-quality embeddings. The specific model we considered, CLIP ViT-B/32, is trained at lower resolution and with larger patch size, and cannot represent as much image information. In contrast, mean-pooled tokens from a stronger ViT (CLIP ViT-L/14 @ 336px) retain richer and more expressive embeddings, making this strategy particularly effective in tasks requiring fine-grained details.

Why do you choose such a pooling window? Will this direct pooling harm the extraction of the fine details?

Thank you for raising this point because our original experiments only showed results for a single compression setting. We have included additional ablations to address this point below.

In general there exists a tradeoff between the amount of compression and the quality of the embeddings passed to the LM. For simplicity, we originally used a uniform compression level of 36 tokens per patch in our original results, which we empirically found to strike a balance between compression, compute efficiency, and performance. Based on your comment, we have added new and more rigorous ablation results in Tables 7-8 showing performance for a range of compression ratios.

The key takeaways are as follows: aggressive pooling, such as reducing to 9 tokens per crop in high resolution or 16 tokens in medium resolution, significantly hurts performance by discarding fine details. Performance improves notably when increasing to 36 tokens in medium-resolution and 16 tokens in high-resolution, after which gains taper off. Overall, these results confirm that zooming can lead to significant performance gains if we strike an appropriate balance between efficiency and embedding quality, even when this involves significant compression via mean pooling.

Performance Comparison for Compression Levels for Mean-Pooling in the Low + High Resolution Model. Tokens are per crop level.

Is there a comparison of the inference time or flops of different methods?

Thank you for the comment. We have conducted a FLOPS comparison for all models trained within our codebase, which we summarize in the table below (Table 11 in the paper):

Dragonfly’s approach of zooming in and processing multi-crops requires more FLOPs than alternative methods, although it is comparable to LLava-1.5-HD. The computational cost is a trade-off for achieving enhanced performance and supporting higher-resolution images. However, as confirmed by our new ablations discussed above, even 16 tokens per sub-image for high-resolution can perform nearly as well as 36 tokens per sub-image; based on this insight, we trained a Dragonfly* model with 64 tokens for low resolution, 144 tokens for medium resolution, and 576 tokens for high resolution, resulting in a total of 784 tokens. This configuration requires 25.10 TFLOPs for the visual encoding and has performance comparable to the main Dragonfly model, as shown in Table 12.

Our main contribution is demonstrating the benefits of zooming in, and this currently requires more FLOPs to process multiple image sub-crops. An interesting future direction, which we now mention in the Conclusion, is exploring ways to achieve similar performance gains without additional VLM FLOPS, potentially through alternative vision encoder pretraining strategies or post-processing techniques like CLIPSelf or locality alignment to better capture fine-grained details.

Thanks to the authors for addressing some of my concerns. I appreciate the feedback, but I still feel like the method doesn’t have enough of an inspiring or fresh approach. I keep my original rating.