Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models

Thank you for your detailed feedback on our paper. We appreciate your recognition of the effectiveness and performance of Ferret-v2. Below, we address your concerns and comments:

Incremental Technical Originality

While it is true that individual components have been touched separately in previous works, the novelty of Ferret-v2 lies in the unique design and combination of these techniques to enhance fine-grained multimodal performance comprehensively. Specifically:

a. Any resolution: While previous works used “any resolution” strategies and “direct upsampling” for high-resolution images, our analysis across four tasks (ROC, REC, TextVQA, Ferret-Bench) shows “any resolution” optimally balances high-res image use with pre-training knowledge preservation, crucial for scaling. We designed Ferret-v2’s novel “referring” and “grounding” mechanisms. For referring, we integrate global semantics and local feature maps via a spatial-aware visual sampler. For grounding, our framework aligns with the LLM to delineate output regions.

b. Multi-granularity visual encoding: We appreciate the reference to [3]. Unlike [3], which focuses on multi-level feature fusion of CLIP and DINOv2 features before feeding into LLMs, Ferret-v2 encodes global and sub-patch features separately with distinct encoders and then feeds them into LLMs simultaneously. This approach: 1) Treats global and local patches differently, respecting their unique characteristics. 2) Uses encoders' pre-training paradigms suited for global (CLIP) and local (DINOv2) patches. 3) Maintains separate tokens for global/local features, improving performance on referring and grounding tasks.

c. Multi-stage training: We appreciate the reviewer’s acknowledgment of multi-stage pre-training trends. Our approach has distinct aspects compared to existing works, which often define training stages based on "dataset scales" without model changes. In contrast, our training includes a unique Stage II, using a "coarse-to-fine" approach by initializing two projectors and a visual sampler, setting the foundation for a nuanced strategy. Stage III unfreezes CLIP and DINOv2, "a bold move rarely seen in recent multimodal works " that the reviewer acknowledges and appreciates.

Thank you again for your valuable feedback. We believe Ferret-v2 offers significant practical value and demonstrates satisfactory novelty, regardless of its extending series. We look forward to improving our paper based on your insights.

Thank you for your positive and constructive feedback on our paper. We appreciate your recognition of the clarity of our writing, the innovative techniques we introduced, and the comprehensive set of experiments provided. We address your concerns and questions below.

1. Ethical Considerations and potential risks for real-world deployment

Please refer to Response #2 for Reviewer NGY1 with shared ethical considerations. Ferret-v2 mitigates harmful outputs by using public, academic datasets for transparency and bias reduction, integrating fine-grained spatial knowledge to minimize object hallucinations, and incorporating instruction-following capabilities to handle unclear queries.

Additionally, specific real-world deployment concerns arise from Ferret-v2's referring and grounding capabilities, especially in interactive scenarios where the model provides location instructions and interacts with users, including those with disabilities such as blindness or deafness. These concerns include:1) Contextual Understanding: Incorrect contextual interpretation can lead to erroneous location outputs. 2) Ambiguity in References: Users might provide ambiguous or unclear reference points/boxes/scribbles, which may result in inaccurate outputs. We will address these points in the revised paper.

2. Model training costs and Reproducibility

Ferret-v2 utilizes a multi-stage training paradigm. For Stage-II, Ferret-v2 is trained on densely partial annotated LVIS data (200k) for 1 epoch, optimized by AdamW with a learning rate of 1e-4 and a batch size of 256. we freeze two vision encoders and LLMs, and only update the two projectors as well as the visual sampler in this alignment stage and the training takes ~7 hrs on 8 40G A100 GPUs.

For Stage III, Ferret-v2 is trained on aforementioned GRIT data and task-specific VQA and OCR data (~1.6M) for 3 epochs. optimized by AdamW with a learning rate of 2e-5 and a batch size of 256. In this stage, we render the encoders, projectors, region samplers, and the LLM itself trainable. The training takes ~20/27 hrs on 32 40G A100 GPUs for a Ferret-v2 7B/13B.

We are committed to releasing the training/inference codes and model checkpoints for Ferret-v2. To comply with the double-blind policy, we created a public anonymous repo (https://anonymous.4open.science/r/ferret_v2_anno-85AF/). Upon our institution's approval, we will further organize the code and open-source the full training data and pre-trained model checkpoints.

Thank you for your thoughtful and constructive feedback on our paper. We are pleased that you found Ferret-v2 novel and effective and that you appreciated the clarity of our presentation. We address your concerns below.

1. Efficiency and scalability

We appreciate your concern regarding the efficiency and scalability of Ferret-v2. Here, we added a detailed analysis of Ferret-v2's performance on Referring (LVIS) and Grounding (REC) and a comparison of the computational cost (in terms of inference time and average token per sec) with the previous Ferret baseline. Additionally, we discuss the architectural choices that contribute to this balance. (*All model inference is based on a 7B scale and performed with 8 Nvidia A100 GPUs using greedy decoding.)

We find that "any resolution" and adding an extra DINOv2 encoder significantly improves performance. While this design leads to increased computational demands, the model's performance gains justify the additional computational costs. We will include this discussion in the revised paper.

2. Potential Harmful Outputs and Ethical Considerations

We acknowledge the importance of addressing potential harmful outputs and ethical considerations. Ferret-v2 mitigates potential harmful outputs through several mechanisms: it utilizes public, academic datasets for transparency and bias reduction; it integrates fine-grained spatial knowledge to accurately ground descriptions and minimize object hallucinations; and it incorporates instruction-following capabilities from some datasets (e.g., ShareGPT-40k) dataset to handle unanswerable or unclear queries appropriately. We will include a detailed discussion on these points in the revised version.