INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning

Dear reviewer no3B,

Thank you for your time and thoughtful comments. We are encouraged by your recognition of our contributions as well-justified and novel. Below, we address your concerns in detail.

---

[W1, Q2] Including stronger and more recent baseline models.

Thank you for the valuable suggestion. We have conducted additional experiments using the recent Qwen2.5-VL-3B-Instruct model to further evaluate the effectiveness and generalizability of our INST-IT. The results are presented in the table below.

As shown, integrating INST-IT into Qwen2.5-VL consistently improves performance across a range of benchmarks. Specifically, we observe gains on general image understanding (MMMU, GQA), general video understanding (NeXTQA, VISBench), and particularly clear improvements on instance-level understanding benchmarks (RefCOCOg, INST-IT-I, INST-IT-V). These results are consistent with our previous findings on LLaVA-NeXT, further validating the effectiveness and cross-model applicability of INST-IT.

* We used the lmms-eval [1] toolkit for model evaluation and re-evaluated Qwen2.5VL-3B-Instruct using the same inference parameters and runtime configurations as our model.

[W2] Expanding the size of the human-reviewed subset.

Thank you. We have further expanded the human study by including an additional 200 data points. The scores and the time spent reviewing each sample are shown in the table below. Consistent with the findings from the initial subset of 500 samples, the results indicate that the annotation quality in INST-IT remains satisfactory. However, since manual evaluation is very labor-intensive, we were only able to add 200 additional samples during the rebuttal phrase. We will further scale up the human study in the revised version to improve the representativeness of the evaluated subset.

[W3] The use of visual prompting is effective but not fundamentally novel.

While visual prompts have been used in previous work for referring to specific objects during model inference [2] or for training image understanding models [3], to the best of our knowledge, INST-IT is the first work to directly perform grounded instruction tuning using explicit visually prompted images and videos, and empirically demonstrated that this approach can effectively enhance the model's spatiotemporal fine-grained understanding capabilities.

[W4]The potential computational overhead when using visual prompts.

The SAM2 series models [4] can be used for automated generation of SoMs visual prompts. The table below lists the officially reported [4] inference speeds of different model sizes (measured on an A100 using Torch 2.5.1 and CUDA 12.4).

Compared to LMMs with billions of parameters, the overhead introduced by SoMs construction is almost negligible. Users can choose video segmentation models of different sizes based on the latency requirements of their applications. Thank you for your valuable suggestion, and we will include this discussion in the revised version.

[Q1] Discussion on the potential bias and hallucination introduced by GPT-4o.

Thank you for raising this important concern which is indeed a common challenge faced by all works foucused synthetic data.

To address this, we first conducted small-scale trial annotation (hundreds of samples), followed by manual review to assess annotation quality and identify common hallucination or bias patterns. Based on these observations, we refined the task prompts and optimized the annotation pipeline. Large-scale annotation was only launched once we were satisfied with the output quality.

During initial trials, we observed two main issues: (1) When scenes contained many instance objects, the model occasionally confused their attributes or relationships; (2) For longer videos, it exhibited temporal hallucinations and struggled to maintain accurate spatiotemporal grounding of specific objects.

To mitigate these issues, we introduced several strategies:

• Rather than directly annotating raw videos or images, we inserted SoM visual prompts and used instance IDs to guide the model’s attention. This significantly improved GPT-4o’s spatial and temporal grounding, even in complex scenes.
• For long video sequences, we adopted a sliding-window approach with adjacent frames and explicitly prompted the model to focus on local temporal dynamics, which effectively reduced long-range hallucinations.
• During training, we carefully balanced general data with INST-IT to prevent overfitting to a narrow distribution. Furthermore, we froze the first 12 layers of the vision encoder to avoid biasing the model toward visually prompted inputs.

These efforts substantially alleviated the hallucination risk. Performance gains on both general benchmarks and instance-level understanding tasks further demonstrate the this claim.

[Q3] Typical duration of videos and frame sampling strategy; risk of missing important frames.

• Video duration. Specifically, the INST-IT dataset contains 21,000 video examples, with an average duration of 18.4 seconds and a maximum of 234 seconds. Among them, 5,580 videos are longer than 30 seconds, and 1,483 videos are longer than 60 seconds. We would like to respectly note that the primary goal of INST-IT is to support instance-level understanding, which prioritizes the richness and precision of fine-grained annotations over the length of the videos.

• Frame sampling strategy. For videos sourced from VidOR, keyframes are determined based on the original human annotations; for videos from other sources, we adopt a fixed-interval frame sampling strategy. Indeed, applying a uniform sampling interval across all videos may risk missing critical frames. To mitigate this, we manually inspected a random subset of videos from each source. Based on differences in visual information density and video characteristics, we then assigned different sampling intervals tailored to each source. We present the annotation intervals (annotated frames per second) for different sources in the table below. The average annotation frequency in INST-IT is 0.74 FPS, which is significantly denser than that of previous works such as LLaVA-Hound (0.008 FPS) [5] and ShareGPT4Video (0.15 FPS) [6]. This higher density substantially reduces the likelihood of missing critical frames.

[Q4] The total cost of annotating the entire dataset.

We use GPT-4o API to provide detailed and dense annotations for 51K images and 21K videos, and the total API cost is approximately $6000.

---

Thank you again for your valuable feedback and professional comments. We have done our best to address your concerns and hope our responses are satisfactory. Please feel free to reach out to us if you have any further questions. If you feel they are resolved, we would sincerely appreciate your re-evaluation of our work~ Looking forward to hearing from you!

Best regards,

The authors

References

1. LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
2. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V, Arxiv 2023
3. ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts, CVPR 2024
4. SAM 2: Segment Anything in Images and Videos, Arxiv 2024
5. Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward, Arxiv 2024
6. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, NeurIPS 2024

Dear reviewer SMig,

Thank you for your time and effort, which are greatly appreciated in helping us improve our work. We will address your concerns one by one.

---

[W1] Method novelty

While the dataset is indeed one of our key contributions, our work goes beyond data collection in several important aspects. Specifically, we (1) investigate the effectiveness of instruction tuning with explicit visual prompts for enhancing fine-grained spatiotemporal understanding; and (2) propose a continuous instruction tuning paradigm that includes carefully designed data combination strategies to balance general capabilities and instance-level understanding, as well as a tuning scheme tailored for the visual encoder.

Rather than modifying the model architecture itself, our research focuses more on developing a model-agnostic training framework aimed at enhancing multimodal instance understanding.

[W2, Q1] The effectiveness of SoM in improving data quality

Thank you for raising this question. In fact, SoMs visual prompts play a critial role in instance-level annotation by marking numerical IDs on each instance in the frames and use the corresponding ID in to reference them. This significantly enhances GPT-4o’s grounding ability, enabling it accurately focus on the specific instance of interest.

Without SoMs, GPT-4o struggles to produce fine-grained annotations that meet the requirements, the key challenges are:

• GPT-4o struggles to precisely locate target instances, especially in crowded scenes or when occlusion occurs.
• In video frames lacking instance ID prompts, GPT-4o struglle to recognize an object as the same one after it disappears and reappears.

By providing a straightforward referencing (direct use of instance IDs) and explicit spatiotemporal grounding, SoMs effectively enable fine-grained spatiotemporal annotation at the instance level, thereby improving annotation quality.

To further investigate, we replace the SoMs in the INST-IT Dataset with language descriptions, to simulate model training without visual prompts. As shown in the table below, the model trained with SoM annotations outperforms the one without SoMs, confirming its effectiveness.

[Q2] How to ensure the temporal consistency of instance IDs across frames

Maintaining the consistent instance IDs across frames is indeed a crucial concern. To ensure high data quality, we use video segmentation and multi-object tracking datasets, in which the IDs are manually annotated. This guarantees the temporal consistency, even when instances disappear or reappear.

In addition, recent advances in video instance segmentation [1][2][3] have shown strong performance in ID matching across frames, especially when instances disapper and reappear. These models can serve as promising tools to annotate instance IDs automatically, and we leave this exploration for future work.

---

Thank you once again for your valuable feedback and thoughtful comments. We hope that our responses can address your concerns. If you have any additional questions, please feel free to reach out to us. If you feel your concerns are resolved, we kindly hope that you can consider reevaluating our work. We look forward to your reply!

Best regards,

The authors

References

1. Matching Anything by Segmenting Anything, CVPR2024
2. SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory, Arxiv 2024
3. SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction, Arxiv 2025

Dear reviewer 42ms,

We sincerely appreciate your valuable feedback and insightful comments, which are greatly helpful in improving our work. We are highly encouraged that you thought our paper is good. We will address your concerns as follows.

---

[W1] Regarding how to refer to objects in videos, visual prompts or language descriptions?

Great question! For instance-level understanding, how to inform the model of the instances we are interested in is an important design consideration. The main approaches include 1) visual prompting and 2) language referring. Each method has its own advantages and limitations.

• Using visual prompts allows for explicitly and accurately marking the region of interest in the frame, which can more directly guide the model's attention to the specified area. However, the labor required to annotate visual prompts is indeed an issue that needs to be considered.
• Using language to refer to specific instances is more natural and aligns better with human communication habits. However, due to the potential ambiguity of language, such references might become unclear when there are many objects. This also places higher demands on the model’s grounding ability to locate the region referred to by language within the visual scene.

Additionally, in terms of user experience, with the rapid advancement of interactive segmentation paradigms such as Segmentation Anything (SAM [1,2]) and SEEM [3], users can now directly generate object masks by simplely clicking or swiping, offering a natural and intuitive interaction to work with visual prompts.

It is worth noting that although INST-IT is trained using explicit visual prompts, our model also exhibit improved understanding of language-based references. For instance, in the example shown on the left side of Figure 1 in the main paper, we directly use terms such as “man” and “woman” for instance reference, and our model demonstrates better comprehension compared to other models. We speculate that this improvement may stem from the grounded instance-level annotations in INST-IT, which implicitly enhance the model’s ability to interpret referring expressions and perform visual grounding.

Thank you for bringing this up. We will include a discussion on this point in the revised version.

---

We sincerely thank you once again for your insightful questions and are glad to have a meaningful discussion with you. We have shared our thoughts on instance referring in videos, and hope to engage in further discussion with you:-) Looking forward to your reply!

Best regards,

The authors

References

1. Segment Anything
2. SAM 2: Segment Anything in Images and Videos
3. Segment Everything Everywhere All at Once

Dear reviewer 67yH,

We sincerely appreciate your valuable feedback and insightful comments, which are greatly helpful in improving our work. We will address your concerns as follows.

---

[W1] Track appropriateness and primary contribution

We appreciate the reviewer's feedback and are highly encouraged by the recognition of our dataset as highly useful and our paper is comprehensive.

While the dataset is indeed an important part of INST-IT, we would like to clarify that our contributions go beyond data creation. Specifically, we

• Demonstrate the effectiveness of explicit visual-prompted instruction tuning, showing that directly training with images and videos with visual prompts can effectively enhance the model’s fine-grained spatiotemporal understanding capabilities.
• Desgin a training recipe that enhances the model’s instance-level understanding without compromising its general capabilities, which includes a detailed data combination strategy and an investigation of tunable layers within the vision encoder.
• Emphasize the importance of instance-level understanding, a challenge that has been relatively underexplored compared to holistic scene-level understanding. Our work identifies key limitations in existing models and proposes INST-IT as a solution to this challenge.

Therefore, our work not only introduces a novel dataset but also highlights the importance of instance-level understanding and proposes the use of explicit visual-prompted instruction tuning along with an effective training recipe to address this challenge.

[Q1] Breakdown of Video-MME performanceg

Thank you for you suggestion. We present the performance breakdown across different axes of Video-MME in the table below.

The results demonstrate that the INST-IT Dataset effectively enhances the performance on VideoMME, including the long video understanding capabilities. When employing Vicuna-7B and Qwen2-7B as LLM backbones, we achieved improvements of 7.2% and 14.9% respectively on the Long segment of VideoMME.

[Q2] Statistics on the duration of the video examples

Specifically, the INST-IT dataset contains 21,000 video examples, with an average duration of 18.4 seconds and a maximum of 234 seconds. Among them, 5,580 videos are longer than 30 seconds, and 1,483 videos are longer than 60 seconds.

We would like to respectly note that the primary goal of INST-IT is to support instance-level understanding, which prioritizes the richness and precision of fine-grained annotations over the length of the videos. Moreover, since we use a sliding window over adjacent frames for temporal dynamic annotation, the data pipeline of INST-IT is also well-suited to long videos, we leave this extention for future works. Thank you for your suggestion. We will add the statistics on video durations to the revised manuscript.

[Q3] Temporal referencing method in both training and evaluation

Thank you for the great question. In fact, timestamps and frame indices can be converted between each other. We believe that using timestamps is a more suitable choice for long video understanding, especially when leveraging LLMs with extended context lengths (e.g., 128k in Seed1.5-VL[1]), while frame indices are more appropriate for short video understanding and LLMs with relatively shorter context lengths (e.g., 16k/32k in LLaVA-NeXT[2]). Since our current dataset and benchmark focus on relatively short videos, we adopt frame indices for reference. We plan to explore long video understanding in future work.

---

Many thanks to Reviewer 67yH for the professional and valuable reviews. We have try our best to address each of your concerns, and hope you can feel satisfied with our responses~ Please let us know if you have any further questions. Looking forward to hearing from you!

Best regards,

The authors

References

1. Seed1.5-VL Technical Report, Arxiv 2025
2. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge