Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

W1: It seems that the proposed approach have negative impacts on the general ability of MLLMs, as shown in Table 2.

Thanks for pointing this out. As mentioned in line 370 of the original manuscript, this phenomenon is primarily attributed to the limited amount of image-level instruction data used for fine-tuning. To further validate this, we increased the image-level training data. The experimental results are shown in the table below.

As the results show, increasing the general image-level training data led to substantial improvements in the VP-MLLM's image-level understanding, allowing it to approach and even surpass the initial MLLM's performance on certain tasks, which demonstrates the potential of the VP-MLLM framework.

W2: Lack of comparison to important baselines. See more detailed comparisons between ViP-LLaVA and VP-LLaVA

Thanks for pointing this out. Below, we present additional detailed comparisons between the results of ViP-LLaVA and VP-LLaVA:

Q1: How does different design choices affect the general capabilities of the MLLM in Table 4?

As the results shown, we can observe the following:

• The use of alpha blending, text prompt referencing, and our visual prompt encoder yields comparable performance in terms of general capabilities.
• The one-stage training approach significantly impacts the model’s general capabilities, likely due to the imbalance between visual prompt data and general VQA data. In one-stage training, the higher proportion of visual prompt data may lead to catastrophic forgetting of image-level knowledge.
• The MDVP-Instruct-Data enhances image-level understanding within the VP-MLLM framework.

Q2: According to Sec 3, visual prompt encoder only supports points and bounding boxes. How does it accept free-form inputs as in Table 1?

In Sec. 3.2 (Simulation Training for Free-Form Visual Prompt Inputs) of the original manuscript, we describe how free-form visual prompt inputs are handled. In summary, we simulate free-form inputs during training by adding noise to the corresponding bounding boxes. During inference, free-form inputs are preprocessed into bounding boxes for modeling. Based on experimental results and user feedback, this approach is both straightforward and effective

Q3: Is there any relations between the <region 1> token and the first token given to the visual prompt encoder?

Yes. During the data construction, we aligned the order of <Region X> with the order of the visual prompts. <Region 1> corresponds to the first token provided to the visual prompt encoder, and <Region 2> corresponds to the second token.

Thank you for your valuable feedback. We sincerely hope our response has addressed your questions and we would be glad to engage in further discussion if needed.

W1: The paper provides limited information on key implementation specifics, particularly regarding model parameter settings and data preprocessing steps.

Thank you for pointing this out. We have included additional implementation details regarding both the model parameter settings and data preprocessing steps in the revised paper (Appendix A). Furthermore, we will release all the code, model checkpoints, and preprocessed datasets to enhance reproducibility and make the implementation details more accessible to readers.

W2: The framework's reliance on pre-trained models may limit its performance based on the initial capabilities of the base MLLMs, potentially constraining generalization across drastically different datasets or new models.

Thank you for pointing this out. After conducting further experiments, we believe that the performance of VP-MLLMs are not constrained by the initial base MLLMs and has the potential to surpass them. Here are the reasons:

• We constructed a Stage 2 training dataset that excludes visual prompt (VP) data to train VP-MLLMs and evaluate whether VP data enhances general image-level understanding. The results, as shown in the following table, indicate that combining VP data with general data significantly improves general capabilities. This demonstrates that using both types of data during training holds the potential to surpass the performance of MLLMs trained solely on general image data.
• When we increased the general image-level training data, the VP-MLLM exhibited substantial improvements in image-level understanding, approaching and even surpassing the initial MLLM’s performance on certain tasks. This suggests that the VP-MLLM framework can generalize to a broader range of datasets and has the potential to exceed the capabilities of the initial MLLM with more diverse and abundant training data.

The two-stage training process and reliance on large datasets are resource-intensive, requiring considerable computational power for alignment and fine-tuning, which may not be accessible to all practitioners.

Thanks for the comments. In practice, VP-MLLMs benefit significantly from pre-trained base MLLMs, enabling rapid convergence during fine-tuning.

Based on our experiments, with eight A100 GPUs, Stage 1 requires only 2 days of training, while Stage 2 takes approximately 3 days to produce a satisfactory VP-MLLM. Therefore, compared to other visual prompting MLLM frameworks, our approach is relatively more efficient and accessible for individual researchers.

Q1: How easily can the Draw-and-Understand framework be adapted to other MLLMs? Are there specific architectural features required in the MLLM for seamless integration?

Integrating the VP-MLLM framework into other MLLMs is simple and straightforward. The steps are as follows:

1. Inherit the entire model structure of the initial MLLMs.
2. Incorporate our visual prompt encoder to handle the visual prompt inputs.
3. Concatenate the image tokens, visual prompt tokens, and text tokens sequentially to form a single input for the LLM.

This entire framework is highly efficient and streamlined, without any additional specific architectural modifications or features.

Q2: The paper mentions that box prompts sometimes underperform compared to point prompts. Could you explain why this discrepancy occurs?

After reviewing the responses of our VP-MLLMs, we observed that for tasks involving a greater number of visual prompts within an image, point-based prompts tend to perform better. Conversely, for tasks with fewer visual prompts, box-based prompts exhibit significantly better performance. In MDVP-Bench, a key evaluation dimension and challenge lies in handling multiple visual prompts simultaneously, contributing to this phenomenon.

We think one potential reason for this is that an excessive number of box prompts can lead to overlapping regions, resulting in semantic ambiguity and making it more challenging for the model to interpret. In contrast, point prompts offer a clearer reference to individual target objects, minimizing overlap and reducing semantic ambiguity.

Q3: Could you provide information about the hardware used for your experiments?

During training, we utilized eight 80GB A100 GPUs; for inference, we only needed one 48GB A6000 GPU. (Four A100 GPUs are enough to train a 13B VP-MLLM)

Thank you for your valuable feedback. We sincerely hope our response has addressed your questions and we would be glad to engage in further discussion if needed.

Q1 and W1: Provide more information on the data used in stage 1. Specific details on the dataset types and the nature of the pre-training tasks would enhance reproducibility.

Thanks for your comments. Here are details about the data and pre-training tasks in stage 1.

Pre-training tasks:

In stage 1, the objective is to effectively align the visual prompt features with the image and text features. Therefore, similar to the image captioning tasks commonly used in mainstream MLLMs pre-training stage, we introduce pixel-level and region-level semantic classification tasks, with only the visual prompt encoder and the projection layer are trainable. Specifically, we first collected open-source datasets related to object detection, instance segmentation, and semantic segmentation. From their annotations, we can obtain ground truth pairs such as (bbox, label), (mask, label), and (pixel, label). Next, we used bbox, mask (randomly sample a few points), and pixel (point) as visual prompts inputted into the VP-MLLM to train the model to respond with the corresponding category. Here are some examples of conversations used during stage 1:

[{"input": "Please identify the labels of each marked point in the image."}, {"output": "<Mark 1>:Label 1\n<Mark 2>:Label 2\n..."}]

[{"input": "Please identify the labels of each marked region in the image."}, {"output": "<Region 1>:Label 1\n<Region 2>:Label 2\n..."}]

[{"input": "Please recognize the text of each marked region in the image."}, {"output": "<Region 1>:Text 1\n<Region 2>:Text 2\n..."}]

Pre-training data:

All datasets used in stage 1 are open-source detection and segmentation datasets. A complete list of the datasets can be found in Table 5 of the Appendix. Specifically, we collected data from five different image domains to enhance data diversity:

(1) Natural Images: containing over 10k real-world semantic categories;

(2) Document Images: including layout classifications such as titles, abstracts, paragraphs, and images;

(3) OCR in the Wild: mainly for recognizing text in natural scenes, such as billboards and signage;

(4) Remote Sensing: for identifying different regions in remote sensing images, such as playgrounds, vehicles, and pedestrians;

(5) Mobile and Web Interfaces: recognizing key elements in mobile and desktop user interfaces, such as icons, text, and search bars.

We have provided more details on the data used in stage 1 and stage 2 in the revised paper. (Appendix A.2)

W2: There is limited insight into the specific impact of the visual prompt encoder's internal mechanisms. A more granular ablation of this component would clarify its effectiveness relative to simpler alternatives.

To further demonstrate the effectiveness of our visual prompt encoder (VPE), we decompose it into two parts:

VPE-base represents the fundamental components, only including positional encoding[1] and an MLP output layer;

VPE comprises trainable parameters that enhance modeling capabilities and a validity identifier used to dynamically determine the number of input visual prompts.

Based on this decomposition, we conducted an ablation study. As results shown, with only the basic component, the VPE outperforms using text alone, which highlights its effectiveness. Furthermore, by incorporating a few trainable parameters and the validity identifier, the VPE shows significant performance improvement. This enhancement not only strengthens the modeling capability of VPE but also enables the VPE to dynamically determine the effective number of input visual prompts, leading to more accurate responses.

Note*: SS is Semantic Similarity, SIoU is Semantic IoU. w/ text means converting the visual prompts' coordinates into text and combined with text prompts for referring.*

[1] Fourier features let networks learn high frequency functions in low dimensional domains.