InstructDET: Diversifying Referring Object Detection with Generalized Instructions

Response for Weakness 2. Specifically, the performance on refcoco is not extraordinarily high and is not compared with SOTA methods such as poly former.

Thanks for your comments. We would like to illustrate our initial motivation on showing Table 3. We aim to demonstrate that models trained on our InDET dataset do not decrease accuracies on existing test sets. We have checked PolyFormer and find it achieves a higher accuracy on RefCOCO. There are two reasons for this. First, our DROD model is a prevalent ROD model that is not specifically designed for a particular dataset, while PolyFormer is. The backbone of our DROD and PolyFormer differs significantly (i.e., ResNet 50 v.s. Swin-L) as well. Such difference naturally brings different abilities of feature representation. Second, we do not specifically tune hyperparameters of our DROD model for RefCOCO. This is because we do not focus on achieving accuracy gain on RefCOCO as much as possible. We only aim to show our DROD model performs well on existing ROD datasets. In contrast, we focus on demonstrating our model generalizations by evaluating on our InDET test set where our DROD model achieves higher accuracy than existing methods. Specifically, we have appended the results of Polyformer together and more of our DROD results in Appendix J.2. (blue content) in our revised manuscript.

 

Response for Weakness 3.

We apologize for the confusion and would like to make a clarification here. The LLaVA we mention here is indeed a multimodal paradigm that maps visual features into token embeddings with further alignment. This paradigm is similar to minigpt4 and differs from the structure from the original paper. We have changed this expression into "initializing a derived model from LLaVA with minigpt4 weights''.

Revised Content: We have added the footnote on page 4 for clarification.

Dear Reviewer n3DU,

We would like to thank you for providing valuable comments and we answer the raised questions as below. We have included all discussions and results in our revised manuscript (blue content).

 

Response for Weakness 1.

We apologize for the confusion regarding our dataset and we will clarify in the following. In the global prompt pipeline (i.e., single modality pathway), we leverage a variant model of LLaMA named as Vicuna 13B. The Vicuna model has gone through instruction tuning and thus is capable of following our instructions related to user expression generation. For the GPT-4 employment, we have found out that we need to pay for its API usage and have estimated the fee cost to produce the total amount of our required user expressions (i.e., around tens of millions). This cost has exceeded our budget. Nevertheless, we have evaluated on a toy dataset to validate that in our ROD scenario, the expressions generated by GPT-4 are not obviously advantageous compared to those from the Vicuna model.

In our local prompt pipeline (i.e., multi-modality pathway), we observe that a direct expression generation by LLaVA introduces model hallucinations. To mitigate such limitation, we employ the following three steps. First, we finetune LLaVA on the REC dataset. This finetuning transfers the model attention from global images into local objects (as shown in Appendix B.2). Second, we leverage the chain-of-thought and reflection approaches (e.g., in-context learning) to prompt LLaVA to infer results via reasoning (as shown in Appendix B.3) . Third, we further filtered the generated expressions using a CLIP model (Section 3.3). We evaluate the quality of expressions generated in each step by manual verification. We randomly sample 500 expressions from each step and verify the amount ratio of human-like expressions. The results are shown in the following table where we can gradually improve the quality of generated expressions.

Revised Sections:

(1) We add footnote 2 on the fourth page to specify the fine-tuned version of LLaMA that was utilized.

(2) We provide detailed explanations in Appendix B.2 and B.3 regarding the methods employed to enhance the generation quality of LLaVA through fine-tuning and thought of chains.

 

Response for Weakness 2. The experiments are insufficient. The author only presents the result in their own proposed dataset and the original ref coco benchmark. I am not certain how much the dataset actually helps the performance of the model, or helps equip the model with new capabilities.

Thanks for your valuable comments. From Table 2, we observe that ROD models trained on the existing datasets are not effective to ground objects in our InDET test set. This is because our InDET dataset contains much more diversity of human-like expressions than existing datasets. To this end, the evaluation on the RefCOCO test set, whose diversity is limited, is not effective to demonstrate the superior model generalizations brought by our InDET dataset.

In order to further evaluate how our dataset helps model to improve performance or equip new capabilities, we have performed a new evaluation below. We train existing models (i.e., MDETR and UNINEXT) using different training sets (i.e., RefCOCO/+/g, Flicker, and our InDET). Then, we select 2000 test samples from our InDET test set where these samples focus on logical reasoning in ROD. The evaluation results are shown in the following table:

This table indicats that exisiting models achieve significant performance gain by using our InDET dataset. As our test set focuses on logic reasoning, we can conclude that our InDET improves the logic reasoning of representative existing models. We have added this comparison in Fig.7 together with visual examples.

Revised Sections:

In Sec. 6, we incorporate further discussions on multimodal logical reasoning capabilities. Additionally, Fig. 7 provides a quantitative and qualitative analysis of this ability.

Dear Reviewer uAXw,

We would like to thank you for providing valuable comments and we answer the raised questions as below. We have included all discussions results in our revised manuscript (blue content).

Response for Weakness 1.

We apologize for making you confused while presenting our method and would make a clarification here. The "pathway'' we use is to describe that in our framework, there are two pipelines to produce expressions (as shown in Fig. 2). We have replaced "pathway'' with "pipeline'' in our revised manuscript. For "dropout'', we mean to wipe out expressions that are inaccurately generated via both LLaVA and LLaMA. We have replaced "dropout'' with "filter'' in our revised manuscript. Moreover, we have thoroughly proofread our paper to improve our presentation for reducing distractions of readers. For example, in Sec 3.1, we revised "The text prompt we use for LLaVA contains our language guidance that the specific object category shall be mentioned" into "The text prompt we use for LLaVA contains our language guidance emphasizing that several specific interested object categories shall be mentioned". And in Sec 3.3, we revised "The global image description is mitigated via our model finetuning to focus on local object, but not completely disappeared" into "The tendency to generate global image description is mitigated via our model finetuning to focus on local object, but not completely disappeared".

 

Response for Weakness 2.

We apologize for the confusion and make the clarification here. We agree that "single modality pipeline'' refers to the global image description that involves LLaVA for prediction. Specifically as illustrated in Sec. 3.1, we first obtain global image description and then sent this description, object bbxs axis, and in-context examples to LLaMA. Originally, we mean "single modality'' because we feel all the elements sent to LLaMA are text tokens. And we mean "multi-modality'' because we send visual images with marked bbxs, together with text tokens, to LLaVA as illustrated in Sec. 3.2.

For your comments, we agree that it is inappropriate to name same form of inputs and outputs as "single modality'' and "multi modality''. To this end, we have modified our illustration that we replace "single modality pathway'' with "global prompt pipeline'', and replace "multi-modality pathway'' with "local prompt pipeline''. This replacement is based on the functions of LLaVA. In "global prompt pipeline'', we use LLaVA to obtain the global text description (i.e., Sec. 3.1). In "local prompt pipeline'', we use LLaVA directly to perceive the local visual objects (i.e., Sec. 3.2). We have revised these illustrations in our revised manuscript according to your advice.

The reason why we use an out-of-box LLaVA model in "global prompt pipeline'' (i.e., "single modality'') is that LLaVA only provides global image description for an input image. In this pipeline, expressions are generated via LLaMA that takes only text tokens as inputs. As LLaVA is initially trained via image and text pairs, there is no need to finetune LLaVA to obtain global image descriptions. However, in the "local prompt pipeline'' (i.e., "multi modality''), we directly leverage LLaVA to generate expressions by taking images with marked bbxs and text tokens as ipnuts. As we expect LLaVA to effectively perceive local regions around each object, a finetuning is required to transfer the global image description of LLaVA into local perception. In Figure 14 of Appendix H, we show that without fine-tuning, LLaVA achieves only 11.4% in generating local target descriptions. However, when fine-tuned on the REC dataset, LLaVA is improved up to 80.2% .

Response for Weakness 3.

We apologize for the unclear presentation and would elucidate more data generation details below .

For the procedure to generate prompts in Sec. 3.1 and appendix C, we have added more explanations in our revised manuscript to make a clearer illustration for prompt generation. The procedure of single modality pathway (i.e., global prompt pipeline) is designed to generate text instructions targeting for interested objects in an given image. The generation procedure can be summarized as follows,

Inputs. An image and it's bounding boxes of interested objects.

Step1. Obtaining global image description through LLAVA. Prompts for LLAVA to generate image descriptions are empirically designed and provided in Table 7.

Step2. Generating text instructions corresponding to interested objects by LLAMA. Prompts for LLAMA consist of three parts: image information, task description, and in-context examples.

• Image information: the global image description generated in step 1 and it's bounding boxes of interested objects.
• Task description: describing the instruction generation task and emphasizing the outputs to LLAMA. The prompts are empirically designed and provided in Table 9.
• In-context examples: designing three examples of input-output pair for LLAMA to elucidate the task clearer. These examples are also empirically designed and one of them is provided in Table 10.

The task description and in-context examples are fixed and applied for all images. Only image description is generated according to different images. Furthermore, we will release the code of our generation procedure for public usage.

 

For the LLaVA finetuning, we employ the entire dataset from RefCOCO/+/g for our fine-tuning endeavors, which primarily comprises triplets of images, bounding boxes, and instructions. The overall dataset consists of 386k instructions paired with 66k images. During fine-tuning, only the final mapping linear layer is trained. This fine-tuning approach enables the model to enrich the attention on the target object within the red bounding box in the input image while ensuring that the overall model avoids catastrophic forgetting. We initialize our model (i.e., a variant of LLaVA) with MiniGPT-4 weights. We employ Vicuna 13B as our Large Language Model (LLM). The batch size and number of epochs are set to 32 and 30, respectively. The initial learning rate is set as $3\times 10^{-5}$. Model training is conducted via 4 threads on a single A100 80GB GPU. We have provided additional details and pertinent explanations in Appendix B.2 for reference.

 

For the post-processing step, after the generation procedure, we have checked all the generated expressions and removed the the same expressions. We use LLaMA to further diversify the generated instructions by performing synonymous rewriting, the exact prompts we use is added to the Sec.I in our revised manuscript.

We will release the related code for our implementation for further research.

 

Response for Question 1.

Sure, we'd love to share more details of the generation procedure in InDET.

To be clear, the diversified instructions in InDET are generated by the procedure we illustrate in Sec. 3, which leverages LLAMA and LLAVA. The prompts for LLAMA and LLAVA are empirically designed.

We propose both local and global pipelines to generate diversified instructions in InDET. These steps are illustrated in Sec. B and Sec. C. The objective and prompt in each pipeline can be summarized as follow:

• local prompt pipeline (i.e., multi-modality pipeline)
  In this pipeline, we obtain diversified instructions of one single bbx in an image by querying LLAVA for multiple times. The prompts for LLAVA consist of visual and textual content. The visual content is the image with a red object bbx marked on it, and the textual content is empirically designed and provided in Table 4.

• global prompt pipeline (i.e., single modality pipeline)
  In this pipeline, we obtain instructions related to multiple interested objects in a given image by LLAMA. The prompts for LLAMA consist of task description, in-context examples and current image information. The details of these elements are explained in Response for Weakness 3.

To design appropriate prompts for LLAMA and LLAVA, we start from prompts commonly used in previous works or in the LLM community. We have tried some methods such as editing language expression, adding in-context examples, etc, to optimize the current prompts until achieving expected output.

For services and cost of the procedure, thanks to the contributors of LLAMA and LLAVA, we can run those methods locally on our machines based on available implementations. And we add the details of hardware resource and time consumption cost of running the two methods in Appendix K.

 

Response for Question 2.

Yes. We will release our InDET dataset to benefit future ROD research.

Dear Reviewer nJxy,

We would like to thank you for providing valuable comments and we answer the raised questions as below. We have included all discussions and results in our revised manuscript (blue content).

Weakness 1. It's not clear from the paper that the method comparison in table 2 is fair because it's not clear if the models were trained on the same images and bounding boxes

We would like to clarify that in Table 2 where we evaluate on the InDET test set, existing methods (i.e., MDETR, G-DINO, UNINEXT) are trained on RefCOCO/+/g datasets where the images and bounding boxes are the same. For a fair comparison, our DROD model in Table 2 only utilizes RefCOCO/+/g datasets but with diversified instructions. Our superior results indicate that our diversified instructions enrich the alignment of visual objects and language expressions in DROD model.

On the other hand, we perform another evaluation below where all the methods only use RefCOCO dataset for training and testing. The training data of existing methods is the RefCOCO training set, while the training data of our DROD is the same training set but with diversified instructions. For the testing data, all the methods use RefCOCO testing set but with diversified instructions. Through this configuration, we would like to show that existing models trained on RefCOCO dataset are not effective to generalize, while our diversified instructions improve our model generalizations to achieve favorable performance.

The results on the table show that the diversified instructions improve our model performance under the same image and bbx dataset.
 

Weakness 2. One additional interesting experiment that would make this a stronger paper - run SAM on web scale images to get bboxes for images and then scale up this approach further to understand if the benefits saturate

Thanks for your comments. Our data-centric method is capable of generating image-bbx-instruction triplets (i.e., expanding dataset) automatically given bbxs predicted by SAM on web scale images. And our ultimate goal is to train a ROD model with this dataset to generalize ROD on a wide range of images and user expressions. As the size of dataset increases, we need to redesign this ROD model from the perspectives of model size and training configurations accordingly. Such effort needs a significant amount of time and we are unable to accomplish it at present. Nevertheless, we will definitely set this direction as our future work and introduce possible solutions (e.g., advanced learning strategy such as curriculum learning) to overcome the saturation if it really happens.

Dear Reviewer Vtb3, We thank for your comments and answer the raised questions below.

Weakness 1. More experiments are expected. For instance, previous REC models should be trained on InDET and evaluated on RefCOCO/+/g.

We have conducted evaluations on previous REC models under the suggested configurations. The results are shown on the below table.

For each REC method, we perform two experiments. One is to finetune the model on RefCOCO/+/g datasets directly. The other one is to pretrain on InDET and then finetune on RefCOCO/+/g datasets. For a fair comparison, here our InDET dataset only utilizes RefCOCO/+/g datasets but with diversified instructions.

For MDETR, the original MDETR performs pre-processing on the expression to extract root of the expression using a dependency parser, and train the model to predict bbox match the root token only. This training framework is unsuitable for ROD task where the explicit name of the referring target might not show up in the expression. Inspired by UNINEXT, we replace the predefined linear class embedding layer with a visual-language alignment module to compute the matching extent between boxes and the whole expression. For G-DINO, the results are produced by the model retraining. The above table shows that the performance of existing REC models (i.e., MDETR and G-DINO) is improved by training with our InDET dataset, compared to the results by training with RefCOCO/+/g datasets.

Back in 2019, we published VQDv1 in NAACL, which has a lot of similar properties to the dataset in this paper (we called it visual query detection), where the system must output 0-K bounding boxes per prompt.

Here is the website: https://www.manojacharya.com/vqd/

NAACL paper: https://arxiv.org/abs/1904.02794