CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

We thank you for your time and valuable comments. Below we answer the main concerns raised in the review and would be happy to provide further clarification if suitable.

Q1. What would be the time complexity for each individual inference? Would it be very slow?

In our implementation, we use the YOLOX detection head as our object detection model, which is very lightweight compared to the auto-regressive language model and CLIP vision encoder. The number of parameters for the object detection model is negligible compared to the whole model.

We also measure the latency for the object detector and the vision encoder + LLM, separately. From the table above, the object detector takes only 2.1% of the total latency. Since the object detection model is not activated every time during training and inference, it will take less than 2.1% time when we generate a sequence of tokens.

Q2. How resilient the model is to inaccuracies or ambiguities in token placement.

Our model is resilient to inaccuracies or ambiguities in token placement. We evaluate the token placement accuracy and our model's resilience to inaccurate token placement on RefCOCOg/RefCOCO+/RefCOCO datasets.

• Token placement accuracy measures how accurate <obj> and </obj> is placed compared to the ground truth. We handcraft several templates to contain the referring expression into a sentence. For example, one template is "I can see {expression} in the image". If both <obj> and </obj> are placed correctly, we count it as a true positive in overall accuracy. The placement accuracy for <obj> and </obj> are also separately evaluated.
• Localization accuracy measures the correctness of the predicted bounding box using either ground truth placement or predicted placement for <obj> and </obj>.

For token placement accuracy, the results show that the overall accuracy is not bad except on the RefCOCOg dataset, which mostly consists of long expressions. Therefore, it is harder for our model to correctly identify the whole expression. We further analyze the placement accuracy for <obj> and </obj> separately, and we can find that our model can generate <obj> in the right place in most cases, while the generation of </obj> is not always correct. We empirically find that our model often fails in cases such as "an adult giraffe scratching its back with its horn". It will be parsed as "<obj>an adult giraffe scratching its back</obj> with its horn" or "<obj>an adult giraffe</obj> scratching its back with its horn".

From the results of localization accuracy, we can see that the accuracy of using predicted token placement and the accuracy of using ground truth token placement is close. It indicates that our model is robust to inaccurate or ambiguous token placement.

Q3. Fundamental oversimplification of compositionality.

Bringing compositionality to VLM is a challenging problem, and we are the very first step to handle this challenge. The good results in representative compositional reasoning benchmarks show the effectiveness of our method in improving the compositional reasoning ability of VLMs, and we acknowledge that there are some limitations in our method such as not coping much with object-attribute compositionality and spatial event compositionality, which could be crucial future directions.

Q4. Rigid token implementation. If we have a "a man between two dog", how to deal with the <box> for multiple instances of dog?

Our model can deal with the situation that there are multiple objects in an image, thanks to our detector (i.e., a YOLO-style model) can localize them simultaneously.

Taking the text "a man between two dogs" as an example, a <visual> token is automatically generated after the word "dogs". Then, the object detector takes this token as input and outputs two boxes, each for one dog. All the features of predicted boxes will be sent back to LLMs, using one <box> for each detected box. Thus, the updated text would be "a man between two <obj>dog</obj><visual><box><box>". This gives our model the maximum flexibility to handle various scenarios.

Training data

• Diverse data. Some previous works such as [a,b,d] rely on rule-based templates to create specific training data. MosaiCLIP [a] uses a scene graph generator to generate the scene graph of the image, then construct captions based on this scene graph information using simple templates. SyViC [b] utilized synthetic data generated by a simulation platform. They then compiled a list of sentences, including a caption prefix, object enumeration, pairwise positional relations between objects, scene description, and action and clothing descriptions for each human. The sentences were concatenated to form a complete dense caption for the image. TSVLC [d] first parses the text into components such as nouns, verbs, etc., and replaces one word in that text using either pre-defined word lists or LLM prediction to form new captions. Our method relies on raw caption text and can be applied to any large-scale image-text pair datasets using an automatic labeling pipeline, so we have the maximum diversity in data.
• Realistic data. SyViC [b] relies on synthetic data, which may induce a domain gap and lack of variety in content compared to using real-world image-text data. Our method can use data from any image-text data source.
• Faithful data. Faithful data means that the text should be related to the image and there are no hallucinations in data or introduced during the data preprocessing. DAC [c] relies on LLM to generate more captions for a given image and its ground truth caption by asking LLM "What should I expect to see in an image of {caption}?". Some facts that generated by LLM are likely to be hallucinated and unrelated to the image since the LLM cannot see the image directly. Therefore, unfaithful data is generated for training which will harm the model. Unlike DAC, we keep the raw caption, and the grounding process will only ground the objects that are in the image to the textual description in the text, doing our best to reduce hallucinations.

Since only TSVLC releases their pre-trained weight, we choose to compare the compositional reasoning ability of CLIP, TSVLC, and our model using the ARO Top-1/5 metric to further illustrate the importance of diverse, realistic, and faithful data.

The results show that although TSVLC can improve the compositional reasoning ability for CLIP, our method can perform much better, thanks to the diverse, realistic, and faithful training data we have.

We thank for the Reviewer pointing out the overclaim issue, and we have already toned down this statement in our revision. Nevertheless, our method has significant differences from previous works as analyzed above. We test our method on various datasets and tasks including ARO, Cola, HICO-DET, RefCOCO, and VQAv2, and get significant performance improvement, showing the effectiveness of our method. The simple but effective nature of our method is also helpful when scaling up the model or pre-training data, leading to even better performance and wider application for much larger LLMs.

Reference:

[a] Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality. ICCV 2023.

[b] Going Beyond Nouns With Vision & Language Models Using Synthetic Data. ICCV 2023.

[c] Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models. NeurIPS 2023.

[d] Teaching Structured Vision & Language Concepts to Vision & Language Models. CVPR 2023.

We thank you for your time and valuable comments. Below we answer the main concerns raised in the review and would be happy to provide further clarification if suitable.

Q1. Generalization capabilities.

We thank the reviewer for recommending a related benchmark PACO [a] that is more suitable to provide some measure of generalization. We evaluate the AR@1 metric on all L1 queries under the zero-shot instance detection setting.

The results show that our method can generalize well to other image domains and harder tasks. Please see G3 for a detailed explanation of the results.

Q2. There are very few qualitative results presented..

Thanks. We have added more qualitative results and discussion in Appendix A.5 in our revision.

Q3. There are no ablations presented. What does the choice of encoders/pre-training data/tokens have on the model?

Thanks for your advice. We have added an ablation study on communication tokens in G1.

The ablation study results demonstrate the necessity and effectiveness of the proposed communication tokens. Please refer to G1 for detailed explanation.

For the choice of encoder, we follow previous works such as BLIP-2 [d] and LLaVA [e] to use pre-trained CLIP ViT-L [f] as our vision encoder, as the pre-trained CLIP ViT is proved to be a powerful model in various downstream tasks. For the choice of pre-training data, the only concurrent public large-scale grounded image-text dataset is from KOSMOS-2, but KOSMOS-2 only releases a 20 million subset of its full 90 million data. As the data quantity is important for multimodal pre-training, we choose to create our own large-scale grounded image-text dataset based on the widely used BLIP-2 pre-training data. Our pre-training dataset consists of over 97 million image-text pairs, comparable to KOSMOS-2's full pre-training data.

Due to the short period for rebuttal and the time-consuming nature of multimodal pre-training, it is hard for us to conduct an ablation study on encoders and pre-training data during the rebuttal period. We leave the additional ablation study on different encoders and different sets of pre-training data in our future work and we are happy to have more discussion with the Reviewer if needed.

Q4. How sensitive is the model to the template used for prompting?

Our model is robust to the template used for prompting. We evaluate the performance of using different prompts in RefCOCOg/RefCOCO+/RefCOCO benchmarks.

The evaluation results do not differ much when using different prompts, indicating that our model is robust to the prompt template.

Reference:

[a] PACO: Parts and Attributes of Common Objects. CVPR 2023.

[b] MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. ICCV 2021.

[c] Detecting Twenty-thousand Classes using Image-level Supervision. ECCV 2022.

[d] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023.

[e] Visual Instruction Tuning. NeurIPS 2023.

[f] Learning Transferable Visual Models From Natural Language Supervision. PMLR 2021.

Q6. No adequate attribution to previous works.

Thanks for your reminder. During our data-creating process, we have borrowed some techniques from KOSMOS-2 including using spaCy to parse the text and use <obj></obj> to enclose an object. We also have some differences from KOSMOS-2. For example, we use GroundingDINO to directly ground the textual description in the text to the object in the image. In contrast, KOSMOS-2 first parses the sentence to extract all noun chunks. Then, they eliminate certain abstract noun chunks that are hard to recognize such as "time" and "love". Finally, they use GLIP to obtain the bounding boxes for these noun chunks. In comparison, our data-creating method is more simple but still effective. We have revised to give more adequate attribution to previous works in our revision.

Q7. Lack of justifications for the selection of sub-components in the proposed method.

Thanks for your reminder. We have added the ablation study on communication tokens. Detailed results and analysis are shown in G2, which indicates the effectiveness and necessity of these tokens.

Q8. Pretrained data and downstream tasks (i.e., RefCOCO/RefCOCO+, VQA v2) share the same pool of visual data taken from COCO and Visual Genome.

It is worth noting that in KOSMOS-2's instruction tuning stage, LLaVA-Instruct [b] data is used to fine-tune the model, and the images in LLaVA-Instruct data are from COCO. Therefore, KOSMOS-2's training data and downstream tasks also share the same pool of visual data from COCO. The visual data overlap concern is not unique to our model.

To prove the generalization of our model, we further evaluate our model on PACO [a] as recommended by reviewer Aw71.

Our model continues to perform well in this dataset, indicating our model's good generalization to other image domains. The dataset description and result analysis for the PACO dataset can be found in G3.

Reference:

[a] PACO: Parts and Attributes of Common Objects. CVPR 2023.

[b] MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. ICCV 2021.

[c] Detecting Twenty-thousand Classes using Image-level Supervision. ECCV 2022.

Q3. The proposed model outperforms the KOSMOS-2 model in referring expression comprehension tasks but with a small gap. While the grounding of vision-language in KOSMOS-2 was reasonably good compared to the proposed model, it exhibited significantly poorer performance in the previous three tasks. Can you provide insights into these differences?

The performance gap is small in referring expression comprehension tasks because KOSMOS-2 and our model both adopt a similar manner for testing. This gap is large in the previous three tasks because the communication tokens in our method can facilitate our model to understand object relations and focus on a localized ROI to generate more faithful object predictions. The deeper analysis is as below:

1. For referring expression comprehension tasks, both of us use a format of <obj>{text}</obj> as the prompt to the model, and then enable the model to output location information. KOSMOS-2 outputs some location tokens to represent the location, while our model outputs the bounding box through the object detector. The advantages of communicative decoding and the bidirectional communication between the vision module and the language module are not fully exploited in this task, thus the performance gap between KOSMOS-2 and ours is reasonably small.

2. The HICO-DET benchmark asks the model to detect all human-object interactions in the image. Many images contain more than one human-object interaction, that is, there are multiple objects corresponding to the same textual description that need to be identified correctly. We empirically find that compared to KOSMOS-2, our model will make it easier to detect more correct objects corresponding to the same correct textual description in the image. Therefore, our model can find out the more complete human-object interactions within an image. This is thanks to our YOLO-like object detector that can naturally identify all correct objects in the image. The input image will be encoded by the vision encoder to become a map of 16x16 patches. When performing the object detection, each patch will be concatenated with the hidden state of <previsual>/<visual> from LLM, and then fed to the object detector to get a score indicating the probability that this patch contains the center of the object, and also get a regressed bounding box coordinates indicating the left-top and right-bottom coordinates of the object if exists. Each patch can produce a probability and object bounding box individually, thus it is natural and convenient for our model to identify multiple objects.

3. ARO requires that given entity A and relation, the model should predict entity B to which the relation is applied. It requires the model to understand where is entity A and what's the relation and use this information to infer the entity that A is applying the relation to. The design of <previsual>/<prebox> communication tokens can handle this relation information well. By using the information of entity A and the relation, <previsual>/<prebox> communication tokens can guide the LLM to focus on the potential regions for entity B, so it would be much easier to correctly predict entity B. Cola benchmark also require the model to understand both the objects and their relation, thus <previsual>/<prebox> communication tokens can help in this task. Since KOSMOS-2 lacks communication tokens and explicit guidance for LLM to focus on a specific region, it does not perform well in such tasks.

Q4. Can you clarify the large margins between KOSMOS-2 and the proposed method in Table 1? What would be the main contributor to this significant performance leap?

The large margins between KOSMOS-2 and our proposed method in Table 1 are mainly because our model has adopted a novel bidirectional communication between the vision module and language model via communication tokens, which brings a large improvement in compositional reasoning ability. Our ablation study especially shows the importance of the proposed communication tokens for compositional tasks (please refer to G2). KOSMOS-2 does not have this kind of design and thus performs much poorer in these tasks.

Q5. The presentation of the method is brief, making it hard to understand. It is not clear how the communication tokens are generated. I recommend providing a pseudo algorithm to describe the method and making the input/output for each step more readable.

Communicate tokens are automatically generated during the auto-regressive generation process. We have added an explanation of the communication token generation in G1 and a pseudo algorithm in Appendix A.1 in our revision.

We thank you for your time and valuable comments. Below we answer the main concerns raised in the review and would be happy to provide further clarification if suitable.

Q1. Why do you create your own dataset rather than using data from KOSMOS-2.

KOSMOS-2 has made their data public, but they have only released a subset of it, about 20 million image-text pairs instead of the full dataset with 90 million image-text pairs. A large-scale and high-quality dataset is essential for pre-training, therefore, we turned to create our own dataset which has a similar scale compared to KOSMOS-2. The dataset we create consists of over 97 million image-text pairs. We will make our full dataset public in the future.

Q2. Could you provide further analyses of the contributions of the communication tokens?.

Communication tokens are designed to enable bidirectional communication between the vision module and the language module. They can summarize the compositionality information within text to guide the vision module to detect the desired objects, and can also bring the visual feature of the object back to the language module to guide the LLM to notice the localized visual feature, thus helping it to better generate more faithful and related tokens thereafter. We have added an ablation study on communication tokens in G1.

The ablation study results demonstrate the necessity and effectiveness of the proposed communication tokens. Please refer to G1 for detailed explanation.

G3) How well does the model generalize to the dataset that does not overlap with the training data?

To further evaluate the generalization ability of our model, we conducted further experiments on PACO [a] recommended by reviewer Aw71. The images for this dataset come from LVIS [b] and Ego4D [c], so it does not have a significant overlap with our training data. As a result, it can provide some measure of our model's generalization.

PACO is a difficult task for existing methods because it requires the model to identify whether the object in the image is exactly the same as the query. For example, if the query is a long wooden chair, then the long plastic chair in a given image should not be grounded. There is only one positive image that contains the query object, while 100 negative images, containing an object very similar to the query object, are presented as distractors.

We evaluate the AR@1 metric on all L1 queries under the zero-shot instance detection setting. MDETR R101 [d] and Detic Swin-B [e] are two baselines presented in PACO benchmark paper.

The evaluation results show that even though this task is hard, our model performs better than other zero-shot baselines. The result strongly supports the generalization of our model to datasets in other domains.

Revision Summary

We made the following modifications in our revision to address reviewers' questions (highlighted in blue in the revision):

1. We provide a pseudo algorithm in Section A.1 to illustrate how the communication tokens are generated (onDF and fA94).
2. We conduct ablation studies on communication tokens in Section 4.3 (onDF and Aw71).
3. We present more qualitative results and analysis in Section A.5 (Aw71).
4. We add more clear attribution to previous works for our data-creating pipeline (onDF).

Reference:

[a] PACO: Parts and Attributes of Common Objects. CVPR 2023.

[b] LVIS: A Dataset for Large Vocabulary Instance Segmentation. CVPR 2019.

[c] Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022.

[d] MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. ICCV 2021.

[e] Detecting Twenty-thousand Classes using Image-level Supervision. ECCV 2022.

G2) Ablation study on the communication tokens.

In our paper, we have designed four special tokens for communication between language and images, namely <previsual>, <visual>, <prebox>, and <box>. According to their function, we divide them into two categories:

• <previsual> and <visual> summarize the text information so far, and send the summarized information to the object detector to detect ROIs;
• <prebox> and <box> contain the visual feature for the detected ROI and are put back into the language sequence to make LLM notice these localized visual features for better generating future tokens.

To evaluate the effectiveness of these communication tokens, we conduct an ablation study on Compositional VLM 1.4B model. We design four settings:

• To evaluate the overall effectiveness of communication tokens, we designed setting 1, which has no communication token.
• To evaluate the effectiveness of putting grounding process right after the object (this is similar to what KOSMOS-2 does), we design setting 2, which adds <visual> and <box> to enable the model to predict the object's location after the object is generated.
• Based on setting 2, to evaluate the effectiveness of <previsual> and <prebox>, we further add <previsual> and <prebox>. With <previsual>, the model can predict the ROI of the object based on the former object and the relation information. With <prebox>, the model can put the localized visual feature of the ROI back into the sequence before generating the object. This forms setting 4, which also represent the full set of communication tokens.
• To evaluate what's the impact of sending the localized visual feature back to LLM, we design setting 3, which eliminates <prebox>/<box> so that the model will only detect the ROI but not put the visual feature of the ROI back into LLM.

The results in the table above reveal that our proposed communication tokens play a critical role in compositional reasoning tasks, improving ARO from 29.60 to 32.46, and Cola from 34.76 to 44.29. Going deeper into these results, we have the following insights:

1. By comparing setting 1 and setting 4, we can see that with no communication token, our model's performance on compositional reasoning tasks is similar to BLIP-2. This is reasonable because we share the same pre-training data with BLIP-2 and the additional grounding information is not used during training.
2. By comparing setting 1 and setting 2, we can find that generate <visual>/<box> after the predicted object does not help compositional reasoning. We hypothesize that it is because adding extra information after the predicted object description won't help the generation of that object description itself as this is an auto-regressive generation process: the context after one certain token won't have an impact on the generation of that certain token.
3. By comparing setting 2 and setting 4, we can notice that adding <previsual> and <prebox> before the object that will be predicted can boost the performance on both ARO and Cola. We hypothesize the reason is that the ROI prediction and the inserted localized visual feature before the object that are about to be predicted will help LLM to better generate that object description. This hypothesis is based on a simple but important rule: the context before one certain token will have an impact on the generation of that certain token in auto-regressive generation.
4. By comparing setting 3 and setting 4, we can conclude that not putting visual features of ROIs, represented by <prebox>/<box>, back into the generated sequence will hurt the compositional reasoning ability for complex object description. ARO Top-1 accuracy does not hurt much if we do not put <prebox>/<box> back, while the performance for Cola will drop significantly. We speculate this is because the predicted object in ARO is usually a simple word, while the object in Cola is a complex phrase. In this case, <prebox>/<box> tokens which contain more fine-grained information about the visual entity can play an important role in assisting LLM to focus on these complex phrases and generate more faithful and related tokens thereafter.