RelationVLM: Making Large Vision-Language Models Understand Visual Relations

Thank you for your positive comments and valuable suggestions. We response to your concerns in detail as below.

Q1: The authors introduced how to construct data for RelationVLM but did not explain the overall training. Figure 2 shows the overall training pipeline, but there are no sections or sentences that refer to Figure 2.
A1: Thanks for this suggestion. We add overall training pipeline to Sec. 3.3 that refers to Figure 2 in our revision. Besides, the detailed descriptions for each training stage are also presented in this section.

Q2: Relation Score is a new evaluation metric based on the assessment from an LLM. However, the evaluation cannot reply on an LLM as RelationVLM was trained based on a dataset constructed based on an LLM-based decoder.
A2: In fact, evaluating the performance of large-scale models remains a highly challenging issue currently. We employ a LLM-based assessor for the model evaluation, following the current common practices [1][2][3] in this field. We will explore this aspect in our future study.

Q3: About the typos.
A3: Thanks for your reminder. We have fixed them in our revision.

Q4: What is the $N$ in $D = \{ (x_i, y_i) \}_{i=1}^N$?
A4: Sorry for this confusion. $N$ is the number of training samples. We add this description in the revision.

Q5: Does $R_{n1}(\cdot)$ on Page 5 have any typo?
A5: Sorry for this confusion. But we confirm that there is no typo in $R_{n1}(\cdot)$.

Q6: How do the authors handle multiple relations? Some pairs may contain more than one relations.
A6: In fact, a large number of such cases are already included in our current training data, i.e., some pairs contain one or more relations. In other words, one pair may have annotation for different relation types. It should also be noted that our model is an instruction-following agent. This allows us to handle a specified relation upon user instructions, even if several types exist at the same time.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

[1] Maaz, M., Rasheed, H., Khan, S. and Khan, F.S., 2023. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. arXiv preprint arXiv:2306.05424.
[2] Liu, H., Li, C., Wu, Q. and Lee, Y.J., 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
[3] Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P. and Hashimoto, T.B., 2023. Alpacaeval: An automatic evaluator of instruction-following models.

Q6: The paper would benefit from a more comprehensive discussion on the limitations of the proposed model and approach, as well as potential areas for future research and development. This would provide a balanced perspective and guide subsequent efforts in advancing the field.
A6: Thanks for this suggestion. One of our current main limitations lies in the data used for handling geometric transform is synthesized by us, rather than being collected from the real world. We have not yet found any publicly annotated data that can be utilized, so we synthesize them. We will further improve our model in this aspect in the future study.

Q7: How does RelationVLM handle ambiguous or unclear visual relations in images or videos?
A7: As an initial exploration for teaching LVLMs to understand visual relations, we make our covered relation as basic and widely used as possible to ensure the generalization ability in different scenarios. For ambiguous or unclear visual relations that beyond our modelling range, frankly speaking, we are currently unable to make sure how the model performs. This may require case-by-case discussions, and is worthy of a further exploration in the future.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

Thank you for praising the novelty, effectiveness, reliability, writing quality of our work and the significance of our contributions. For your concerns, we response to them in detail as follows.

Q1: Extending the range of benchmarks and comparisons with existing models.
A1: Thanks for this suggestion. We have performed a broader comparison over extended benchmarks with more models. The corresponding results are in Table 5 of our revision.

Q2: A thorough analysis of data diversity and strategies to mitigate bias would contribute to the robustness and reliability of the model.
A2: Thank you for this valuable suggestion. Currently, the diversity and bias of our data depend to some extent on GPT-4, which is recognized as the most advanced large-scale model. In fact, evaluating the performance of large-scale models, including the diversity and bias of their data, remains a highly challenging issue. We have not yet found a scientific evaluation method from existing publications, and will explore this issue in our future research.

Q3: The complexity of the model architecture and training scheme necessitates a discussion on the computational resources required, as well as the scalability and efficiency of the model across different settings and applications.
A3: We detail the complexity analysis as follows. The vision backbone has 1B parameters, and the vicuna model we used has 7B parameters, totally number of trainable parameters is 0.5B. In the first stage, we conduct training over 100,000 steps using 4$\times$A100 GPUs, with a global batch size of 96 and a base learning rate of 2e-3. This stage is completed in approximately 30 hours. In the second stage, the model is trained for 20,000 steps on 4$\times$A100 GPUs, maintaining a global batch size of 96 and a base learning rate of 2e-4, taking around 6 hours. Finally, the model undergoes an additional 5,000 training steps on 4$\times$A100 GPUs, with a global batch size of 64, completed in about 1.5 hour. The maximum learning rate remains constant at 2e-5 during this final stage. We have added this analysis in our revised supplementary materials.

Q4: The paper aims to enhance the model's comprehension of visual relations, but the definitions and explanations of these relations are somewhat concise. Expanding on the types of visual relations, along with providing more examples, would offer clearer insights into the model's understanding and categorization of these relations.
A4: Thanks for this valuable suggestion as well. Our work is the first one of its kind to explore how to enable the capabilities of understanding visual relations for LVLMs. As an initial study, we start with the three most basic and widely applied high-level categories of visual relations. Here, we further clarify their definitions. For the first class, semantic relation, there are the similarity and the contrast. Among them, the similarity refers to the degree for a given object of being similar to the other one in another image in terms of their category. If they are of the same category, the model should ground them spatially. The contrast is defined as the difference between two instances in terms of their category as well as a series of semantic attributes. The temporal association and the geometric transform refer to the motions or transforms over time and in spatial, respectively. We will attempt to expand the types of visual relations in our future investigation.

Q5: The evaluations presented primarily focus on controlled settings. Incorporating assessments of the model's performance in real-world scenarios would demonstrate its applicability and effectiveness outside of experimental conditions.
A5: Our in-context learning evaluation on anomaly detection or medical diagnosis could be viewed as in-the-wild assessment in real-world scenarios, as their corresponding data and tasks are unseen for the model training. Besides, we will release a model demo for more assessments in the wild once completing our internal reviews.

Thank you for your postive comments and valuable suggestions. We provide detailed responses for addressing your concerns below.

Q1: Comparison with CLIP-based VLMs like "When and why vision-language models behave like bags-of-words, and what to do about it?"
A1: Thank you for this suggestion. The work you list belongs to CLIP-based VLMs. We would like to point out that it's actually diffcult to compare ours with them on most evaluation tasks we used since they lack the capability to handle language input prompts to execute multiple different relation tasks, respond with language output to describe relations, or perform grounding to locate relations. Despite this, we still make a preliminary comparison with them in a comparable experimental setting. Specifically, we evaluate these CLIP-based VLMs in our contrast task using a method similar to the one we used to perform the comparisons with traditional reid models in our experiments. The comparison results are reported as follows:

Q2: It is unclear if the improved performance is due to more data or the curated relation-aware instruction tuning data.
A2: The improved performance mainly comes from our curated relation-aware instruction tuning data, instead of more data. We draw this conclusion based on our experiments in Table 1, 2, 4, wherein all model are trained on the same training data sources while their differences lie in that our model has adopted our proposed relation-aware data curation strategies but others do not. In other words, the performance improvements benefit from the data curation strategy instead of the data itself.

Q3: What is the baseline approach in Table 1? Is it Shikra or Kosmos-2?
A3: The results in Table 1 correspond to comparisons at the model level, instead of an ablation study. Both Shika and Kosmos-2 should not be viewed as our baseline models in the ablation study.

Q4: Please explain the link between anomaly detection and relation understanding.
A4: We apply the task of anomaly detection for evaluating the capability of in-context learning with a few-shot evaluation setting. In this setting, as shown in Figure 5, normal and anomaly samples are given as references. When performing anomaly detection for a given sample, a visual comparison is necessitated between the test sample and the reference samples, in which visual relations are required to be captured for completing such comparison.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

[1] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021, July. Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

[2] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D. and Zou, J., 2022, September. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?. In The Eleventh International Conference on Learning Representations.

[3] Momeni, L., Caron, M., Nagrani, A., Zisserman, A. and Schmid, C., 2023. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15579-15591).

Thank you for your positive comments and valuable suggestions. We response to your concerns in detail as below.

Q1: About the paper writing.
A1: Thanks for pointing this out. We have revised our paper by fixing several typos, simplifying the superscript and subscript of equations, adding additional experimental results, and re-organizing the experiments section in the supplementary materials. Please check the specific changes in the revision.

Q2: About the evidences for the statement "Nonetheless, current LVLMs still struggle to precisely understand visual relations".
A2: For this statement, we propose intuitive illustration in Figure 1, as well as numerical evidences in Table 1, Table 2 and Table 4 in our paper. We demonstrate this through the shown inferiority of them when compared to our proposed model in understanding different types of visual relations.

Q3: It is not rigorous to assume only three factors affect the visual relations.
A3: In fact, we do not make such an assumption, nor do we study the factors affecting the visual relations in this paper. The semantic relations, temporal associations and geometric transforms are the three main types / categories of relations that we define based on the characteristics of visual relations.

Q4: Why are some LVLMs still not considered (such as LLaVA, MiniGPT-4, mPLUG-OWL, etc.)?
A4: In this work, we aim to enable the LVLMs with new capabilities in understanding visual relations that these model does not have. Thus, when we evaluate these target new capabilities, these LVLMs cannot deliver comparable results with ours for specific comparison. We add their corresponding evaluation results in Table 5 of our revision. These experimental results indicate that these models are very deficient in these capabilities. Besides these results, please also not that we have considered the comparisons with these models on the POPE benchmark un Table 8 of our supplementary since they are comparable on this benchmark.

Q5: Why do some tables have different models for comparisons?
A5: We evaluate different capabilities by performing the corresponding comparisons in different tables. To ensure the fairness of the comparison, we only compare our model with the ones that use the same training data in these aspects.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

We appreciate your positive comments on our targeted problem, proposed method and paper writing. We response to your suggestions in detail below.

Q1: Regarding highlighting the contributions on data construction.
A1: Thanks for your suggestion in this regard. We indeed compensate for the deficiency of current large vision-language models in understanding visual relations, based on commonly used model architectures. We have highlight this in the revision, marked in red.

Q2: Could the visual relation comparison benefit other visual tasks, e.g., image retrieval or fine-grained classification?
A2: Thank you for this question. We conduct a fine-grained classification experiment on CUB200-2011 dataset, main results of which are presented in the following table. We add these results in Table 6 and illustrate a representative case study in Figure 8 of our revised version. These results demonstrate that our proposed RelationVLM outperforms SOTA LVLMs and some traditional task-specific models by a clear margin.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

Q9: In Tab 6, the Yes Ratio and Rec. show disadvantages, any discussion?
A9: It should be noted that the Yes Ratio is not necessarily better when it's higher or lower. Thus, the results do not indicate disadvantages. We follow [1] to measure the bias degree of our proposed model with this metric on its public benchmark in which the answers Yes and No account for half each. So a Yes Ratio near 50% indicates a less biased model. Here, our RelationVLM outperforms other LVLMs in this regard. Besides, a lower Rec. (Recall) do not indicate the disadvantages as well. Balancing recall with precision is crucial to the effectiveness of any system. Overemphasizing one aspect can compromise the other. For instance, our model has a lower recall compared to LLaVA, which is a result of LLaVA's near 100% Yes Ratio. This implies LLaVA's inclination to answer 'Yes' to all questions, which may inflate its recall, but doesn't necessarily indicate enhanced performance, considering its lower accuracy, precision, and F1 score. On the other hand, our model excels in both accuracy and F1 scores, demonstrating its overall superior performance. Therefore, despite the lower recall, our model's balanced performance metrics signify its effectiveness.

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

[1] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." In EMNLP, 2023.

Thank you for your positive comments and valuable suggestions. We response to your concerns in detail below.

Q1: Any bias analysis of the text generated from GPT?
A1: We apologize that we have no idea on how to conduct a bias analysis, and we also have not found relevant analysis in related publications for reference. If possible, we hope you could share specific requirements and methods for bias analysis to help us in this regard. Thank you!

Q2: The high-quality subset was manually picked: any details of the cost, quality, and process?
A2: We select the high-quality subset based on the CLIP score, bbox confidence, and answer length. Specifically, we opt for data where the CLIP score is greater than 0.34 (if the CLIP score is available), the bounding box confidence is greater than 0.88 (if the bounding box confidence is available), and the text length exceeds 40 words. We have provided these details in the revision.

Q3: How to ensure the rationality of the geometric transformation?
A3: As an initial exploration, we have not found off-the-shelf annotated datasets to use regarding geometric transform. Therefore, we segment natural images and apply random basic geometric transformations to the segmented objects for synthesizing the corresponding data. Although the dataset is synthesized, we strive to maintain a broad diversity of the synthesized data to enhance the generalization capability of the model as much as possible.

Q4: Better illustration.
A4: Thank you for your valuable suggestions. In the revision, we simplify the superscript and subscript in Eq.1 and add Figure 3 for visualizing the data construction. Besides, we also highlight the best results in experimental results in bold as you suggest.

Q5: On scaling and generalization performance of experimental results.
A5: Given that the evaluation of large-scale models is still a problem worth studying, here, we attempt to conduct some empirical experiments to further analyze the scaling and generalization performance of our experiment results. In terms of the scaling performance, we scale up the test set for evaluating the capability of grounding the objects of the same category to five times its original size. Its corresponding test results are reported in the table below. We can observe that our proposed RelationVLM consistently outperforms other LVLMs on this scaled test set. Regarding the generalization performance, we provide the few-shot results evaluated in the wild on real-world scenarios such as anomaly detection and medical diagnosis, in Figures 5, 6, and Table 3.

Q6: Be careful about using the word AGI.
A6: Thank you for the reminder. We have refined relavant statements by removing this word in the revision. We aim to narrow the gap between the capabilities of existing models and the anticipated AGI ambitions by compensating for the capabilities of existing LVLMs.

Q7: In Fig. 1, the shadow also differs.
A7: Yes, this is a failure case of ours. As the first attempt to explore how to enhance LVLMs' understanding of visual relevance, our model has not yet reached the human level. And there are still some failures on challenging samples that have subtle differences.

Q8: About the performance of visual relation understanding within one image/frame.
A8: Our proposed model can support the visual relation understanding within one image/frame. To demonstrate this, we conduct a validation experiment by combining two images of pedestrians into one and input it into the model for testing the accuracy of judging whether the two people on the left and right are the same person. The results are as follows: