ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models

We would like to express our appreciation for recognizing the usefulness and practicality of our study. We are happy to address the concerns raised by the reviewer as below.

1. Problem classification for Visual Commonsense Reasoning. We would like to clarify that we didn’t redefine the VCR problem. VCU focuses on perceiving and understanding literal visual contents in an image, while VCI involves drawing novel conclusions or explanations from these visuals, often using (non-visual) commonsense knowledge and rules. The objective of this categorization is to leverage the strengths of LLMs in commonsense reasoning and VLMs in visual perception (i.e. understanding literal visual contents). The findings in Table 1, which reveal a significant contrast in the behaviors of VLMs and LLMs when it comes to VCU and VCI, support dividing VCR into VCU and VCI is an effective categorization.

2. Why perform experiments on VCR datasets? As mentioned in the 1st point, the problem we are studying is visual commonsense reasoning, and we validate the efficacy of our problem classification in Section 6.1.

3. Definition of VCI. We would like to kindly point out that the definition of VCI is in the Sec. 3.2 of the paper. We’ve made the distinction between VCI and VCU clearer in our revision.

4. Relation between VCU and image-text retrieval. We appreciate the reviewer's insight regarding the connection between VCU problems and image-text retrieval. While acknowledging this connection, we argue that this relationship is not a limitation of our work. As per the ICLR 2024 Reviewer instructions, “originality may arise from […] creative combinations of existing ideas, application to a new domain.” In this vein, our contribution is unique in that we are the first one to solve a subset of VCR, namely VCU, using image-text alignment capabilities in VLMs. We believe that our approach of categorizing problems is significant, as it has the potential to reduce costly API requests to LLMs.

5. Reason for figure 4. We appreciate the reviewer for the suggestion. However, we believe combining the example images and the analysis as a figure does not affect the presentation.

6. Evaluation of the reliability of the problem classification by LLMs. The reliability of the problem classification can be reflected in how the LLMs and VLMs paradigms perform in the classified problems. First, as explained in the first paragraph in Section 6.1, the significant contrast between the performance of both LLM and VLM-based decision models on the classified subproblems demonstrates that the problems classified in different categories have significant differences. Second, the fact that treating the problems differently by problem category balances the performance and efficiency well also shows the effectiveness of the problem classification by LLM.

   Since the definition of the two subproblems is not proposed in the original dataset creation, we could not evaluate the accuracy of the problem classification on a large scale. However, we randomly select 50 examples from AOKVQA dataset and manually label the category of them. The comparison between the LLM classification and human labeling is shown below:

   	Human VCU	Human VCI
   LLM VCU	32	4
   LLM VCI	4	10

   As we can see, the accuracy of the LLM classification is 84%, which shows that LLM’s classification has a high correlation with human’s classification.

We sincerely appreciate the reviewer for their recognition of the insight of our study and the effectiveness of our proposed method. We are happy to address the concerns and questions raised in the review below:

1. The rationale for branching based on the problem category and the confidence. This is an important question. In the decision of choosing the branch for solving the question, we consider two aspects: performance and efficiency, evaluating by the number of LLM calls.

   (a) From Table 1, we can observe that when the LLM is confident about its initial reasoning, its performance is superior or nearly the best for both VCU and VCI problems. In addition, [LLM w/ Caption] is the most compute-efficient. Therefore, using [LLM w/ Caption] is a cost-effective option.

   (b) When the LLM is not confident about its initial reasoning on VCI problems, [LLM w/ Caption+LLM clue] significantly outperforms other decision paradigms. On VCU problems, we can observe that the performance of BLIP2 is similar to [LLM w/ Caption+LLM clue]. However, the [LLM w/ Caption+LLM clue] requires five LLM calls, which is three LLM calls more than using BLIP2. Therefore, using BLIP2-ITA is a cost-effective option in this case.

   We briefly mentioned the efficiency aspect in the original version. We’ve updated the explanation for the branching rationale in Section 6.1.

2. Results on more examples and decoding configurations of VCR dataset. Thank you for the suggestion. We agree running on more examples and more decoding configurations could better help validate our findings. Therefore, we sample 3000 examples from the VCR dataset. The results are shown below: | Model | VCU conf | VCU !conf | VCI conf | VCI !conf | |-----------------|----------|-----------|----------|-----------| | BLIP2-Pretrain | 70.0 | 56.3 | 59.2 | 47.4 | | BLIP2-Pretrain-clue | 70.6 | 56.7 | 63.3 | 49.2 | | Caption | 75.3 | 46.6 | 65.3 | 41.9 | | Caption + VQA Clue | 75.9 | 51.9 | 65.3 | 47.3 | | Caption + LLM Clue | 72.9 | 58.1 | 57.1 | 52.9 |

   As shown above, the conclusions drawn from Table 1, and the branching strategy explained above (and in the revision of the paper) remain unchanged compared with the results on 500 examples.

   We also run the [Caption + LLM Clue] version in Table 1 with extra temperatures and with different in-context examples in the prompt for thinking visual observations to support each choice (Fig.3 Right).

   Caption + LLM Clue	VCU conf	VCU !conf	VCI conf	VCI !conf
   Original	72.9	58.1	57.1	52.9
   temp 0.1	72.4	58.8	57.1	53.9
   Temp 0.2	72.4	58.5	61.2	52.2
   ICL example	74.7	56.7	59.2	54.2

   We can also observe that, using different LLM temperature and in-context examples only have a small influence on the performance, and does not affect the conclusions. We’ve updated the results in the revision.

3. Explanation of LLM+VQA workflow and update of Figure 2. We are sorry for the confusion, as explained in Sec 4.3, in our framework, both the VQA module and LLM clue could be the source of extra visual information in the uncertain scenario. We only include the LLM clue pipeline in Fig. 2 due to its superior performance in our experiments. We’ve fit the VQA pipeline into Fig.2 in our revision.

4. Concatenating the original question and the question about the visual factor as joint inputs to the VQA model. Providing the VQA model in the context of the original question is a great idea that may solve the issue of the VQA model, and we thank the reviewer for bringing this out. We’ve implemented this method on VCR dataset and the result is as below: | ViCor (GPT 3.5) | VCR | |------------------------------|------| | Caption + VQA Clue | 50.6 | | Caption + VQA Clue w/ context| 50.5 | | Caption + LLM Clue | 57.3 |

   The results are on the 3000 subset sampled in the response above. In the comparison, we can observe that this method doesn’t improve the performance compared with the original VQA prompt. This is because answering the visual factor question conditioned on the main question is a complex problem, which is never seen in the training of BLIP2 or the LLM used by BLIP2 – Flan T5 XL. Therefore, the BLIP2 still fails to provide more relevant visual information to the original question.

5. Explanation of ICL. We are sorry for the confusion. ICL means in-context learning, in which several human-defined examples are provided as demonstrations of the task to help the model perform a task without training. We’ve added an explanation of it in Table 3.

We sincerely appreciate the reviewer’s recognition of the motivation of the study and the proposed method. We are happy to address the concerns and questions raised in the review below:

1. Why use words to transfer information between LLMs and VLMs. This is an important question. We acknowledge that relying solely on text for communication between LLMs and VLMs is a limitation of our study, a point we also mentioned in the limitations section. However, we believe our submission is a valuable addition to the literature due to following reasons:

   (a) Currently, connecting top-performing closed-source proprietary LLMs directly with perception (e.g. image, video, point cloud, etc.) modules through embeddings is challenging. As a result, the use of captions remains the only method for establishing this connection. Hence, most current research in multi-modal applications of LLMs primarily relies on caption-based approaches.

   (b) Utilizing text for communication between LLMs and VLMs provides a clear advantage in terms of interpretability. This interpretability is crucial for comprehending the behavior of our method. While embeddings could serve as an alternative medium, they tend to act as a "black-box", offering limited interpretability. This lack of clarity can lead to challenges in debugging and elongate the development process.

   (c) In our initial investigations, we observed a comparable effect when linking LLMs and VLMs through embeddings. Specifically, these models can neglect important visual cues in scenes characterized by clutter and occlusions because they do not actively search for visual cues based on the context of the scene and the question. One possible extension of our approach could involve implementing an attention mechanism over visual embeddings, guided by a high-level comprehension of the scene and contextual information, much like the methodology we have employed.

2. Implementation details of experiments and BLIP2 with LLM. Thank you for the suggestion. We included the implementation details of the compared methods and our method in Sec. 5.2 and Sec. 5.3. We also add the results of the implementation of BLIP2 with LLM below and in the paper. | Model | AOKVQA | VCR | |--------------|--------|------| | BLIP2-Pretrain | 65.6 | 51.2 | | BLIP2-T5xl | 71.6 | 53.6 | | ViCor (ours) | 75.6 | 59.8 |

   As shown above, the BLIP2 with LLM version performs better than the version without LLM, which is mainly because the LLM of BLIP2 improves the commonsense reasoning capability based on the perceived information. Equipping BLIP2 with larger LLMs could potentially improve its performance on visual commonsense reasoning. Currently, connecting top-performing proprietary LLMs like GPT-4 directly with BLIP2 through embeddings is not possible. As a result, the use of captions remains the only method for this connection, underlining the importance of our approach.

3. Explanation of $j$ in $o_j$ As in Eq. 6, $o_j$ corresponds to the $j^{th}$ visual factor. The range of $j$, which is the number of visual factors, is determined by the visual factor reasoning performed by the LLM as in Fig. 2 (e.g. there could be one or multiple visual factors required to answer the question). The prompt for visual factor reasoning is in Fig. 3 middle.

4. Construction of the in-context examples. The questions in the in-context examples in the prompt are selected from the training set.

5. Results of InstructBLIP on VCR. Thank you for the suggestion. The structure of InstructBLIP doesn’t support few-shot multi-modal in-context learning. Therefore, we test the zero-shot result of InstructBLIP on VCR dataset, and the result is shown below. | Model | VCR | |-------------------|------| | InstructBLIP-Vicuna7b | 46.2 | | InstructBLIP-T5xl | 54.6 | | ViCor (ours) | 59.8 |

   From the result, we can see that instruction tuning on a large number of standard vision-and-language benchmarks improves the generalization abilities of VLMs to new tasks compared with BLIP2 using the same T5-XL LLM. However, the performance of InstructBLIP-Vicuna7b version is significantly worse, and our ViCor method still significantly performs better.

6. LLM size of the compared baselines. The number of parameters in GPT-3.5-turbo or GPT-4 is unknown. However, concurrent in-context-learning-based methods and chain-of-thought baselines listed in Table 2 and Table 3 use the same LLMs as ours. BLIP2 and InstructBLIP compared above are using T5XL with 3B parameters, and Vicuna with 7B parameters.

We would like to express our heartfelt gratitude to the reviewer for recognizing the novelty and strong results of our work. We would love to address the concerns raised in the review below.

1. The design motivation of our method and experiments on more datasets. Thanks to the reviewer for mentioning these VQA datasets. We agree that a system that can improve all kinds of visual question answering would be the best. However, the motivation of the design of our framework is twofold: firstly, to leverage complementary strengths of VLMs and LLMs under VCU and VCI problems; and secondly, to enable let LLMs direct VLMs in gathering relevant visual elements that aid in making commonsense inferences. Therefore, our study and framework does not aim to improve the grounding and spatial perception capabilities, which general visual question answering benchmarks like VQA and GQA emphasize.

   As suggested, we apply our framework to another visual commonsense reasoning dataset, specifically OKVQA. It's important to highlight that OKVQA represents an open-ended question-answering (QA) benchmark, distinguishing it from the multiple-choice QA benchmarks presented in our original submission. We randomly sample 1000 examples from OKVQA dataset and test our framework on them, the results are shown below:

   Model	OKVQA
   LLM+Caption	34.3
   BLIP2-T5xl	36.0
   ViCor (ours)	38.4

   Since OKVQA is an open-ended QA dataset, we use the Caption+VQA clue version of our method in Table 1 to tackle unconfident VCI problems. We use GPT-3.5 as the LLM modules in the experiments. As shown above, our framework can still leverage the complementary capabilities of both VLMs and LLMs to achieve better results owing to problem classification and active visual information acquisition.

2. How to divide VCU and VCI problems in the experiments. This is a good question. In the ablation study presented in Table 1, the division between VCU and VCI is based on the output from GPT-3.5-turbo. The prompt we used is shown in Figure 3. As shown in our response to Reviewer eVt8 (Evaluation of the reliability of the problem classification by LLMs), the problem category classification by LLMs aligns well with human categorization, ensuring reliability in identifying problem types. It is important to note that in real usages (results of Tables 2 and 3), LLMs will attempt to classify the question into either VCU or VCI problem only if it is not confident in the initial reasoning, as shown in Fig. 2.

3. Clearer definitions. Thanks to the reviewer for the warm suggestion. We’ve defined these terms in our revision.

Below, we reiterate our answers to key questions raised by the reviewers.

[Results on an additional benchmark]

As suggested by Reviewer GM3o, we apply our framework to another visual commonsense reasoning dataset, specifically OKVQA. It's important to highlight that OKVQA represents an open-ended question-answering (QA) benchmark, distinguishing it from the multiple-choice QA benchmarks presented in our original submission. We randomly sample 1000 examples from OKVQA dataset and test our framework on them, the results are shown below:

Since OKVQA is an open-ended QA dataset, we use the Caption+VQA clue version of our method in Table 1 to tackle unconfident VCI problems. We use GPT-3.5 as the LLM modules in the experiments. As shown above, our framework can still leverage the complementary capabilities of both VLMs and LLMs to achieve better results owing to problem classification and active visual information acquisition.

[Results on a larger subset of VCR dataset]

As suggested by eVt8, we run our method on more examples and more decoding configurations to better validate our findings. Specifically, we sample 3000 examples from the VCR dataset (500 examples were used in the initial submission). The results are shown below:

As shown above, the conclusions drawn from Table 1, and the strategy remain unchanged compared with the results on 500 examples.

[The rationale for branching based on the problem category and the confidence]

In the decision of choosing the branch for solving the question, we consider two aspects: performance and efficiency, evaluating by the number of LLM calls.

(a) From Table 1, we can observe that when the LLM is confident about its initial reasoning, its performance is superior or nearly the best for both VCU and VCI problems. In addition, [LLM w/ Caption] is the most compute-efficient. Therefore, using [LLM w/ Caption] is a cost-effective option.

(b) When the LLM is not confident about its initial reasoning on VCI problems, [LLM w/ Caption+LLM clue] significantly outperforms other decision paradigms. On VCU problems, we can observe that the performance of BLIP2 is similar to [LLM w/ Caption+LLM clue]. However, the [LLM w/ Caption+LLM clue] requires five LLM calls, which is three LLM calls more than using BLIP2. Therefore, using BLIP2-ITA is a cost-effective option in this case.

We briefly mentioned the efficiency aspect in the original version. We’ve updated the explanation for the branching rationale in Section 6.1.

[Why use words to transfer information between LLMs and VLMs]

We acknowledge that relying solely on text for communication between LLMs and VLMs is a limitation of our study, a point we also mentioned in the limitations section. However, we believe our submission is a valuable addition to the literature due to following reasons:

(a) Currently, connecting top-performing closed-source proprietary LLMs directly with perception (e.g. image, video, point cloud, etc.) modules through embeddings is challenging. As a result, the use of captions remains the only method for establishing this connection. Hence, most current research in multi-modal applications of LLMs primarily relies on caption-based approaches.

(b) Utilizing text for communication between LLMs and VLMs provides a clear advantage in terms of interpretability. This interpretability is crucial for comprehending the behavior of our method. While embeddings could serve as an alternative medium, they tend to act as a "black-box", offering limited interpretability. This lack of clarity can lead to challenges in debugging and elongate the development process.

(c) In our initial investigations, we observed a comparable effect when linking LLMs and VLMs through embeddings. Specifically, these models can neglect important visual cues in scenes characterized by clutter and occlusions because they do not actively search for visual cues based on the context of the scene and the question. One possible extension of our approach could involve implementing an attention mechanism over visual embeddings, guided by a high-level comprehension of the scene and contextual information, much like the methodology we have employed.