Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

We thank the reviewer for the valuable feedback. We are uncertain if there is some misunderstanding and would like to clarify that VLMs discussed in our paper specifically refer to large visual language models (LLaVA, GPT-4v, etc.) designed for solving general visual language tasks (as stated in Line 23 and Line 26). We are encouraged that the reviewer believes our work "proposes a novel framework to evaluate disentangled perception and reasoning abilities of VLMs". We address the reviewer's main concerns below.

Q1. The Evaluation Might Be Inaccurate

The reviewer suggests that Prism evaluation might be unfair for VLMs that are pre-trained for specific purposes and generate concise responses. However, we would like to clarify that such a premise doesn't align with the assumption of this work. We adopt the Prism framework to evaluate large visual language models (LLaVA, GPT-4v, etc.) designed for solving general visual language tasks. As stated in Line 27, we define the perception capability as "extracting necessary information from the image", which is beyond "seeing" (we are not sure if a VLM can "see" if it cannot clearly express what it sees). By design, VLMs for general tasks should be able to follow the instructions and express detailed information, regardless of response patterns or training strategies. Thus, we believe that perception evaluation under the Prism framework is a reasonable approximation of true perception capacity.

Q2. Effects of Pre-training Dataset on VLM's Abilities

We adopt the LLaVA architecture with InternLM2-7B to investigate the relationship between VLM's pre-training datasets and their perception abilities under the Prism evaluation framework. We construct four different datasets based on ALLaVA dataset to study the impact of descriptive and text-only data:

• ALLaVA-Caption: a set of all descriptive data in ALLaVA.

• ALLaVA-QA: a set of all the QA data used for instruction fine-tuning in ALLaVA with the same amount and images as ALLaVA-Caption.

• ALLaVA-Caption-1xText: a mix of one copy of text-only data in ALLaVA with ALLaVA-Caption, with a ratio of 1:5 (text-only:image-text)

• ALLaVA-Caption-2xText: a mix of two copies of text-only data in ALLaVA with ALLaVA-Caption, with a ratio of 2:5 (text-only:image-text)

The performance of their perception abilities on MMStar is shown below.

The results of ALLaVA-Caption and ALLaVA-QA reveal that utilizing the descriptive data better triggers the VLM’s ability to extract and articulate visual information compared with QA data. Focusing on text-only data, one can observe that training with a small amount of text data can improve the VLM's ability to extract and express information. When the ratio of text data increases, VLM shows a degradation in the perception performance of generic instructions, indicating that an appropriate data recipe is crucial for VLM's perception capabilities.

As for the reasoning aspect of VLM, it is difficult for Prism to give a precise metric to quantify it. Prism evaluation framework is more about assessing whether the overall performance of the VLM is limited by its reasoning capabilities. We will further investigate the particular impact of training data on reasoning in subsequent work.

Q3. Instruction Following Failures

Excluding all cases of instruction following failures can potentially be a way to assess the true perception capabilities more accurately. However, such practice is rarely adopted. First of all, if a VLM fails to extract the relevant information according to given instructions, it's difficult to know whether the issue lies with the instruction following or the VLM's perception capabilities. Meanwhile, most existing multi-modal benchmarks (MMMU, MathVista, etc.) do not make specific adjustments to mitigate the effect of instruction following.

Q4. Image Caption Benchmarks

Directly evaluating the quality of image description can always be the best way to assess the perception capability. However, accurately evaluating image description quality is a significant challenge, and using traditional caption metrics is not a viable solution. Traditional image caption metrics are too sensitive to the caption styles and are not comprehensive. To validate this, we conduct thorough experiments on COCO Caption (val). The partial results are as follows.

Excellent VLMs, such as GPT-4v, GPT-4o, and InternVL-Chat-v1.5, struggle to score well, with all metrics significantly lower than the low-profile Qwen-VL-Chat (the reason is that those advanced VLMs generate much longer responses compared to the ground-truth caption). This contradicts the consensus and further indicates that the caption benchmarks are unsuitable for assessing VLMs' perception abilities. In Prism, we use an external LLM to answer the question based on the image description generated by VLMs to evaluate the VLM's perception capability. Compared to the caption evaluation, the score obtained with Prism is more comprehensive and less sensitive since whether an LLM can successfully answer the question is basically determined by the quality of the descriptive text, not the style. Moreover, existing image caption benchmarks are often limited to specific domains. In Contrast, Prism can be applied to any general multi-modal benchmark to study the VLM's capability.

Dear Reviewer onkG,

We greatly appreciate the time and effort you've taken to review our submission. We hope that our response has addressed the concerns raised in your initial reviews, and we look forward to your feedback.

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you need any further information or clarification. We are fully open to engaging in further discussions to improve our work.

Best regards and thanks,

Paper 8561 Authors

We thank the reviewer for the encouraging comments. We are pleased to note that our response could address some of the reviewer's concerns.

Regarding the proportion of the different types of failures, we sample 100 cases from all VLMs and conduct quantitative analysis based on the categorization in our response to the reviewer KmBE. There are 72 perception failures and 34 reasoning failures in all 100 cases. In some cases, both perception and reasoning errors exist. The detailed proportion result of errors (in perception/reasoning) is as follows:

• Perception: Factual Errors (43.1%), Incomplete Details (40.2 %), Instruction Following (16.7%)
• Reasoning: Logical Errors (67.6%), Lack of Knowledge (26.5%), Misunderstanding of the Query (5.9%)

For the cases where perception and reasoning errors both exist, here is an example from MMStar:

Index: 1222

Image: A bar chart titled "Accuracy of different algorithms". The vertical axis is labeled "Accuracy" and ranges from 0 to 10. The horizontal axis has two categories: "ivory" and "calf." The height of "ivory" is 9, and that of "calf" is 4.

Question: What is the sum of the accuracies of the algorithms calf and ivory?

LLaVA-NeXT (Vicuna-13B) succeeds in obtaining the heights of two categories but gives the wrong horizontal interval, 0-9, which is a perception error. ChatGPT gives the wrong sum, 11, even if the heights 4 and 9 are correctly expressed by the VLM, which is a reasoning error.

Once again, we thank the reviewer for the time and insights. We hope this comment could address the reviewer's concerns and will make sure to properly incorporate the additional discussions in the rebuttal into our revised paper.

We thank the reviewer for the encouraging comments and address the main concerns below. Limitations and broader impacts have been discussed in Appendix. D. We are inspired that the reviewer believes our work "provides some valuable insights" and "brings a new approach to solve the tasks that require complex reasoning process." These are indeed what we want to emphasize.

Q1. Prompt Design

We carefully designed the prompts used in the Prism framework:

1. Generic Instruction (Perception): The generic instruction provides common perspectives when describing images. We refer to MathVista and consider scenes, objects, relationships, and text. We further add instance location into the instruction and obtain better results. We experiment with variants of generic instructions and choose the relatively better one, as shown in Table 2.
2. Query-Specific Instruction (Perception): We incrementally construct the query-specific instruction by merging the generic instruction with the query-specific part. Generating the query-specific part is a reasoning task assigned to LLM. Meanwhile, it is essential to ensure that only the contents to observe appear in the instruction since redundant content (questions, etc.) can stimulate the reasoning capability of VLMs. We design precise tasks and add in-context examples for the LLM, as shown in Fig. 7.
3. Instruction for Reasoning: Visual descriptions and questions can be lengthy and complex, which easily leads to confusion for LLMs. We prompt the LLM to act like a text-based reasoning expert to derive the answer based on the provided information, which shows the best results during our iterations. The prompt template is shown in Fig. 8.

Q2. Computational Cost and Latency

1. Training Cost: For a large end-to-end VLM (like VLM based on Llama3-70B), the training cost is extremely high regarding hardware resources and time. Prism offers an alternative which only requires training a lightweight VLM as the perception module, significantly reducing the VRAM usage (GPUs with 40/80GB VRAM are still scarce and expensive) and the training computation. Prism doesn't tune the LLM. The total training cost of Prism is much less than that of an end-to-end VLM equipped with large language encoders (like Llama3-70B).
2. Deployment Cost: The user spends much less cost to deploy a lightweight visual captioner compared to a large-scale VLM. On the LLM side, thanks to the advanced deployment techniques of LLMs, the LLM inference APIs are available at an extremely low price (also see Appendix D), so the user can take advantage of vast numbers of LLM APIs with low financial cost.
3. Latency: Prism requires a VLM to generate intermediate descriptive text, bringing additional computation overhead. When solving a single visual question (VQ), a higher latency is expected compared to the end-to-end VLM with a language encoder of the same size. However, when one asks for multiple VQs corresponding to one image, Prism will display a smaller latency since the descriptive text can be reused. Moreover, the advanced LLM deployment techniques (compared to VLMs) further help reduce Prism's latency.

Q3. Lack of Linguistic Expressions for Some Images

In Appendix. D, we mentioned that the current PrismCaptioner (learned on the general multi-modal corpora ALLaVA) may struggle to describe images in unseen domains (GUIs, medical images, etc.). However, we do believe that almost all images can be described in detail by linguistic expressions. For example, a well-trained doctor can describe a medical image in detail with natural language (only noteworthy things, not all pixels, need to be described) and a graduate student can convert a screenshot of math equations into natural language descriptions or latex. With sufficient high-quality visual instruction tuning data, PrismCaptioner also has the potential to master these professional tasks.

Q4. Relevance and Differences of Referred Works to Prism

We would incorporate those literature in related works and include a detailed discussion. The relevance and differences between Prism and these works lie in the following aspects:

1. Motivation: Prism and referred works have different motivations: referred works mainly focus on evaluating visio-linguistic compositional knowledge or fine-grained understanding (mostly perception tasks), while Prism focuses on analyzing the perception and reasoning abilities of VLMs in a decoupled manner.
2. Methodology: Prism and referred works all establish an evaluation system to analyze the capabilities of interest. [1, 2, 4] construct benchmarks to evaluate certain capabilities, [3] establish a new paradigm to study the compositional knowledge of VLMs, while Prism creates a decoupling framework to study the perception and reasoning capabilities, respectively.
3. Application: From the application perspective, Prism can not only serve as an evaluation framework but also effectively address visual language tasks.

Q5. Decoupling Versus End-to-End (e2e)

1. Cost and Latency: As discussed in Q2, to reach similar performance, the decoupling paradigm features a smaller training and deployment cost (VRAM usage, hardware resources, financial costs, etc.) compared to the e2e approach. For a single visual question, Prism shows a higher latency compared to the e2e VLM. Meanwhile, the average latency of Prism can be lower when Prism deals with a batch of VQs corresponding to the same image.
2. Performance: With abundant computation resources and training data to train a large-scale VLM (equipped with large language encoders such as Llama3-70B), adopting the decoupling paradigm may not be a good choice for the final performance. However, under resource-limited (computation, training data, financial budget, etc.) scenarios, Prism can outperform e2e VLMs regarding performance and flexibility (please refer to Tables. 5, 6).

Dear Reviewer rB4V,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope our response has effectively addressed the concerns raised in your initial reviews, and we eagerly await your thoughts and further guidance to refine our work.

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require any additional information or clarification. We are more than willing to engage in further discussions to enhance our work.

Best regards and thanks,

Paper 8561 Authors

We express our gratitude to the reviewer for the constructive feedback. We are encouraged to learn that our response could address some of the reviewer's concerns.

In terms of Q3: Lack of Linguistic Expressions for Some Images, we believe most images can be described in detail by linguistic expressions, even if they are medical pathology images or math contents. However, we acknowledge that natural languages may struggle to express some obscure visual elements, especially some aesthetic contents, e.g., abstract artworks and surrealist paintings. In these cases, it is difficult to obtain high-quality descriptions. Thus, we will further focus on potential representation in a more abstract way instead of linguistic expressions.

Once again, we thank the reviewer for the time and effort. If our response could address the reviewer's concerns, we hope the reviewer could raise the score. (now it is 6.) We will ensure that the additional discussions in the rebuttal are properly incorporated into our revised paper.

We express our sincere gratitude to the reviewer for the constructive feedback. We are glad that the reviewer appreciates the "good analysis, findings, and insights" presented in this work and recognizes the potential of the Prism framework in both evaluation and as a task solver. We address the concerns of the reviewer below:

Q1. Lacks Novel Contributions in terms of algorithms and model design:

In this work, we aim at designing a straightforward framework to disentangle and analyze the perception and reasoning capability of VLMs. In pipeline design, we come up with the query-agnostic and query-specific settings to present more comprehensive evaluation results. When developing the PrismCaptioner, we also aim at providing the simplest possible baseline to validate the hypothesis that Prism equipped with a small-scale VLM and a large-scale LLM can potentially be a powerful multi-modal task solver. Thus, we select the most widely adopted LLaVA architecture and simply adopt ALLaVA as the instruction tuning corpora. Developing more sophisticated tuning algorithms or more advanced model architecture would improve the Prism framework's overall performance, but it's not our focus, and we leave it to future works.

Q2. Error Mode and Hallucination

We conduct a thorough analysis and categorize errors in perception and reasoning into the following major modes (the content will be added to the refined version of this manuscript):

1. Major Error Modes (Perception):
   • Factual Errors: VLMs may describe images with inaccuracies, such as stating that prominent elements are "not visible".
   • Incomplete Details: Even in the absence of factual errors, VLMs may lack detailed content, resulting in insufficient information for reasoning.
   • Instruction Following: VLMs sometimes fail to follow instructions when providing corresponding descriptions.
2. Major Error Modes (Reasoning):
   • Logical Errors: LLMs may produce incorrect conclusions or reasoning processes due to limited reasoning abilities.
   • Lack of Knowledge: The absence of relevant domain knowledge prevents LLM from solving corresponding problems, especially in specific fields.
   • Misunderstanding of the Query: In rare cases, the query-specific part generated by LLMs deviates from the original question, misleading the perception of VLMs.

Regarding the issue of hallucinations, identifying their source is crucial in our cascading framework. We can leverage various powerful LLMs to reason about the descriptions generated by a VLM and analyze the patterns in their reasoning results. Here are three potential scenarios.

1. If most LLMs yield the correct answer while one LLM yields an incorrect one, the latter may struggle with reasoning or experience a hallucination.
2. If most LLMs indicate a wrong answer, it likely means that hallucinations generated by the VLM cause some misdirection.
3. If the answers of the LLMs are chaotic or if there are refusals to answer, it may indicate that the VLM did not provide sufficient detailed information.

We conduct experiments with various VLMs, by selecting a set of powerful language models, including GPT-3.5-Turbo-0125, GPT-4-Turbo-0125, Llama-3-70B-Instruct, and DeepSeek-v2-Chat. For each question, we gather results from four LLMs and consider cases where the same choice appears three times or more as "agreement". In agreement cases, we define the corresponding choice as "voted choice" and focus on the following cases:

• Case 1: With descriptive texts generated by a specific VLM, the voted choice is wrong.
• Case 2: With descriptive texts generated by a specific VLM, the voted choice is correct.
• Case 3: For a specific LLM, it makes the same choice as the voted choice by all LLMs, while the voted choice is also the correct one.

For each VLM, we analyze the rate of case 1, which indicates the potential hallucinations of VLMs. For each LLM, we correspondingly calculate the rate of case 3 to case 2 to observe the alignment of its predictions with voted choices. The higher alignment rate means more robust reasoning since voted options are considered better. The notions and results are as follows.

The results show that stronger VLMs show lower error rates, indicating that more capable VLMs experience fewer hallucination issues. By delving deep into the specific cases, we find that VLMs are prone to hallucinations in spatial awareness and fine-grained perception.

All LLMs show relatively good alignment rates, demonstrating relatively robust reasoning performance. GPT-3.5-Turbo-0125 is more unstable by comparison. The cases where LLM's prediction does not align with the voted choice may stem both from reasoning ability issues and hallucinations. It requires further manual checking and labeling to clarify whether the failure in alignment is caused by LLM's hallucination.

Dear Reviewer KmBE,

Thank you for the time and patience you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work.

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require additional information or clarification. We are eager and ready to engage in further discussions to enhance and elevate our work.

Best regards and thanks,

Paper 8561 Authors