Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

We sincerely thank you for the reviewer's recognition on our work, and highly agree with the concerns and suggestions. The additional results as required are provided as follows.

Question:

It is definitely a must-to-ask question that how the performance of GPT-4-Vision on Q-Bench, which shall establish the baseline for commercial SotA MLLM.

Answer:

Thank you for your kind suggestion. We totally agree with the opinion of the reviewer that, the performance of GPT-4Vision (GPT-4V) is a must-to-ask question in our benchmark. Therefore, since the day it releases public use, we have been discovering the feasibility and testing the GPT-4V on three sub-tasks of Q-Bench. They are analyzed as follows:

1. The perception task [Feasible]

The task is based on Multi-Choice Questions (MCQ), with objectively-defined correct choices for each question. We primarily test with GPT-4V and make sure they are very good answering MCQs, and will directly give a choice A/B/C/D given the input. Therefore, we test it as follows:

Example:

[Image Upload] What level of blurriness does the background skyscrapers of this image have?

Choose between one of the options as follows:

A. Slight

B. Severe

C. Moderate

And GPT-4V returns C. Moderate for this question.

With this method, we evaluate the quantitative result of GPT-4V on the low-level perception task as follows (This is tested on the test subset of LLVisionQA, with 1495 image-question-answers. The answers in this subset will never be released to public, so as to avoid contamination and keep the fairest comparison among methods.)

As is shown in the table, GPT-4V can already reach the level of a non-expert human (who had not participated in related studies or jobs before), outperforming all open-source approaches by a large margin, but still notably behind a experienced human (who had participated in other related studies, but independent to our author team.)

Nevertheless, as the data used in GPT-4V is a black box, related data (low-level visual recognition, image quality assessment) may have been collected and used for training this powerful model. Thus, the comparison is kind of "not equal" to the open-source methods, which are confirmed without direct instruction tuning on low-level-related data. Instead, this exciting result points a hopeful future for open-source MLLMs to also reach the same ability, with specifically-designed tuning for low-level visual perception.

2. The description task. [Not Yet Feasible]

As this task needs GPT itself to evaluate and compare accuracy between two descriptions, it might not be feasible to test GPT-4V on this task in this stage, as some biases might happen (especially in comparison with other MLLMs). We will seek other feasible evaluation protocols that can test GPT-4V more reliably.

3. The assessment task. [Not Yet Feasible]

As is mentioned in our methodology, due to the requirement of a quantifiable output, the assessment task is based on next-word probabilities in an auto-completion-like task. Regretfully, at present, providing log probabilities of words are disabled for all GPT variants (including the initial versions), including both the web interface and the API. We look forward to seeing this feature return to GPT API one day and we will test its assessment performance then.

In summary, we have tested the low-level visual perception ability of GPT-4V, and validate its advantage towards open-source models (though still improvable to replace professional human). We will continue to figure out new strategies to evaluate GPT-4V in more low-level visual tasks in the future.

Thank you again for providing this advice to test GPT-4V on Q-Bench.

Best Regards,

Authors of the Bench

[Thread 3/4]

Question 2 (Q2)

As a benchmark paper, the author may update the recent SOTA MLLMs into the leadboards of this paper, e.g., QWen-VL [1], InternLM-XComposer [2], and LLaVA-1.5 [3].

Answer for Q2 (Part 2)

Updated Results for (A2): Description

Abbreviations for dimensions: comp: completeness, prec: precision, rele: relevance

[Thread 1/4]

We thank the reviewer for the recognition of our work's contributions, solidity of validations, and presentation.

We also sincerely consider the two suggestions as proposed by the reviewer:

1. Include synonym ensemble for image quality assessment (IQA). This can bring general improvements (up to 2.1%) on five top methods on the existing IQA benchmark (task A3 of Q-Bench), while the best synonym combination is not the same for all methods. We will recommend this strategy as an additional boost strategy in our paper. See this thread [Thread 1/4] for details.

2. Include more MLLMs into Q-Bench. Holding the same opinion with the reviewer (that a benchmark should be up-to-date), we keep tracking on MLLMs, and have internally expanded the benchmark to 15 open-source MLLMs. These internal results are updated here and will be update to our revised paper. See this thread [Thread 2/4] to [Thread 4/4] for the updated results.

Question 1 (Q1)

In the evaluation of LLDescribe, the softmax is calculated between good and poor. How about also considering their synonyms, e.g., great, excellent, fine, bad, low, etc? Here is a possible solution: (1) merge the logits of great, excellent, fine into good, and bad, low into poor. (2) calculate the final score with merged logits.

Answer to Q1

We sincerely thank the reviewer for providing this solution. In short, this can bring improvements.

As shown in the tables below, the results of the synonym ensemble on top-5 methods (InternLM-XComposer-VL, QWen-VL, LLaVA-v1.5-13B, mPLUG-Owl, and LLaVA-v1.5-7B) can in average lead to up to 2% accuracy improvement. We believe it is a very useful suggestion (esp. during application) to improve performance of MLLMs on IQA, and we will carefully record it in our paper.

Nevertheless, we also notice that different MLLMs perform best with different specific prompt combos, adding difficulties to us when evaluating new models. While good/poor (which is overall the best single word pair) is only in average 1.3% inferior to ensemble performance and shows the same inter-model ranking, we decide to keep the current strategy in Q-Bench to evaluate MLLMs, but set an independent chapter to recommend the ensemble strategy while applying MLLMs for IQA.

Results of different methods are as follows.

1. LLaVA-v1.5-13B (improves 2.1%)

2. LLaVA-v1.5-7B (improves 1.4%)

(see follow up)

[Thread 2/4]

Question 2 (Q2)

As a benchmark paper, the author may update the recent SOTA MLLMs into the leadboards of this paper, e.g., QWen-VL [1], InternLM-XComposer [2], and LLaVA-1.5 [3].

Answer for Q2 (Part 1)

Updated Results for (A1): Perception

In the updated version, we split the LLVisionQA set into dev set (half) and test set (half). The former will be released for future MLLMs to test locally and develop their methods, and the latter will not be released to avoid contamination (future methods can evaluate on test via our service).

1. Accuracies on dev

2. Accuracies on test

• GPT-4V and human:

GPT-4V is primarily a human without expertise on low-level visual perception. Please see responses to reviewer rhgp for more analysis.

• Open-source MLLMs:

[Thread 4/4]

Question 2 (Q2)

As a benchmark paper, the author may update the recent SOTA MLLMs into the leadboards of this paper, e.g., QWen-VL [1], InternLM-XComposer [2], and LLaVA-1.5 [3].

Answer for Q2 (Part 3)

Updated Results for (A3): Assessment

(following above)

3. mPLUG-Owl (improves 1.2%)

4. Qwen-VL (improves 1.7%)

5. InternLM-XComposer-VL (improves 0.1%)

We would like to again thank the reviewer for providing this solid strategy, which brings only negligible additional cost but can improve the accuracy of MLLMs on the image quality assessment (IQA) task. We believe this can help improve our work by including this as additional discussions, and contribute to the IQA community.

[Thread 1/3]

General Response

We would like to sincerely thank the reviewer for recognition on our motivation, organization, and experimental soundness. We also sincerely appreciate the profound questions as raised by the reviewers. We also agree that improving the truthworthiness of the evaluation is also very important, and we would like to carefully respond to the questions as follows.

Question 1 (Q1)

As the ChatGPT also suffers from hallucination, how can we ensure the reliability and confidence of the GPT-assisted evaluation?

Answer to Q1

We are also concerned with the accuracy of GPT-assisted evaluations. In our benchmark, the GPT-assisted evaluation is employed in both the perception (A1) and description (A2) task evaluation.

1.1. Reliability of GPT-assisted Evaluation for the Perception task (A1)

For the perception task, the GPT-assisted evaluation helps determines the correctness of the MLLMs’ responses to the multi-choice questions (MCQ). This is because some MLLMs might not respond as the required format of multiple choice answer. For example, they might respond to the prompt of “How’s the level of blur in the image? Choose between the following answers: A. Some blur, B. Not blurry at all, C. Very blurry.’” with “The image is severely blurry”. The answer is correct (C is the right choice) but not in the required format. GPT can help us precisely determine the correctness for such responses.

As answering MCQ can only be correct or incorrect without an intermediate state, this judgment is relatively simple and has fairly objective standards, that we can check the correctness of GPT evaluation. We notice that a single turn evaluation only results in 93.2% accuracy, which is not reliable enough. To solve this, we propose a 5-round voting protocol, that lets GPT to judge 5 times and take the majority vote as the evaluation result. It improves the evaluation accuracy to 98.4%, which can support the general conclusions in the Q-Bench.

We hope this can clarify the concerns of GPT evaluation reliability on the perception task (A1).

1.2. Discussion on GPT-assisted Evaluation for the Description task (A2)

For the description task (A2), we have to acknowledge that judging whether a description matches the gold description is a somewhat subjective process. Even when evaluated by humans, the scores rated for the MLLM descriptions are subject to individual differences.

Similar as the previous task, we also employ the 5-round GPT-assisted evaluation protocol, which average the score of five GPT-rated scores ($s_i, i=[0,1,2,3,4]$), each in one of the integers [0,1,2], into a floating point score ($s$).

$s = \frac{\sum_{=0}^{4} s_i}{5}$

We understand this is an imperfect way to evaluate the low-level descriptions (that may reduce hallucinations but cannot eliminate), yet this could be the most reliable and reproducible way at present. Guided by your valuable feedback, we will continue to explore how to design a more reliable evaluation protocol for the low-level visual description task in our follow-up works.

[Thread 3/3]

Question 3 (Q3)

As current models typically leverage large amounts of data, how can we avoid contamination of the evaluation dataset in the training set?

Answer to Q3 / Statements on Data Contamination

Same as the reviewer's opinion, we also believe that avoiding data leakage in a benchmark is crucial. Here as follows is a brief statement on potential data leakage in our benchmark.

We will add this report in the appendix of the revised paper.

In short, our benchmark contains three tasks, where the first two tasks (A1: perception and A2: description) are evaluated with our own datasets proposed with the paper. For these two tasks, the questions, answers, or low-level descriptions in the two datasets are not seen by any existing MLLMs. We will continue to keep the labels in half of LLVisionQA and full of LLDescribe private, to avoid being added into the training sets of any MLLMs. We hope that this measure will allow the Q-Bench to have long-term significance as an indicator of low-level visual abilities.

For the third task, (A3) assessment, the situation is a bit more complicated. We will discuss open-source models and close-source models separately.

For open-source models as tested, almost all of them have provided their technical reports, and as far as we know, no image quality assessment (IQA) dataset has participated in the multi-modality training stages of them. While knowledge about image quality assessment might be injected to them (e.g. a blurry image is a low quality image) during their pure-language training stages, we think this should not be regarded as data contamination for image quality assessment (IQA), because the images cannot be seen by a language model. Instead, they are important knowledge that helps them to better link particular visual attributes (blur) to human opinions (quality), which is one of the reasons why we would like to explore MLLMs for these tasks.

For close-source models, e.g. GPT-4V (mainly discussed in response to R3), we are not sure about their data contamination situation on existing datasets (i.e. IQA datasets for task A3 assessment). However, as they are also not included in the respective comparisons in task A3, this will not cause potential data contamination issues in the Q-Bench.

Considering the evidences above, we conducted the results of Q-Bench at present are not affected by known contaminations We will continue to pay attention to future data contamination issues and, in the future, differentiate between models on the leaderboard based on the potential risks of these issues.

[Thread 2/3]

Question 2 (Q2)

What is the difference between the proposed metric and perplexity (only considering good/bad)? Is the PPL equivalent to this new metric?

Answer to Q2

Thank you for this question regarding the proposed quantitative evaluation (assessment task, A3). In short, the main difference between the proposed metric and common perplexity (PPL) metric is that our metric is specially constrained to provide quantitative scores for images.

Main Difference: the Constraints

Considering the evaluation setting, we use MLLMs to generate responses starting with "The quality of the image is" to prompt them to answer related to image quality. The goal is to create a straightforward, quantifiable score, and avoid the non-quantifiable free-format sentences (e.g., "The quality of the image is blurry, with the vehicle's details totally invisible.").

To achieve this setting, we pose two constraints to the generation process:

1. Prediction Limitation: Only one subsequent token is predicted.
2. Output Restriction: The output is limited to a pair of opinion-related antonyms (e.g., "good" and "poor"), to avoid distractions from non-quality-indicative tokens like "very", "not", "blurry".

These constraints help us to limit an MLLM into a binary classifier, which helps us to obtain a score on visual quality, without requiring to train a new classification/regression head for MLLMs.

Equivalence Under the Constraints

Given the constraint, we denote the constrained logits as $\hat l = [l_{good}, l_{poor}]$, and then the constrained probability vector is as $\hat p = [\frac{e^{l_{good}}}{e^{l_{good}}+e^{l_{poor}}},\frac{e^{l_{poor}}}{e^{l_{good}}+e^{l_{poor}}}]$. Then, the proposed metric is to use $\hat p_0$ directly as the output quality score (as per Eq. 1 in our draft), and the PPL of label 'good' under our constraints is $-\log \hat p_0$. This is the negative logarithmic value of our proposed output quality score.

We opt for the $\hat p_0$ score since it falls within the [0,1] range and is monotonously related to the PPL-based quality score. This makes it more intuitive (image quality scores are bounded, and better images have higher scores) and aligned with our evaluation needs.

Summary

In conclusion, we leverage the text generation ability of MLLMs and use constraints to turn it into a binary classifier, which is specially designed for this task. The scores we obtained can be considered as nearly equivalent to PPL metric under the constraints.

We hope the proposed metric, as a constrained set perplexity-like metric, can bring the insights of using LM losses as metrics into new tasks.

Thank you very much for the extremely thorough responses and the revision of the paper. The quality of the paper has improved considerably, thus, I'd like to raise the rating to Accept. Great job!