Enhancing Descriptive Image Quality Assessment with a Large-scale Multi-modal Dataset

Thanks to the reviewer's valuable reviews and appreciation of our dataset contribution, comprehensive task paradigm, and analysis of model confidence. We address the concerns point-by-point below.

Q1. More evaluation of in-the-wild images.

We test our model on two in-the-wild datasets, SPAQ and KonIQ. The results are shown below and added to Table A6 of the revised version.

• First, directly evaluating our original EDQA on KonIQ and SPAQ, we achieve comparable results with previous score-centered IQA methods. Note that our EDQA is only trained to compare images with similar contents, as shown in Task 2 of Fig. 2. However, to calculate the quality scores of these two datasets, our model needs to compare images with different contents, since all images in these two datasets contain different contents.
• Second, after finetuning our EDQA on real-world IQA datasets, we outperform baseline methods. We adopt the same training and evaluation setting of baseline methods, i.e., trained on KonIQ then evaluated on SPAQ, and vice versa. To retrain, we sample image pairs from the real-world IQA dataset to formulate instant rating pairs.
• Additionally, we finetune Q-Align using the same LoRA parameters as our model to only tune 0.24% LLM parameters. The performance of Q-Align is slightly inferior to EDQA. This shows that the advantage of Q-Align comes from model size, i.e., fully tuning the whole LLM.

Q2. Will the performance drop if switching the order of two input images?

Thanks for the suggestion. The table below shows that our model achieves more than 0.90 consistency if switching the order of two input images. Also, our comparison accuracy and score correlation are both much higher than Q-Instruct and GPT-4V.

We test our model on the fine-grained dataset released by [R1]. We follow [R1] to report the consistency / accuracy / correlation as metrics, where consistency means the consistency in changing the order of input images. The results of Q-Instruct and GPT-4V are borrowed from [R1]. These results show that our model is robust to the order of input images.

[R1] 2AFC Prompting of Large Multimodal Models for Image Quality Assessment, TCSVT 2024.

Also, we provide the statistics of our model's confidence. The results below show that the confidence of consistent prediction is much higher than inconsistent prediction, which reflects the self-evaluation ability of our model.

We have added the results into Table A11, A12 and L1170 (Sec. D.2).

Q3. The maximum resolution.

Thanks for pointing out this confusion. We have clarified this in L1002 (Sec. C) of the revised version. Actually, without the vision abstractor, the maximum resolution is limited to 672. However, with the vision abstractor, we can process images with much larger resolutions (up to $2500 \times 2500$). In our experiments, the maximum image resolution is $1092 \times 1456$, thus the resolutions of all images are retained.

Q4. Results in real-world images and extreme conditions.

Thanks for your suggestion. In the revised version, we evaluate our model's ability to assess in-the-wild real images in Fig. A14. In Fig. A9, we also assess our model's performance on images with severe distortions. Our model performs well in such extreme cases.

Thanks for the reviewer's efforts and appreciation of our comprehensive task paradigm, effectiveness of dataset construction method, application potential, and strong performance. Below we address the main concerns.

Q1. Training and inference costs.

Thanks for the suggestion. The training and inference costs are presented below and added in Sec. C.1.

• Training costs. The trainable parameters are 70M (54M for vision abstractor and 16M for LoRA), constituting only 0.98% of the total parameters (7.11B). The model is trained on 8 GPUs (RTX A6000). The training is completed in around 22 hours.

• Inference costs. The inference latency depends on the response length and it is tested on a single RTX A6000 GPU. For example, for brief tasks task with the short answer prompt (about 2.92 words), the inference time stands at approximately 2.23s / batch=32, transformed to 0.07s / sample. For the assessment reasoning task (75.84 words on average), the inference time is 22.97s / batch=32 (i.e., 0.72s per response). Despite the incorporation of resource-intensive MLLMs, EDQA remains deployable on a single consumer GPU (e.g., RTX3090).

Q2. Beneficial relationships between multi-modal task and instant rating task.

Thanks for the suggestion. We conduct experiments to study the relationships between multi-modal tasks (here we use comparison reasoning) and instant rating task.

• Comparison reasoning task improves the performance on four instant rating datasets, but decreases the results on two datasets. Averagely, the comparison reasoning task helps the instant rating performance.

• Instant rating task stably improves the performance on the comparison reasoning task.

We have added these results to Table A10 and Sec. D.2.

Q7. Results of Q-Instruct and Co-Instruct with BLEU and ROUGE-L metrics.

Thanks for the suggestion. We have added the results of Q-Instruct and Co-Instruct with BLEU and ROUGE-L metrics in Table 6. The assessment reasoning results in the non-reference setting are shown in the table below. See Table 6 in the revised version for all results.

Q8. Limitations.

Thanks. We have discussed the limitations in Sec. 6. We also provide some possible solutions to address these limitations.

First, the fine-grained abilities requiring more high-level perception skills are still unsatisfactory. One possible way is to take the segmentation model (e.g., SAM) to add various distortions to different regions.

Second, for the convenience of evaluating, analyzing, and improving the model, we mainly focus on standardized answers. To achieve more flexible responses, LLM rewriting and human annotation can be introduced to increase linguistic diversity during dataset construction.

Third, whether our assessment can be used as feedback to improve the quality of generation or restoration models is still under-explored. Existing image restoration models (e.g., SUPIR [R1]) have demonstrated that diffusion-based restoration models can adjust some low-level textures according to the textual inputs. Therefore, one possible application is to co-train a diffusion-based restoration model with both images and quality-assessed texts to enhance the restoration quality.

[R1] Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild, CVPR2024

We thank the reviewer for the reviewer's work and appreciation of our novelty, dataset contribution, and clear writing. Below we address the main concerns point-by-point.

Q1.1 Details of the OOD setting.

To achieve an OOD distortion identification, for one category of distortion (e.g., compression), we train the model on some sub-categories (JPEG compression), and then assess on other sub-categories (JPEG2000 compression). The detailed split of the training and validation distortions are given in Table A2 of the revised version.

Q1.2 Results of LIQE.

Thanks for the suggestion. We have added the results of LIQE in Table 3. LIQE achieves comparable results with Co-Instruct, falling behind our EDQA. Note that LIQE only accepts one image as input, and thus can only be assessed in the non-reference setting.

Q2. Results on KonIQ and SPAQ.

Thanks for the suggestion. We have provided the results on in-the-wild KonIQ and SPAQ datasets in Q1 to reviewer 5EhH and Table A6. The results demonstrate the effectiveness of our method.

Q3.1 Template-like responses.

The template-like descriptions come from the alignment to the formats of training samples. When constructing detailed responses, we include three aspects of information: (a) brief content description, (b) distortion analysis, and (c) overall judgment. After training, the model tends to predict the three aspects of information, forming template-like descriptions.

In the early stages of VLM-based IQA methods, if the descriptions are fully flexible, defining desirable target descriptions will be extremely hard. Because of this, in the current version, we mainly focus on standardized answers, which help easily evaluate, analyze, and improve the model. More flexible responses could be an interesting future exploration.

Q3.2 Results on Q-Bench.

Q-Bench primarily consists of multiple choice questions, while our model emphasizes descriptive ability, creating a disparity for direct assessment. To demonstrate that our dataset can enhance performance on Q-Bench, we select another open-source dataset containing multiple choice questions, Q-Instruct. Co-training on Q-Instruct and our dataset consistently outperforms training solely on Q-Instruct, highlighting the benefits of our dataset. Additionally, the results of LLaVA trained on Q-Instruct, as reported in Q-Instruct, are provided for reference.

Q4. Details of the method for full-reference score regression.

The full-reference score regression results are evaluated on the PIPAL, KADID, TID2013, and CSIQ datasets. These datasets include high-quality reference images and their distorted versions under various distortions. We calculate the win rate of an image against others to determine its quality score.

Specifically, for one image, A, we randomly sample comparison candidates, such as B, C, D, etc., which share the same content as A but have different distortions.

Image A is then compared pairwise with each of its comparison candidates (B, C, D, etc.). In the full-reference setting, three images including the reference image, Image A, and one compared candidate are together input into our model for comparison.

Finally, the win rate of Image A against its compared candidates is calculated as its quality score.

We have added these details in L1059 (Sec. D.1) of the revised version.

Q5. Specific fine-tuning procedures?

No. The model is trained with the commonly adopted next-token prediction loss on our collected datasets. Remind that Table A4 in the original submission becomes Table A5 in the revised version.

Q6. Suggestions to improve the comparison efficiency.

We mainly focus on accuracy in this work, but there are some ways to improve efficiency, like a dynamic Elo rating method.

First, score initialization. Each image is initially assigned an equal score.

Second, random pair comparison. Two images are randomly selected for comparison. The winner's score increases, and the loser's score decreases using the Elo rating system.

Third, dynamic grouping. Images with similar scores are grouped based on a threshold, minimizing unnecessary comparisons between highly divergent pairs.

Finally, intra-group comparisons. Comparisons are performed within each group, and scores are updated dynamically.

The last two steps repeat until scores stabilize or a pre-defined number of iterations is reached. This method can narrow down top-performing images while reducing computational overhead.

Thanks for the reviewer's efforts and appreciation of our dataset contribution, comprehensive task paradigm, and potential for confidence estimation. We address the main concerns below.

Reply to a wrong submission?

The reviewer referred to our method as "DepictQA-Wild", whereas our method is called "EDQA". Additionally, there are other details in the review that do not align with our paper. It seems possible that the reviewer may have responded to the wrong submission/paper. Could we kindly ask the reviewer to double-check if the comments are indeed intended for this submission?

Q1. Well solved in previous work?

The reviewer claims that the pair-wise quality comparison task has been well addressed in Co-Instruct, which contradicts the experimental results.

• As shown in Table 4, the comparison ability of Co-Instruct (52.0) is still largely inferior to PSNR (80.2) and SSIM (78.8) in the full-reference setting. Also, in the non-reference setting, Co-Instruct (69.8) falls behind MUSIQ (76.9) and GPT-4V (75.2).
• As given in Table 5, Co-Instruct's comparison reasoning ability (37.6 / 48.1) is also largely inferior to GPT-4V (66.2 / 60.3).

These results prove that the pair-wise quality comparison task is still far from the reviewer-claimed "well solved".

Q2. Should a high-quality dataset be considered as a contribution?

The review claims that the instruction dataset "is somehow not a considerable contribution", which contradicts the literature on IQA. Most previous VLM-based IQA methods focus on dataset reconstruction, including Q-Instruct (Wu et al., CVPR2024), Co-Instruct (Wu et al., ECCV2024), and DepictQA (You et al., ECCV2024). This is because dataset quality is critical for VLM training, and the current VLM-based IQA datasets are still far from ideal and become a bottleneck in this area.

Moreover, creating a high-quality dataset is not as trivial as simply combining more data. We need to carefully design the distortion library, the brief-detail combined dataset framework, and the prompt method. With our comprehensive distortion library and ground-truth-informed generation process, we successfully synthesize accurate training data.

Q3. Comparison number.

First, the key to the quality comparison task is the comparison accuracy, rather than the comparison number of one-time input. As stated in Q1, the pair-wise comparison ability of Co-Instruct is still largely inferior to PSNR and SSIM in the full-reference setting. Discussing comparison numbers regardless of comparison accuracy is unreasonable.

Second, there are several ways to extend the pair-wise comparison to multi-image comparison, such as Elo rating, round robin, ranked pairs, and so on.

Dear Reviewer BK5E,

Thank you once again for your time and valuable comments.

We have provided detailed responses to your concerns and believe that these address all the issues you raised. Please let us know if any aspects of our work remain unclear. We understand your time is valuable, and we greatly appreciate it if you could review our responses at your earliest convenience.

In particular, regarding the discrepancies in method names and details that do not align with our submitted manuscript, we sincerely hope you clarify this misunderstanding.

Thank you for your kind consideration.

Sincerely,

Authors