Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning

Thank you for your detailed response.

For Q1 and Q2, my concerns have been addressed. Regarding Q3, I am now convinced by the overall contribution. Q4 and Q5 were due to my misunderstandings, and I appreciate your clarifications.

This paper provides a thorough exploration of image captioning. It introduces an evaluation framework that effectively bridges the gap between existing metrics and the current status of vision-language research. I am also impressed by the performance of the simple yet effective preference optimization method. I believe this work makes a solid contribution to the field and will be beneficial for the ongoing development of vision-language models. I will increase my rating to a 6.

Q4. Experimental settings

A4. For the main experiments, we chose LLaVA-Onevision-7B because it is the most capable model recently available, allowing us to showcase the maximum potential of our method. To demonstrate the generalization capability of our proposed method, we also included a variety of VLMs in different experiments, as shown in Table 6. In the ablation study "Preference Data for Reward Model," we utilized LLaVA-1.5-7B to ensure a fair comparison with other models and datasets, maintaining consistency with common baselines. For the "Source of Response" experiments, we employed LLaVA-1.5-13B to ensure diversity and robustness by using responses from both weaker and stronger VLMs. This varied selection helps us better understand how our method performs across different contexts and strengths, providing a comprehensive evaluation of its robustness and generalizability.

Q5. Motivation of including non-descriptive captions

A5. We investigated the influence of non-descriptive elements in DCScore on its alignment with human judgment, as follows:

The results show that including non-descriptive elements during detailed image caption evaluation achieves a higher correlation with human judgment. This improvement occurs because non-descriptive elements, such as background details and inferred information, provide additional context that leads to a more comprehensive understanding of the image content. Consequently, by including these elements, DCScore captures subtle nuances and implicit information critical for fully understanding the image, thus more closely aligning with human judgment.

Therefore, incorporating non-descriptive elements into DCScore provides a more accurate and reliable evaluation of generated captions, addressing a gap in previous metrics that have overlooked these contextual details.

Q1. Human annotators selection & annotation criteria

A1. For the selection of human annotators, we chose individuals with high expertise and language proficiency to ensure objectivity and impartiality. Annotators were familiarized with the caption scoring task but were blinded to the specific study objectives to avoid bias.

Regarding the annotation criteria, detailed guidelines were provided to ensure consistency. Each caption was scored on a scale of 0-4, and the average of three annotators' scores was used as the final score. The scoring criteria are as follows:

• 4: Comprehensive and accurate caption with no key information missed and error-free language.
• 3: Caption meets main requirements with minor issues like typos or missing background elements.
• 2: Caption meets main requirements but has significant errors such as hallucinations or positional inaccuracies.
• 1: Caption has many problems but is slightly better compared to a 0, with errors not exceeding 70%.
• 0: Caption has severe hallucinations, serious grammatical errors, and confused logic, making it unusable.

We will include the process of human annotator selection and criteria for annotating the caption in the revised manuscript.

Q2. Omission captioning details in ground-truth annotation

A2. As we explained to Reviewer DgKk, we compensate for the missing fine-grained details in the ground-truth captions that are present in the model's predictions when computing recall score $s_r$ for DCScore. In concrete, in step 3, we would use GPT-4o to identify the units that are correct but not presented in reference captions. By incorporating the correct additional information from the model's captions into the ground-truth caption, we make the evaluation metric more robust to the omission details in ground-truth captions.

Q3. Restatement of the contribution

A3. Our main contributions are summarized as follows:

• Novel Image Captioning Metric and Benchmark: We present DeCapBench alongside the DCScore metric designed specifically to evaluate detailed image captioning tasks. Unlike current mainstream captioning evaluation datasets that tend to focus on short-caption evaluations, DeCapBench is developed to handle the complexity and richness of detailed image captions. Current evaluation methods often overlook the detailed context and potential hallucinations present in longer, more descriptive captions. By providing a metric that assesses both the quality and hallucination degree of these detailed captions, we fill a significant gap identified by Reviewer Mger.
• New Fine-grained Feedback Collection Method: While generating preference data is a consensus in the research community, the methodology to collect such data significantly influences the accuracy of preference pairs. Therefore, we introduce FeedQuill to automatically collect high-quality preference data by considering both hallucination and richness in the generated captions. This method is scalable, as highlighted by Reviewer A3YP. Additionally, the preference data collected by FeedQuill generalizes much better than other methods, as supported by Reviewer DgKk.
• Comprehensive Experiments: To demonstrate the effectiveness and generalization ability of the proposed FeedQuill, we provide extensive experiments on a diverse range of downstream tasks and across different types of VLMs. The results show reduced hallucinations, superior performance in visual chat compared to GPT-4v, and better detailed image captioning capabilities than GPT-4o.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Thank you for your detailed response.

For Q1 and Q2, my concerns have been addressed. Regarding Q3, I am now convinced by the overall contribution. Q4 and Q5 were due to my misunderstandings, and I appreciate your clarifications.

This paper provides a thorough exploration of image captioning. It introduces an evaluation framework that effectively bridges the gap between existing metrics and the current status of vision-language research. I am also impressed by the performance of the simple yet effective preference optimization method. I believe this work makes a solid contribution to the field and will be beneficial for the ongoing development of vision-language models. I will increase my rating to a 6.

Q3. Sample selection of DeCapBench

A3. DeCapBench's images and captions are sourced from ImageInWords [1], which provides only 400 images with high-quality human-written captions. While 400 images may seem like a small subset, these are the only ones released and are highly valuable due to their exceptional quality and the costly, time-intensive process of creating them. Each caption is hyper-detailed and meticulously crafted by well-educated human annotators, taking approximately 1800 seconds to produce [1]. This high quality is crucial for evaluating detailed image descriptions. Our findings in Table 2 show that higher-quality ground-truth captions lead to better alignment with human judgment, underlining the importance of using high-quality captions.

On the other hand, we tested the performance variance when varying the size of DeCapBench. Specifically, we ran the evaluation multiple times and computed the standard deviation of the performance. The results demonstrate that increasing the sample size from 100 to 400 significantly reduces the standard deviation from 1.27 to 0.07, leading to more stable evaluations. Additionally, we observed that other VLM evaluation benchmarks such as MIABench [2] and HallusionBench [3] employ similar sample sizes, suggesting that a sample size of 400 is a reasonable choice for achieving reliable evaluation outcomes.

[1] Garg, Roopal, et al. "ImageInWords: Unlocking Hyper-Detailed Image Descriptions." arXiv preprint arXiv:2405.02793 (2024).

[2] Qian, Yusu, et al. "Mia-bench: Towards better instruction following evaluation of multimodal llms." arXiv preprint arXiv:2407.01509 (2024).

[3] Guan, Tianrui, et al. "HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q4. Comparison to HallusionBench

A4. HallusionBench is designed for evaluating image-context reasoning. It differs from our DeCapBench in the following aspects:

1. Types of Hallucinations Evaluated: DeCapBench evaluates visual hallucinations in image captions, focusing on aspects such as object existence, correctness of attributes, and relationships. Visual hallucinations refer to the inclusion of objects, attributes, or relationships that do not exist in the image. In contrast, HallusionBench primarily addresses language hallucinations, which stem from overreliance on language priors rather than visual context. Additionally, HallusionBench measures visual illusions, which denote the misinterpretation of accurate visual information—where an existing object or relationship is incorrectly perceived.
2. Types of Evaluation Tasks: DeCapBench evaluates the quality of detailed image captions across various aspects, not limited to hallucinations. It assesses the accuracy and richness of the captions. On the other hand, HallusionBench is designed to evaluate the image-context reasoning capability within a QA format, focusing on how well the reasoning aligns with image context rather than the captioning quality.
3. Coverage of Evaluation: DeCapBench not only evaluates hallucinations in the generated image captions but also assesses comprehensiveness using hyper-detailed human-curated captions. This means it looks at how well the captions cover all relevant details in the image. In contrast, HallusionBench focuses solely on evaluating "language hallucinations" and "visual illusions," specializing in hallucination evaluation without assessing the overall comprehensiveness of responses.

Q5. Fine-tuning with PPO train datasets for Table 3

A5. Instead of instruction fine-tuning, our method employs Proximal Policy Optimization (PPO), a reinforcement learning algorithm. During the PPO training process, the model generates responses on-the-fly based on input prompts rather than relying on pre-existing ground-truth annotations. As a result, it is not feasible to directly fine-tune on PPO training data using cross-entropy loss due to the absence of ground-truth annotations. For reference, we also present the results of rejection sampling, where the response annotations are generated by selecting the best-of-N responses from the base model. These results are included below:

Q6. Data source for PPO training

A6. We randomly sample images and caption prompts from the datasets listed in Table 14 of the Appendix. The image sources include MSCOCO, OpenImages, and ShareGPT4V; no annotated responses are used.

Q7. Necessity of reward model $R_{\phi_r}$

A7. The preference score $c_r$ is indeed a scalar, produced by counting primitive information units decomposed by the model's responses using an LLM. Directly integrating response decomposition for primitive information units with LLM generation into PPO training, although potentially more accurate, is time-consuming. The time complexity for LLM generation is O(N), while that for the reward model is O(1). Hence, we train the reward model $R_{\phi_r}$ to substitute the direct generation of $c_r$ for each response, improving efficiency in the training process. Furthermore, we have evaluated the accuracy of the reward model $R_{\phi_r}$ in predicting the comparative relationship of $c_r$. It achieves an accuracy of 96.3% on pairwise comparisons, demonstrating its reliability in the training process. This high accuracy indicates that $R_{\phi_r}$ can effectively approximate the preference scores $c_r$, ensuring that the training process remains both efficient and accurate.

Q1. Motivation of including non-descriptive captions

A1. We investigated the influence of non-descriptive elements in DCScore on its alignment with human judgment, as follows:

The results show that including non-descriptive elements during detailed image caption evaluation achieves a higher correlation with human judgment. This improvement occurs because non-descriptive elements, such as background details and inferred information, provide additional context that leads to a more comprehensive understanding of the image content. Consequently, by including these elements, DCScore captures subtle nuances and implicit information critical for fully understanding the image, thus more closely aligning with human judgment.

Therefore, incorporating non-descriptive elements into DCScore provides a more accurate and reliable evaluation of generated captions, addressing a gap in previous metrics that have overlooked these contextual details.

Q2. - Open-sourced alternatives to GPT-4o in evaluation

A2. Our evaluation metric, DCScore, can indeed be adapted to open-source VLMs. To demonstrate this, we conducted an experiment using Qwen2-VL, a recently released VLM in the open-source community. We adopted the same evaluation prompts for DCScore without any special tuning and compared the results with those obtained using GPT-4o. The results of using Qwen2-VL in terms of human consistency are as follows:

The comparison indicates that while GPT-4o achieves a higher degree of human consistency, Qwen2-VL also performs robustly, demonstrating the flexibility and adaptability of DCScore to different VLMs. Furthermore, Qwen2-VL's performance with DCScore remains superior compared to other traditional metrics, showcasing the metric's robustness and adaptability. Notably, adapting DCScore to open-source models like Qwen2-VL allows for broader accessibility and cost-efficiency, without significantly compromising the reliability of the evaluation process.

Q6.. Motivation of LLaVA-series models for experiments and performance on other VLMs

A6. To demonstrate the effectiveness of our proposed method, we have applied it to InternVL2-8B model with preference optimization. The results are presented as follows:

As we can observe in the table, our method still achieves performance gain on several benchmarks, showing the generalization of our method. We want to further emphasize that we chose the LLaVA series (1.5/1.6/OneVision) as the base models because they encompass a divserse range of VLMs, differing in terms of (1) training data composition, (2) image processing strategy (Pad / AnyRes), (3) vision backbones (CLIP / SigLIP), and (4) LLM backbones (Vicuna / Qwen2). This diversity allows us to comprehensively evaluate the effectiveness and wide applicability of our proposed methodology. Moreover, these different LLaVA models are trained under varying settings, resulting in significantly different capabilities: LLaVA-OneVision is the latest and most capable model in the LLaVA series, comparable to InternVL2-8B [7], while LLaVA-1.6 is comparable to VILA-7B [8].

[7] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[8] Lin, Ji, et al. "Vila: On pre-training for visual language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q7. Minor Questions

A7. We address these minor questions in the following and will present them more clear in the revised manuscript.

• Preference Data Size: We use 200k preference pairs for the last row of Table 2 as stated in the Appendix A.2.1.
• Statement in Line 319: Yes, the short responses are less-informative, but tend to have fewer hallucinations. Lengthy responses have a higher probability of involving hallucination.
• Typos: Thanks for pointing out. We will correct the typos (e.g. Line 334) in the revised manuscript.

Q4. Sample selection of DeCapBench

A4. DeCapBench's images and captions are sourced from ImageInWords [1], which provides only 400 images with high-quality human-written captions. While 400 images may seem like a small subset, these are the only ones released and are highly valuable due to their exceptional quality and the costly, time-intensive process of creating them. Each caption is hyper-detailed and meticulously crafted by well-educated human annotators, taking approximately 1800 seconds to produce [1]. This high quality is crucial for evaluating detailed image descriptions. Our findings in Table 2 show that higher-quality ground-truth captions lead to better alignment with human judgment, underlining the importance of using high-quality captions.

On the other hand, we tested the performance variance when varying the size of DeCapBench. Specifically, we ran the evaluation multiple times and computed the standard deviation of the performance. The results demonstrate that increasing the sample size from 100 to 400 significantly reduces the standard deviation from 1.27 to 0.07, leading to more stable evaluations. Additionally, we observed that other VLM evaluation benchmarks such as MIABench [2] and HallusionBench [3] employ similar sample sizes, suggesting that a sample size of 400 is a reasonable choice for achieving reliable evaluation outcomes.

[1] Garg, Roopal, et al. "ImageInWords: Unlocking Hyper-Detailed Image Descriptions." arXiv preprint arXiv:2405.02793 (2024).

[2] Qian, Yusu, et al. "Mia-bench: Towards better instruction following evaluation of multimodal llms." arXiv preprint arXiv:2407.01509 (2024).

[3] Guan, Tianrui, et al. "HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q5. Difference of preference data collection method and reliance

A5. To address the issues of bias and unreliability in existing preference data collection methods like VLFeedback [4] and LLaVA-Hound [5], our approach differs in the following aspects:

1. Approach of Generating Preference: VLFeedback and LLaVA-Hound generate preference signals by presenting two candidate responses simultaneously and directly asking the VLM to choose the preferred response. This approach relies on the VLM's holistic judgment, which can introduce biases. In contrast, our method decomposes each response into several primitive information units and uses VLMs to verify the correctness of each unit separately. We then aggregate the fraction of correct units as a score to form the preference pairs. This decomposition and verification mechanism reduces the risk of bias and unreliability by focusing on smaller, verifiable units of information rather than a single, holistic judgment.
2. Reliability of Generated Preference: Directly prompting VLM for holistic preference judgment by giving two responses simultaneously is susceptible to language preference or positional biases, as demonstrated in [6]. In contrast, by breaking down responses into smaller units and verifying each unit separately, our approach adds an extra layer of granularity and rigor to the verification process. This ensures that the preference pairs formed are based on more precise and reliable judgments, thus mitigating the risk of bias and unreliability that might stem from using a single model's overall preference judgment. To further demonstrate this, we manually annotated 156 preference pairs for validation. The results demonstrate that our verification-based method achieves 88.5% accuracy while the VLM holistic preference-based method only achieves 58.97%, highlighting the reliability of our preference collection approach.

[4] Li, Lei, et al. "VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment." arXiv preprint arXiv:2410.09421 (2024).

[5] Zhang, Ruohong, et al. "Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward." arXiv preprint arXiv:2404.01258 (2024).

[6] Shi, Lin, et al. "Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms." arXiv preprint arXiv:2406.07791 (2024).

Q1. Human annotators selection & annotation criteria

A1. For the selection of human annotators, we chose individuals with high expertise and language proficiency to ensure objectivity and impartiality. Annotators were familiarized with the caption scoring task but were blinded to the specific study objectives to avoid bias.

Regarding the annotation criteria, detailed guidelines were provided to ensure consistency. Each caption was scored on a scale of 0-4, and the average of three annotators' scores was used as the final score. The scoring criteria are as follows:

• 4: Comprehensive and accurate caption with no key information missed and error-free language.
• 3: Caption meets main requirements with minor issues like typos or missing background elements.
• 2: Caption meets main requirements but has significant errors such as hallucinations or positional inaccuracies.
• 1: Caption has many problems but is slightly better compared to a 0, with errors not exceeding 70%.
• 0: Caption has severe hallucinations, serious grammatical errors, and confused logic, making it unusable.

We will include the process of human annotator selection and criteria for annotating the caption in the revised manuscript.

Q2. Decomposer for generated captions & ground-truth captions

A2. It is feasible to use the same decomposers for both generated and ground-truth captions. However, we opted to utilize human experts for decomposing ground-truth captions for two main reasons:

1. Accuracy: Using an LLM to decompose ground-truth captions might introduce minor errors.
2. Efficiency: Ground-truth captions are constant, allowing human experts to decompose them only once, which is less costly.

Additionally, we have tested using an LLM to decompose ground-truth captions and observed that it achieved a Pearson correlation of 0.9279 with decompositions performed by human experts in terms of DCScore. Meanwhile, using both LLM decomposed in model and ground-truth captions, the correlation is demonstrated as follows, which shows the superiority of using human experts for decomposing ground-truth captions.

Q3. Functionality of step 3 in DCScore

A3. To clarify, $\mathcal{P}_{true}$ represents the set of all correct primitive information units in the predicted caption, while $\mathcal{Q}$ denotes the common set of correct primitive information units found in both the predicted caption and the ground-truth caption. Therefore, the goal of this step is to compensate for the missing fine-grained details in the ground-truth captions that are present in the model's predictions. Human annotations often omit some of these details, and by incorporating the correct additional information from the model's captions into the ground-truth caption, we make the evaluation metric more robust to these omissions.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?