We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.

We sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:

Q1. Do the authors have some experimental evidence to prove that after CapMAS correction, the new caption is on par with or better than the original caption besides the hallucination degree?

A1. We sincerely thank the reviewer for suggesting this experiment, which allowed us to highlight a new strength of CapMAS in terms of fluency. Due to space limitations, we regret that we cannot provide a detailed response here, and kindly ask you to refer to our response A2 to reviewer quCw for further details.

Q2. Thus, one wonders if a truly powerful MLLM with superb long-context understanding capability is employed, how much gain would CapMAS bring?

A2. CapMAS is designed to improve the captioning performance of MLLMs, which currently have limited long-context capabilities and are prone to hallucinations. As such, its effect may be less pronounced for future MLLMs with powerful capabilities. However, it is important to highlight that even for GPT-4V, one of the current SOTA MLLMs, we were able to improve factuality by over 5% points using CapMAS—without sacrificing coverage—by leveraging relatively weaker open-source models. Given that today’s best MLLMs still fall short of a truly reliable MLLM, our work carries meaningful implications, particularly for visually impaired users.

Q3. How does the corrector LLM take as input the $\pi$-thresholding results and correct the original caption?

A3. It can be easily understood by referring to the corrector LLM's prompt template:

system:
I want to create a caption that includes only facts. Please help me correct the given caption.
The given caption contains things that are not true. Based on the given FACTS and NON-FACTS,
remove the non-factual elements from the caption.

user:
Caption: {caption}
FACTS:
{propositions classified as non-hallucinatory}
NON-FACTS:
{propositions classified as hallucinatory}

Q4. I wonder have the authors check if the hallucination scores would continue to improve if CapMAS is applied iteratively?

A4. CapMAS is designed to remove detected hallucinations rather than correct them. This design choice is based on our observation that attempts at correction often lead to the introduction of new hallucinations. As a result, applying CapMAS iteratively to a single caption has an effect equivalent to applying it once with a lower threshold $\pi$.

Instead, we demonstrate that applying CapMAS individually to multiple captions for a single image and subsequently summarizing the results can lead to an overall improvement in the caption quality for that image.

Table B: Results of LLaVA-Next on a subset of IIW400

Table B demonstrates that the overall quality of the resulting captions improves as the number of CapMAS applications increases.

Q5. I checked Appendix D and found that a larger threshold generally lead to better performances. I wonder, will it eventually plateau?

A5. In CapMAS, as the value of $\pi$ increases, the criterion for judging a proposition as true becomes more relaxed. This results in improved coverage but also causes more propositions to be classified as true, thereby introducing more hallucinations and lowering factuality. Consequently, in Appendix D, further increases in $\pi$ lead to a significant drop in the factuality score, ultimately degrading the overall quality of the captions. We summarize the results of additional experiments in the following table:

Table C: Ablation results on $\pi$ using LLaVA-NeXT-7B

Q6. Could this strategy be interpreted as some kind of inference-time scaling?

A6. Yes, CapMAS can indeed be considered an inference-time scaling strategy, and the experimental results, including those in Table B, support this interpretation. Additionally, we compared CapMAS with Self-Refine, another inference-time scaling method. CapMAS achieves significantly better performance while requiring a comparable level of cost. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our responses A3 and A4 to reviewer bFJq for more information.

We have additional experimental results that we were unable to include here. To help us share them, we kindly ask you to click “Rebuttal Comment” to allow us to leave a supplementary comment.

We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.

We sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:

Q1. The claim on L346 (Sec 4.3) that CapMAS exhibits a factuality-coverage tradeoff is somewhat over-stated.

A1. CapMAS is designed to remove detected hallucinations rather than correct them. This design choice is based on our observation that attempts at correction often lead to the introduction of new hallucinations. As CapMAS may remove not only hallucinations but also factual content, it inherently involves a trade-off between factuality and coverage, as discussed in Section 4.3.

Nevertheless, through inference-time scaling, CapMAS can improve the coverage of captions while maintaining a certain level of factuality. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our response A4 to reviewer fLiQ for more information.

Q2. I would like to see some experiments on a wider set of images.

A2. Using benchmarks like COCO for evaluating detailed image captioning can lead to misleading conclusions:

1. Short captions in COCO bias evaluation metrics to favor simpler captions over accurate, detailed ones.
2. Reference-free metrics, intended to remove this bias, often introduce another bias stemming from the evaluation model itself, which can significantly affect the scores [1].

We believe reliable evaluation requires detailed human supervision. To address your concern, we conducted additional experiments on DOCCI. Due to space limitations, we kindly ask you to refer to our response A2 to reviewer QHjW for details, which confirm CapMAS is effective on DOCCI as well.

[1] LLM Evaluators Recognize and Favor Their Own Generations

Q3. Is it necessary to break the caption down into atomic points before re-captioning, or is just asking the external VLM to re-caption sufficient?

A3. CapMAS can be understood as an inference-time scaling strategy. Motivated by the reviewer’s comment, we tested the effectiveness of inference-time scaling via Self-Refine (SR) [2]. Since validating and revising one's output requires advanced reasoning capabilities, we conducted the experiments using GPT-4V. For SR, we used the following prompt:

You are given an image and its corresponding caption.
Your task is to:
1. Analyze the image and compare it with the caption.
2. Identify and correct any factual or descriptive errors in the caption based on the image.
3. Refine the caption for clarity, correctness, and completeness — even if the original caption is mostly accurate.

Show your reasoning and then provide a final refined caption.

Table A shows that having the model revise its own captions iteratively does not lead to better results.

[2] SELF-REFINE: Iterative Refinement with Self-Feedback

Q4. There's not really any explanation or discussion of the efficiency of the model

A4. CapMAS is an approach that achieves better results by incurring additional cost at inference time. Although it involves a multi-model pipeline, CapMAS can offer a better cost-performance trade-off than Self-Refine (SR) methods for the following reasons:

1. Most of CapMAS’s cost lies in the final step, where the corrector LLM generates a refined caption based on the initial caption and the True/False classification results of the propositions. The decomposition step involves shorter sequences, and the MLLM used for proposition classification can process them in parallel, generating only a single token (True or False) per proposition. SR processes long sequences that include the original caption, detailed feedback, and the refined caption. Considering the length and complexity of the feedback [2], CapMAS and SR can be seen as comparable in cost.
2. As shown in Table 4, CapMAS’s performance does not heavily depend on the capability of the LLM used. This suggests that the LLM-related cost in the pipeline could potentially be further reduced.

Q5. I couldn't find the dataset used for evaluation in Tables 4 and 5, is this on IIW-400 or the DOCCI dataset?

A5. For Tables 4 and 5, we used IIW-400, based on findings [3] indicating that IIW-400 serves as a better reference caption set than DOCCI. As noted in A2, our additional experimental results confirm that CapMAS is effective on both IIW-400 and DOCCI.

[3] ImageInWords: Unlocking Hyper-Detailed Image Descriptions

We have additional results and discussions that we were unable to include here. We kindly ask you to click “Rebuttal Comment” to allow us to add a comment.

We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets.

We sincerely thank you for your thorough review. We have made our best efforts to address your remaining concerns as follows:

Q1. They also proposed a new metric for measuring coverage, but did not compare it to any prior metrics for this purpose – are there any relevant alternatives?

A1. There have been prior attempts to measure the coverage of image captions [1]. These methods typically involve extracting visual entities from reference captions and checking whether they appear in the caption being evaluated. However, such approaches have notable limitations:

1. MLLMs can describe the same visual content in highly diverse ways. As a result, it is often difficult to determine with precision whether the visual content in the reference captions is present in the evaluated caption.
2. These methods can only evaluate pre-defined types of visual content—typically objects, attributes, and spatial relations—since they rely on accurate extraction and comparison of such elements within captions. As a result, they cannot assess whether a caption captures conceptual aspects of an image, such as mood or semantic relationships between entities.

In contrast, our coverage metric avoids these limitations. Any information can be turned into a question and given to an LLM grounded in the caption.

[1] Object Hallucination in Image Captioning

Q2. Which datasets do Tables 4 and 5 correspond to?

A2. For Tables 4 and 5, we used IIW-400, based on findings [2] indicating that IIW-400 serves as a better reference caption set than DOCCI. However, to demonstrate the generalizability of CapMAS, we additionally tested its effectiveness on a subset of DOCCI (400 samples).

Table A demonstrates that CapMAS is also effective on DOCCI.

[2] ImageInWords: Unlocking Hyper-Detailed Image Descriptions

Q3. I believe this pipeline cannot improve the coverage in captions (and only improve factuality). How can we improve coverage?

A3. Through inference-time scaling, CapMAS can improve the coverage of captions while maintaining a certain level of factuality. Due to space limitations, we regret that we cannot provide further details here and kindly ask you to refer to our response A4 to reviewer fLiQ for more information.

Q4. This is a minor point, but for the sake of completeness, you could also evaluate your pipeline on GPT-4o, as it is more widely used than GPT-4v.

A4. We used GPT-4o to evaluate the captions, and it is known that LLMs tend to favor their own outputs [3]. Therefore, to ensure a fair comparison, we did not use GPT-4o at any stage of the proposed captioning pipeline.

[3] LLM Evaluators Recognize and Favor Their Own Generations

Q5. How is coverage computed? I couldn’t find a clear formula for this in the paper.

A5. We apologize for the confusion. Let N be the total number of VQA samples and C the number correctly answered by GPT-4o using only the captions. Then, the coverage score is C/N.

Q6. For measuring factuality, your method uses the image and the reference caption. If the reference caption is already available, is the image really required?

A6. Metrics based solely on reference captions are prone to stylistic bias, favoring or penalizing captions based on phrasing. To illustrate this, we compare a reference-based metric with our factuality metric using human-labeled captions (HUMAN) that are hallucination-free but stylistically different from the references. As shown in Table B, the reference-based metric correlates poorly with human judgment by assigning lower scores to HUMAN while our factuality metric remains robust.

Table B: Meta-evaluation with model- and human-generated captions; see Sec. 4.2 and Appx. B for details.

Q7. The coverage of these questions themselves would be bottlenecked by the vision capabilities of GPT-4o, and the questions might miss some details.

A7. Building a detailed VQA dataset is costly, so we relied on GPT-4o, making our dataset quality depend on its capabilities. To address this, we carefully guided human annotators during the labeling process.

While we considered using reference captions, we first aimed to show that our coverage metric works reliably without them, as reliance on references limits its applicability. Our results confirm this, and we are exploring ways to further improve the method by incorporating reference captions.

We have more results—please click “Rebuttal Comment” to allow us to add a comment.

Thank you for your thoughtful review. We appreciate your recognition of our method’s potential to improve the factual accuracy of image captions and our novel approach leveraging MLLM–LLM collaboration without training a separate corrector. We also thank you for your suggestions, which have enriched our work.

We are pleased to share that, in support of open-source research, we have decided to release our carefully curated VQA dataset and evaluation codes. This dataset includes 1k images, each paired with approximately 36 question-answer sets. We kindly ask that you consider this a contribution to the open-source community.

In the subsequent sections, we address each of your concerns as follows:

Q1. In Table 4, after the captions were corrected, the coverage score decreased. This may because: 1) the design of the metric is not suitable, 2) the method sacrifices coverage for factuality, indicating that the method has some drawbacks. Same phenomenon is observed in Table 5, please provide more explanation.

A1. We would like to clarify a potential misunderstanding regarding the decrease in the coverage score caused by the application of CapMAS. First and foremost, CapMAS was proposed to enhance the factuality of detailed image captions. Ideally, identifying and correcting hallucinations within captions would improve both factuality and coverage. However, through empirical analysis, we observed that such correction attempts often lead to the generation of new hallucinations due to the limitations of current MLLMs. Therefore, CapMAS is designed to remove detected hallucinations rather than correct them. As CapMAS may remove not only hallucinations but also factual content, it inherently involves a trade-off between factuality and coverage, as discussed in Section 4.3. This trade-off is controlled by its hyperparameter, $\pi$. Consequently, the decrease in coverage scores observed in Tables 4 and 5 arises not from a flaw in the proposed metrics, but rather from the design of CapMAS, which reflects the current limitations of MLLMs.

Q2. Since Factuality and Coverage cannot reflect the basic quality of language, how about including the traditional metrics like CIDEr and METEOR?

A2. We employ an LLM-based evaluation to measure the basic quality of language in the captions generated by CapMAS. This approach is motivated by recent studies [1,2] indicating that conventional automatic caption evaluation methods are biased and not well-suited for assessing detailed captions. Specifically, we utilize GPT-4o with the following prompt:

You are a language expert evaluating the fluency of image captions.

Fluency refers to how grammatically correct, natural, and well-formed the text sounds to a native English speaker. A fluent  caption should be grammatically correct, free of awkward phrasing, and read smoothly.

Evaluate the fluency of the following caption and return your output **strictly in JSON format** with:
- "reason": a key reason for your scoring
- "score": a number between 0 (completely disfluent) and 100 (perfect fluency)

Caption: "{caption}"

In addition to evaluating the captions generated by CapMAS, we also assess human-written detailed captions for the same set of images.

Table A

The results in Table A demonstrate that the captions generated by CapMAS achieve even higher fluency scores than the human-generated captions. This can be attributed to the final stage of CapMAS, in which the corrector LLM helps preserve or even improve the fluency of the captions.

We sincerely thank the reviewer for suggesting this experiment, which enabled us to highlight a new strength of CapMAS in terms of fluency.

Q3. In Table 2, it appears that the CLAIR metric can reflect the introduced noises. It's not as problematic as the article suggests.

A3. Yes, CLAIR is indeed capable of reflecting the introduced noise, which is precisely why we adopted it in our experiments. However, as shown in the prompt used below, the meaning of the score it produces is not clearly defined:

On a precise scale from 0 to 100, how likely is it that the candidate caption is describing the same image as the reference  caption? (JSON format, with a key "score", value between 0 and 100, and a key "reason" with a string value.)

In contrast, our proposed metrics have clearly defined interpretations and enable a more detailed analysis of MLLMs in terms of both factuality and coverage.

[1] ImageInWords: Unlocking Hyper-Detailed Image Descriptions
[2] Benchmarking and Improving Detail Image Caption

We have additional experimental results that we were unable to include here. To help us share them, we kindly ask you to click “Rebuttal Comment” to allow us to leave a supplementary comment.