Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

Comment (4): “Is this decoding method useful in more advanced VLLMs, such as Qwen-VL, Intern-VL, etc.?”

Response (4): Thank you for your thoughtful comments. Indeed, our approach can be applied to more advanced LVLMs and effectively mitigates hallucinations in these models. Here, we use Qwen-VL as an example and evaluate its performance on the POPE benchmark:

POPE Benchmark with Qwen-VL Backbone:

Random Setup

Popular Setup

Adversarial Setup

We have provided the full results in Table 1 of the revised paper. Please kindly refer to it for the results on A-OKVQA and GQA datasets.

On the MME-Hallucination benchmark, our approach also achieves state-of-the-art results on the Qwen-VL benchmark, as presented below. We have also incorporated this into Table C2 of the Appendix (due to the page limit).

In summary, these results demonstrate the general effectiveness of our approach across various LVLMs.

---

We hope that our responses have addressed your concerns. If you have additional comments or concerns, please let us know and we will be more than happy to answer.

Best,

Authors

Comment (2): “The authors did not perform more extensive evaluations on comprehensive benchmark such as MMbench, which is crucial for assessing the method's overall performance.”

Response (2): Following your comments, we have conducted an additional experiment to assess our approach on the MMBench benchmark. The overall performance results are presented below:

As shown in the table, our approach achieves superior performance compared to the baseline methods on the MMBench benchmark, highlighting the effectiveness of our proposed method. We have incorporated these results into Table 3 of the revised paper.

In the following table, we also present the detailed performance across six L-2 abilities on the MMBench benchmark: Logical Reasoning (LR), Attribute Reasoning (AR), Relation Reasoning (RR), Fine-grained Perception - Single Instance (FP-S), Fine-grained Perception - Cross Instance (FP-C), and Coarse Perception (CP). We have also included these results in Appendix C.3.

---

Comment (3): “I am confused by the experimental results about POPE, as they do not seem to fully align with the result from LLAVA 1.5. Could the authors clarify the specific settings followed in the experiments on POPE benchmark? How do these settings differ from those used in LLaVA?”

Response (3): In the LLaVA-1.5 paper [1], the model is evaluated using greedy decoding, as stated in their Appendix A.3, where each generated token is selected based on the maximum prediction probability. In contrast, our experiments evaluate all approaches using regular decoding, where the final generated token $y_t$ is sampled from the multinomial distribution with probabilities $p_\theta(y_t)$.This difference leads to performance discrepancies between our evaluation and that in the LLaVA-1.5 paper. Note that regular decoding is a standard benchmarking setting, and we follow prior work [2, 3] in using this setting for evaluation. We have clarified this setting in the Baselines section on page 6 of the revised paper.

Following your comments, we have also conducted an additional experiments using greedy decoding on the POPE benchmark on the COCO random subset:

We can see that using greedy decoding improves the performance of all baseline methods as well as our proposed DeGF, and our approach consistently achieves state-of-the-art performance. We have also added this experiment to Appendix C.4.

[1] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 26296-26306).

[2] Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2024). Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13872-13882).

[3] Woo, S., Jang, J., Kim, D., Choi, Y., & Kim, C. (2024). RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs. arXiv preprint arXiv:2405.17821.

Dear Reviewer prZa,

We greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.

---

Comment (1): “The process of generating images adds time, potentially slowing down LVLM response generation and making it less suitable for real-time applications.”

Response (1): Thanks for pointing out this. We will respond from two aspects:

• Efficiency Comparison. Following your comments, we compare the efficiency of our approach with other state-of-the-art approaches such as OPERA, Woodpecker, and HALC. Specifically, we conduct this comparison on the CHAIR benchmark, with the maximum token length set to 128. The results are as follows:

  Method	Avg. Latency $\downarrow$	GPU Memory $\downarrow$	CHAIR$_S$ $\downarrow$
  Regular	3.44 s ($\times$1.00)	15778 MB ($\times$1.00)	55.0
  VCD	6.91 s ($\times$2.01)	16634 MB ($\times$1.05)	54.4
  OPERA	24.70 s ($\times$7.18)	22706 MB ($\times$1.44)	52.6
  Woodpecker	10.68 s ($\times$3.10)	22199 MB ($\times$1.41)	57.6
  HALC	22.61 s ($\times$6.51)	23084 MB ($\times$1.46)	51.0
  Ours	13.89 s ($\times$4.04)	19119 MB ($\times$1.21)	48.8

  Our approach involves two queries and incorporates a text-to-image generation model to mitigate hallucinations, resulting in a 4.04$\times$ increase in latency and a 1.21$\times$ increase in GPU memory usage. Compared to other approaches, while our method is slower than regular decoding and contrastive decoding-based methods, it demonstrates efficiency advantages over OPERA and HALC. Note that our approach also outperforms the compared methods in performance on the CHAIR benchmark.

• Strategies to Speed Up Our Approach. To further enhance the efficiency of our approach, we propose two strategies to accelerate our approach: limiting the length of the initial response and reducing the number of inference steps in the diffusion process.

  (1) Reducing Diffusion Inference Steps. In the following table, we report the performance on the CHAIR benchmark after reducing the diffusion inference steps in the model. By reducing the diffusion inference steps from 50 to 10, the average latency decreases by 2.85 seconds per instance, while the performance on CHAIR remains robust.

  Diffusion Steps	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  50	13.89 s	48.8	14.6
  30	12.56 s	48.9	14.7
  20	11.87 s	49.2	14.8
  10	11.04 s	48.8	14.9

  (2) Restricting Length of Initial Response. In the following table, we present the efficiency and CHAIR performance results after decreasing the maximum token limit for the initial response. We can see that reducing the maximum number of tokens in the initial response from 128 to 96 decreases the latency by 0.72 seconds per instance while maintaining competitive performance. However, further reductions result in performance degradation, as a shorter initial response fails to adequately cover the entire scene, limiting its ability to generate an image that effectively reflects and mitigates hallucinations.

  # Tokens	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  128	13.89 s	48.8	14.6
  96	13.17 s	48.8	14.9
  64	12.20 s	49.5	14.8
  32	11.33 s	51.2	14.9

  Note that these two strategies are not conflicting; instead, they are complementary. Setting the diffusion steps to 10 and limiting the maximum number of tokens in the initial response to 96 further reduces the inference latency to 10.21 seconds per instance while maintaining robust performance.

We have also added these discussions and additional experiments to Section 4.4 and Appendix C.8 of the revised paper. Again, thank you for your valuable feedback to improve our paper.

Dear Reviewer prZa,

We greatly appreciate the time you have dedicated and the valuable feedback you have provided. As the discussion period draws to a close (Tue, November 26), please kindly let us know if there are any remaining questions. We will be more than happy to provide any further details or clarifications.

Best,

Authors

Comment (2): “Increased test-time computation cost. Diffusion models take dozens of inference steps to generate images, which is computationally less efficient than other decoding approaches. Authors are suggested to include test-time computation comparison to complete this work.”

Response (2): Thanks for pointing out this. We will respond from two aspects:

• Efficiency Comparison. Following your comments, we compare the efficiency of our approach with other state-of-the-art approaches such as OPERA, Woodpecker, and HALC. Specifically, we conduct this comparison on the CHAIR benchmark, with the maximum token length set to 128. The results are as follows:

  Method	Avg. Latency $\downarrow$	GPU Memory $\downarrow$	CHAIR$_S$ $\downarrow$
  Regular	3.44 s ($\times$1.00)	15778 MB ($\times$1.00)	55.0
  VCD	6.91 s ($\times$2.01)	16634 MB ($\times$1.05)	54.4
  OPERA	24.70 s ($\times$7.18)	22706 MB ($\times$1.44)	52.6
  Woodpecker	10.68 s ($\times$3.10)	22199 MB ($\times$1.41)	57.6
  HALC	22.61 s ($\times$6.51)	23084 MB ($\times$1.46)	51.0
  Ours	13.89 s ($\times$4.04)	19119 MB ($\times$1.21)	48.8

  Our approach involves two queries and incorporates a text-to-image generation model to mitigate hallucinations, resulting in a 4.04$\times$ increase in latency and a 1.21$\times$ increase in GPU memory usage. Compared to other approaches, while our method is slower than regular decoding and contrastive decoding-based methods, it demonstrates efficiency advantages over OPERA and HALC. Note that our approach also outperforms the compared methods in performance on the CHAIR benchmark.

• Strategies to Speed Up Our Approach. To further enhance the efficiency of our approach, we propose two strategies to accelerate our approach: limiting the length of the initial response and reducing the number of inference steps in the diffusion process.

  (1) Reducing Diffusion Inference Steps. In the following table, we report the performance on the CHAIR benchmark after reducing the diffusion inference steps in the model. By reducing the diffusion inference steps from 50 to 10, the average latency decreases by 2.85 seconds per instance, while the performance on CHAIR remains robust.

  Diffusion Steps	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  50	13.89 s	48.8	14.6
  30	12.56 s	48.9	14.7
  20	11.87 s	49.2	14.8
  10	11.04 s	48.8	14.9

  (2) Restricting Length of Initial Response. In the following table, we present the efficiency and CHAIR performance results after decreasing the maximum token limit for the initial response. We can see that reducing the maximum number of tokens in the initial response from 128 to 96 decreases the latency by 0.72 seconds per instance while maintaining competitive performance. However, further reductions result in performance degradation, as a shorter initial response fails to adequately cover the entire scene, limiting its ability to generate an image that effectively reflects and mitigates hallucinations.

  # Tokens	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  128	13.89 s	48.8	14.6
  96	13.17 s	48.8	14.9
  64	12.20 s	49.5	14.8
  32	11.33 s	51.2	14.9

  Note that these two strategies are not conflicting; instead, they are complementary. Setting the diffusion steps to 10 and limiting the maximum number of tokens in the initial response to 96 further reduces the inference latency to 10.21 seconds per instance while maintaining robust performance.

We have also added these discussions and additional experiments to Section 4.4 and Appendix C.8 of the revised paper. Again, thank you for your valuable feedback to improve our paper.

Dear Reviewer QadT,

Thank you for your insightful comments and positive recommendation of our work. We provide point-by-point responses to address your concerns below.

---

Comment (1): “A concern originated from numerical hallucinations. Figure 2 presents an overview of proposed approach DeGF, with addressing numeric hallucinations as an example. A key premise for this method is that diffusion models can accurately perceive numbers. However, it seems a common observation that diffusion models fail to accurately interpret numbers [A]. Have authors considered the case when diffusion models fail to generate numerically accurate images? It is good to include analysis on tolerance of proposed DeGF to diffusion model failures.”

Response (1): Thank you for your thoughtful feedback. Following your comments, we evaluated the quality of the generated images across all four subsets of the MME benchmark, including the "count" subset, which focuses on numerical hallucinations, using CLIPScore [1]. For this evaluation, we employed the CLIP model with the ViT-B/32 backbone. The results are summarized below:

As we can see from the table, our text-to-image generative model (specifically, SD-v1.5) achieves an average CLIPScore of over 30 across all subsets, demonstrating its ability to generate high-quality images that align well with the initial response. For comparison, the advanced DALL-E 3 model achieves a score of 32.0, while DALL-E 2 achieves 31.4 [2]. While, as you pointed out, our diffusion model performs worse on the count subset compared to the other two subsets, a CLIPScore of 30.69 indicates that the generated images are of sufficient quality to provide useful self-feedback. We have included this discussion in Appendix C.9.

Additionally, in Figure D3-D6, we provide qualitative evaluations of the images generated by the generative model, including both success and failure cases, across all four subsets of the MME benchmark to better understand the effectiveness of the generative models. Our results in Figure D4 show that, despite occasional failure cases, the generative model can accurately interpret numbers and produce high-quality and realistic images that accurately visualize the initial response. Moreover, our substantial performance improvement (e.g., ~30 points higher than regular decoding) on the count subset of the MME benchmark, as presented in Table 3, demonstrates the robustness of our approach to potential image generation failures and validates the effectiveness of our method.

[1] Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021, November). CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 7514-7528).

[2] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., ... & Ramesh, A. (2023). Improving image generation with better captions. https://cdn.openai.com/papers/dall-e-3.pdf

Comment (3): “May I ask how many samples authors use for plotting Figure 3?”

Response (3): For Figure 3 (Left), we evaluate using 1,000 samples from the CHAIR benchmark, while for Figure 3 (Right), we use all 3,000 samples from the POPE benchmark using the random subset from the MS-COCO dataset. We have added a footnote at the end of Page 4 to clarify this.

This experiment validates the effectiveness of text-to-image generative models in reflecting hallucinations. It demonstrates that these models can provide valuable self-feedback for mitigating hallucinations at both the response and token levels.

---

Comment (4): “For Table 2, may I ask if authors have tested long sequence, by setting max new tokens to larger numbers, such as 128 and 256?”

Response (4): Following your comments, we have conducted an additional experiment by setting the maximum token limit to 128 in the CHAIR benchmark. The results are reported below:

Results with LLaVA-1.5 backbone:

Results with InstructBLIP backbone:

The results indicate that our approach also achieves competitive performance across two LVLMs in mitigating hallucinations during long-sequence generation scenarios. We have incorporated these additional results into Table C1 of the revised paper.

---

We hope that our responses have addressed your concerns. If you have additional comments or concerns, please let us know and we will be more than happy to answer.

Best,

Authors

Dear Reviewer aaN5,

Thanks for your valuable feedback! We provide point-by-point responses to address your concerns below.

---

Comment (1): “The computation costs are unaffordable for the LLM decoding strategy…”

Response (1): Thanks for pointing out this. Following your comments, we compare the efficiency of our approach with other state-of-the-art approaches such as OPERA, Woodpecker, and HALC. Specifically, we conduct this comparison on the CHAIR benchmark, with the maximum token length set to 128. The results are as follows:

Our approach involves two queries and incorporates a text-to-image generation model to mitigate hallucinations, resulting in a 4.04$\times$ increase in latency and a 1.21$\times$ increase in GPU memory usage. Compared to other approaches, while our method is slower than regular decoding and contrastive decoding-based methods, it demonstrates efficiency advantages over OPERA and HALC. We have also added these discussions and additional experiments to Section 4.4 of the revised paper.

While computational complexity is important, we emphasize that performance is equally crucial to consider. Our model strikes a good balance between the two: although it is 2$\times$ slower than VCD, it achieves a significantly lower CHAIR$_S$ score by 5.8. Notably, our approach also outperforms OPERA and HALC on the CHAIR benchmark, while maintaining a lower computational complexity.

---

Comment (2): “This approach utilizes extra pretrained network (i.e., Stable Diffusion). Baselines should contain methods that also employ extra analysis network like woodpecker [r1, r2], etc. Otherwise, it is unfair for other decoding methods.”

Response (2): Thanks for pointing out this. Following your comments, we conducted performance comparisons on the CHAIR benchmark, evaluating our methods against other baselines, including Woodpecker and HALC (reproduced).

CHAIR Benchmark with Maximum Number of Tokens 64

CHAIR Benchmark with Maximum Number of Tokens 128

We can see that our approach achieves superior performance compared to Woodpecker and HALC on the CHAIR benchmark, demonstrating the effectiveness of our approach. We have incorporated these results into Table 2 and Table C1 of the revised paper.

Comment (5): “How about using the middle step of Stable Diffusion to provide generative feedback and reduce the computation costs? How about integrating the generative feedback in the instruction tuning phase? It will avoid inference costs.”

Response (5): Thank you for your insightful comments. Based on your feedback, we have implemented two strategies to accelerate our approach, including utilizing the middle step of Stable Diffusion, which has proven effective. We sincerely appreciate your constructive input!

Specifically, we propose two strategies to improve efficiency: (1) reducing the number of inference steps in the diffusion process and (2) limiting the length of the initial response.

• Reducing Diffusion Inference Steps. To improve the response generation speed, we can reduce the number of diffusion steps. In the following table, we report the performance on the CHAIR benchmark after reducing the diffusion inference steps in the model. By reducing the diffusion inference steps from 50 to 10, the average latency decreases by 2.85 seconds per instance, while the performance on CHAIR remains robust.

  Diffusion Steps	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  50	13.89 s	48.8	14.6
  30	12.56 s	48.9	14.7
  20	11.87 s	49.2	14.8
  10	11.04 s	48.8	14.9

• Restricting Length of Initial Response. Our method involves two queries to the LVLM for self-correcting decoding. To enhance efficiency, we can limit the length of the initial response. In the following table, we present the efficiency and CHAIR performance results after decreasing the maximum token limit for the initial response. We can see that reducing the maximum number of tokens in the initial response from 128 to 96 decreases the latency by 0.72 seconds per instance while maintaining competitive performance. However, further reductions result in performance degradation, as a shorter initial response fails to adequately cover the entire scene, limiting its ability to generate an image that effectively reflects and mitigates hallucinations.

  # Tokens	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  128	13.89 s	48.8	14.6
  96	13.17 s	48.8	14.9
  64	12.20 s	49.5	14.8
  32	11.33 s	51.2	14.9

  Note that these two strategies are not conflicting; instead, they are complementary. Setting the diffusion steps to 10 and limiting the maximum number of tokens in the initial response to 96 further reduces the inference latency to 10.21 seconds per instance while maintaining robust performance.

Besides, incorporating generative feedback into the instruction tuning phase is also an interesting idea. We have added a discussion about this in Appendix E of the revised paper. We will examine this idea in greater detail and explore it further in our future work.

---

Comment (6): “Why not conduct GPT4/GPT4-V analysis like VCD [r1]?”

Response (6): Thank you for pointing out this. In the manuscript, we have included GPT-4V analysis in some of our qualitative examples. Following your comments, we conduct an additional GPT-4V evaluation on the open-ended captioning tasks in the LLaVA-Bench following the VCD evaluation protocol. The settings and experimental details can be found in Appendix D.1. Note that in this setting, GPT-4V only evaluates two responses at a time. Therefore, we divide the experiments into three parts: Ours vs Regular, Ours vs VCD, and Ours vs M3ID. We report the results as follows.

GPT-4V Evaluation with LLaVA-1.5 backbone:

GPT-4V Evaluation with InstructBLIP backbone:

From the results, we can observe that our method achieves higher accuracy and detailedness in generated responses compared to regular decoding, VCD, and M3ID across both evaluated LVLMs. We have incorporated these results into Table 4 of the revised paper.

---

If you have additional comments or concerns, please let us know and we will be more than happy to answer.

Best,

Authors

Comment (3): “No inference costs are listed and compared, which is important for LLM decoding strategies.”

Response (3): Thanks for your thoughtful comments. We hope we have already addressed this in Response (1). Please let us know if you have any further feedback.

---

Comment (4): “Experiments only conducted two LVLM backbones. More results like on QwenVL are better performed.”

Response (4): Thank you for pointing out this. Following your comments, we have conducted additional experiments to evaluate the effectiveness of our approach using the Qwen-VL model. First, we present the results on the POPE benchmark using the MS-COCO dataset:

POPE Benchmark with Qwen-VL Backbone:

Random Setup

Popular Setup

Adversarial Setup

We have provided the full results in Table 1 of the revised paper. Please kindly refer to it for the results on A-OKVQA and GQA datasets.

On the MME-Hallucination benchmark, our approach also achieves state-of-the-art results on the Qwen-VL method, as presented below. We have also incorporated this into Table C2 of the Appendix (due to the page limit).

MME-Hallucination Result with Qwen-VL backbone

In summary, these results demonstrate the general effectiveness of our approach across various LVLMs.

Comment (3): “Experiments are limited to two LVLMs, experiments on more recent LVLMs (e.g., QWenVL, MiniCPMV, etc.) should be conducted.”

Response (3): Thank you for pointing out this. Following your comments, we have conducted additional experiments to evaluate the effectiveness of our approach using the Qwen-VL model. First, we present the results on the POPE benchmark using the MS-COCO dataset:

Random Setup

Popular Setup

Adversarial Setup

We have provided the full results in Table 1 of the revised paper. Please kindly refer to it for the results on A-OKVQA and GQA datasets.

On the MME-Hallucination benchmark, our approach also achieves state-of-the-art results on the Qwen-VL method, as presented below. We have also incorporated this into Table C2 of the Appendix (due to the page limit).

In summary, these results demonstrate the general effectiveness of our approach across various LVLMs.

---

Comment (4): “It’s better to perform experiments on multiple object hallucination [1].”

Response (4): Thank you for your constructive suggestion and for bringing this interesting benchmark to our attention. We address your comment from two perspectives:

• We would like to highlight that our extensively evaluated benchmarks already include scenarios involving multi-object hallucinations, such as in open-ended generation tasks like CHAIR and the GPT-4V evaluation on LLaVA-Bench.
• Given that this benchmark is very new, it has been challenging for us to adapt our codebase and conduct evaluations on it within the short rebuttal phase. However, we have acknowledged this benchmark and included it in our discussion on future work, as detailed in Appendix E of the revised paper. We greatly appreciate your valuable feedback and look forward to exploring this benchmark in future research.

---

Comment (5): “The approach involves various hyperparameters that require careful tuning to achieve optimal performance, which could make practical implementation challenging.”

Response (5): Thanks for your thoughtful feedback. While our approach introduces several hyperparameters to effectively leverage generative feedback, the performance of our approach remains robust across a reasonable range of hyperparameter settings. As shown in our ablation studies (Section 4.3 and Appendix C), on the POPE benchmark using the COCO random subset, setting hyperparameters within ranges of $\alpha_1 \in [0, 4]$, $\alpha_2 \in [1, 3]$, $\gamma \in [0, 1]$, and $\beta \in [0.05, 0.5]$ achieves an accuracy of >87.5% and consistently outperforms VCD and M3ID. Please also be assured that we will make the code implementation publicly available and provide detailed instructions upon acceptance to facilitate easier reproduction.

---

We hope that our responses have addressed your concerns. If you have additional comments or concerns, please let us know and we will be more than happy to answer.

Best,

Authors

Comment (2): “Since the model relies on an addition diffusion model for inference, additional inference overhead should be discussed.”

Response (2): Thanks for pointing out this. We will respond from two aspects:

• Efficiency Comparison. Following your comments, we compare the efficiency of our approach with other state-of-the-art approaches such as OPERA, Woodpecker, and HALC. Specifically, we conduct this comparison on the CHAIR benchmark, with the maximum token length set to 128. The results are as follows:

  Method	Avg. Latency $\downarrow$	GPU Memory $\downarrow$	CHAIR$_S$ $\downarrow$
  Regular	3.44 s ($\times$1.00)	15778 MB ($\times$1.00)	55.0
  VCD	6.91 s ($\times$2.01)	16634 MB ($\times$1.05)	54.4
  OPERA	24.70 s ($\times$7.18)	22706 MB ($\times$1.44)	52.6
  Woodpecker	10.68 s ($\times$3.10)	22199 MB ($\times$1.41)	57.6
  HALC	22.61 s ($\times$6.51)	23084 MB ($\times$1.46)	51.0
  Ours	13.89 s ($\times$4.04)	19119 MB ($\times$1.21)	48.8

  Our approach involves two queries and incorporates a text-to-image generation model to mitigate hallucinations, resulting in a 4.04$\times$ increase in latency and a 1.21$\times$ increase in GPU memory usage. Compared to other approaches, while our method is slower than regular decoding and contrastive decoding-based methods, it demonstrates efficiency advantages over OPERA and HALC. Note that our approach also outperforms the compared methods in performance on the CHAIR benchmark.

• Strategies to Speed Up Our Approach. To further enhance the efficiency of our approach, we propose two strategies to accelerate our approach: limiting the length of the initial response and reducing the number of inference steps in the diffusion process.

  (1) Reducing Diffusion Inference Steps. In the following table, we report the performance on the CHAIR benchmark after reducing the diffusion inference steps in the model. By reducing the diffusion inference steps from 50 to 10, the average latency decreases by 2.85 seconds per instance, while the performance on CHAIR remains robust.

  Diffusion Steps	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  50	13.89 s	48.8	14.6
  30	12.56 s	48.9	14.7
  20	11.87 s	49.2	14.8
  10	11.04 s	48.8	14.9

  (2) Restricting Length of Initial Response. In the following table, we present the efficiency and CHAIR performance results after decreasing the maximum token limit for the initial response. We can see that reducing the maximum number of tokens in the initial response from 128 to 96 decreases the latency by 0.72 seconds per instance while maintaining competitive performance. However, further reductions result in performance degradation, as a shorter initial response fails to adequately cover the entire scene, limiting its ability to generate an image that effectively reflects and mitigates hallucinations.

  # Tokens	Avg. Latency $\downarrow$	CHAIR$_S$ $\downarrow$	CHAIR$_I$ $\downarrow$
  128	13.89 s	48.8	14.6
  96	13.17 s	48.8	14.9
  64	12.20 s	49.5	14.8
  32	11.33 s	51.2	14.9

  Note that these two strategies are not conflicting; instead, they are complementary. Setting the diffusion steps to 10 and limiting the maximum number of tokens in the initial response to 96 further reduces the inference latency to 10.21 seconds per instance while maintaining robust performance.

We have also added these discussions and additional experiments to Section 4.4 and Appendix C.8 of the revised paper. Again, thank you for your valuable feedback to improve our paper.

Dear Reviewer XJFu,

We really appreciate your thorough review of our paper. We address the raised concerns and questions below.

---

Comment (1): “The effectiveness of the proposed method is heavily influenced by the quality and realism of the generated images. So the authors should perform more experiments and analysis on the quality of the generated images (both quantitative and qualitative, especially for long captions and real images).”

Response (1): Thank you for your thoughtful feedback. Following your comments, we have evaluated the quality of the generated images on all 4 subsets on the MME benchmark using CLIPScore [1]. Specifically, we utilize the CLIP backbone with ViT-B/32 backbone for our evaluation. We list the results below:

As we can see from the table, our text-to-image generative model (specifically, SD-v1.5) achieves an average CLIPScore of over 30 across all subsets. For comparison, the advanced DALL-E 3 model achieves a score of 32.0, while DALL-E 2 achieves 31.4 [2]. These results highlight the capability of our model to generate high-quality images that closely align with the initial response. We have incorporated this discussion into Appendix C.9.

Additionally, in Figure D3-D6, we provide qualitative evaluations of the images generated by the generative model, including both success and failure cases, across all four subsets of the MME benchmark. These evaluations provide deeper insights into the effectiveness of the generative models. Our results show that, despite occasional failure cases, the generative model consistently produces high-quality and realistic images that accurately visualize the initial response, providing effective self-feedback.

[1] Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021, November). CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 7514-7528).

[2] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., ... & Ramesh, A. (2023). Improving image generation with better captions. https://cdn.openai.com/papers/dall-e-3.pdf