To reviewer wv1z

We thank the reviewer for the detailed and encouraging feedback. We appreciate your recognition of the novelty of our work in addressing the underexplored issue of semantic hallucination in OCR scenarios, which is distinct from previous studies on object or attribute hallucination. We're glad you found the analysis of layer-wise hallucination tendency and its correlation with visual grounding both original and empirically sound. We also value your positive comments on the clarity and motivation of the paper, particularly in highlighting a failure mode in existing LMMs through intuitive examples and architectural insights. In addition, we’re encouraged by your acknowledgment of the method’s consistent performance improvements across benchmarks, especially on hallucination-prone samples, and how these gains align well with the method’s design.

Below, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):

[W1/Q1]: Incomplete Evaluation on OCR Benchmarks

[A1]: Thank you for your valuable feedback. We agree that incorporating more OCR-related benchmarks can better demonstrate the robustness and generalizability of our method. To address this, we have complemented our evaluation with experiments on both MME and MME-VideoOCR (a concurrent work). The results show:

On MME, our method brings improvements in perception-based subtasks such as OCR, counting, and positioning. However, due to the binary (yes/no) nature of many MME questions, where models can often guess the answer, the absolute improvement is relatively modest.

On MME-VideoOCR, we observe significant gains, especially in complex scene-text-centric video question answering tasks, where visual text grounding is more challenging.

These results, along with the existing experiments (e.g., Table 5), provide strong evidence of the robustness and general applicability of our proposed method across diverse OCR and multimodal reasoning benchmarks. We will include these new results in the revised version of the paper.

[W2/Q2]: Unverified Methodological Assumptions.

[A2]: Thank you for pointing out this important aspect. Our proposed ZoomText module is based on two core assumptions: (1) that query-to-image attention can effectively highlight relevant visual regions such as signs or posters, and (2) that background or non-semantic tokens exhibit relatively stable attention dynamics across layers. To empirically validate these assumptions, we conducted the following analyses:

First, to assess the effectiveness of ZoomText in capturing relevant text regions, we manually annotated 100 samples across TextHalu-Bench, TextVQA, and ST-VQA, with quadrilateral bounding boxes marking the target text areas. We then performed an IoU-based ablation study on Qwen2.5-VL-3B, comparing three variants: (a) a baseline selecting top-k tokens using final-layer attention, (b) baseline + Glimpse (which uses query-to-image attention, Eq. 4), and (c) baseline + Glimpse + Refocus (which incorporates attention variation dynamics, Eq. 5). Results show a consistent improvement in IoU scores, confirming the benefit of each step in refining the focus on relevant visual regions.

Second, to validate the Refocus assumption, we computed the coefficient of variation (CV) of attention scores across layers for each visual token. Tokens were categorized into: (i) foreground (within bounding boxes), and (ii) background (outside boxes but with high attention). We compute the CV for each token $i$, defined as:

$\mathrm{CV}_i = \frac{\mathrm{Std}(\alpha^i_1, \alpha^i_2, \dots, \alpha^i_L)}{\mathrm{Mean}(\alpha^i_1, \alpha^i_2, \dots, \alpha^i_L)}$

where $\\alpha^i\_L$ is the attention score of token $i$ at layer $L$. The analysis reveals that **foreground tokens exhibit significantly higher attention variation across layers (higher CV)**, while **background tokens remain relatively stable (lower CV)**, supporting our hypothesis that "sink tokens" can be identified through their consistent attention profiles.

Due to rebuttal constraints, we are unable to include visualizations at this stage, but we will incorporate these empirical results and more qualitative examples into the revised version of the paper to strengthen the methodological soundness of our design.

To reviewer 6wHJ

Thank you for the thoughtful feedback. We're glad you found our focus timely and novel, and appreciated both the framing and attention analysis. We also appreciate your recognition of TextHalu-Bench.

Below, we address each Weakness (W) or Question (Q) with a corresponding Answer (A):

[W1/Q1]: Discussion about the causal relatioinship between visual attention grounding and halluction generation.

[A1]: Thank you for the thoughtful question. We agree that claiming a direct causal relationship may be too strong, and we appreciate your suggestion to phrase it more cautiously. It is indeed plausible that both poor attention and hallucination stem from a shared issue—such as the model’s failure to recognize meaningful text.

That said, we believe it's valuable to explore whether inaccurate attention contributes to hallucination, given the auto-regressive decoding process in LLMs. Misalignment during the encoding stage could bottleneck downstream reasoning.

To examine this, we ran a new ablation on 100 samples from TextHalu-Bench, TextVQA, and ST-VQA. We manually annotated ground-truth text boxes and artificially reallocated attention from these to background regions using:

Here, $\alpha_{\text{gt}}$ and $\alpha_{\text{bg}}$ denote the original attention scores over ground-truth and background tokens, respectively. The denominator computes the total adjusted attention score to ensure that $\alpha'$ remains a valid probability distribution. As λ increases, hallucination scores rise significantly (same metric as Fig. 2), confirming that misallocated attention worsens hallucinations.

We will incorporate this analysis in the revised paper and revise our claim from a “causal relationship” to a “strong correlation”, ensuring a more accurate and robust interpretation.

[W2]: Discussion about the model`s ability limitation.

[A2]: Thank you for raising this important point. We agree that our method functions as a correction mechanism, not a capability enhancer. Hallucinations in multimodal models broadly fall into two types: (1) Faithfulness Errors – the model perceives the information but responds inaccurately; (2) Knowledge Gaps – the model fails to perceive or understand the required input.

Our training-free framework targets Type 1 errors and cannot improve models lacking fundamental OCR ability (e.g., LLaVA-NeXT). Still, we believe addressing faithfulness is a necessary step before tackling deeper knowledge gaps.

We have noted this scope in Appendix "Limitations" (Lines 597–602) and will make it more explicit in the revised version. We also plan to explore external knowledge integration in future work to expand the framework’s coverage.

[W3]: Heuristic Nature of Refocus.

[A3]: We agree that a deeper analysis of the Refocus step would enhance the clarity and credibility of our method. Although we are unable to include visualizations during the rebuttal phase, we conducted two ablation studies to empirically support the underlying heuristics: 1. IoU Evaluation: On the same validation set in [A1]. We compared three configurations using Qwen2.5-VL: (1) Baseline: Selects top-$k$ tokens using attention from the last LLM layer. (2) +Glimpse: Utilizes query-to-image attention (Eq. 4) to select tokens. (3) +Refocus: Filters irrelevant tokens using relative attention dynamics (Eq. 5).

The average IoU between predicted and ground-truth regions increases notably after adding Refocus, indicating improved localization precision.

2. Coefficient of Variation (CV) Analysis. To test the assumption that background tokens exhibit stable attention, we computed the coefficient of variation (CV) of attention scores across layers for each visual token:

$\mathrm{CV}_i = \frac{\mathrm{Std}(\alpha^i_1, \alpha^i_2, \dots, \alpha^i_L)}{\mathrm{Mean}(\alpha^i_1, \alpha^i_2, \dots, \alpha^i_L)}$

where $\\alpha^i\_L$ is the attention score of token $i$ at layer $L$. We categorized tokens into: (1) Foreground: Tokens inside bounding boxes; (2) Background: Tokens outside bounding boxes but with high attention

Our analysis shows that foreground tokens have significantly higher CV, while background tokens remain stable (lower CV). This validates our heuristic: stable tokens likely represent less semantically relevant areas and can be effectively filtered via Refocus.

[Q2]: The analysis of hyperparamter K.

[A2]: We agree that understanding the role of hyperparameter K in ZoomText is essential for clarity and reproducibility. We conducted ablations on Qwen2.5-VL and Mini-Monkey across four datasets with varying image resolutions, leading to three key observations: 1. Model-Agnostic Behavior: The optimal K generalizes well across architectures. Performance is not highly sensitive to the backbone. 2. Robust Value Range:
While there's no fixed best value, performance remains stable when K ∈ [96, 160]. Small K may miss text; large K may introduce noise. We use K = 128 as a balanced default. 3. Resolution Robustness: Most datasets (≤1000px) show no performance drop with K = 128. On high-res images from ReCTS (>2000px), the optimal K shifts to [160, 224], suggesting larger values help in ultra high-resolution scenarios. We acknowledge that K is empirically chosen. Still, its consistent performance across diverse datasets demonstrates robustness. We will highlight the lack of a resolution-aware selection method in the Limitations section.

[1] Icdar 2019 robust reading challenge on reading chinese text on signboard. ICDAR 2019.

[Q3]: Discussions about the failure cases.

[A3]: Thank you for raising this important point. Due to the limitations of the rebuttal stage, we are unable to include visual failure case examples at this time, but we will provide more qualitative samples in the appendix of the revised version. Regarding failure modes, we clarify the following: (1) Minimal Over-correction: Our method does not impair recognition of semantic or idiomatic text. As shown in Table 1, OCR performance is slightly improved, indicating no adverse effects. (2) Preserved Generalization: Table 5 shows stable performance on general benchmarks, suggesting hallucination mitigation does not come at the cost of reasoning ability. (3) Typical Failures Reflect Model Limits: Errors mainly occur in inherently difficult cases—e.g., distorted/occluded text, rare scripts, or questions requiring external knowledge—where even base models struggle. As a training-free method, ours ensures faithful outputs but cannot recover missing knowledge.

We will include detailed failure cases in the revised version to clarify the scope and limitations.

[Q4]: Further explanation about efficiency.

[A4]: Thank you for the insightful question. The added latency from our method occurs only during the prefilling stage, where intermediate attention scores are computed. This is a one-time cost per query, independent of the output length. In long-form generation tasks (e.g., captioning), our experiments with Qwen2.5-VL show that prefilling adds less than 35% to the total inference time when outputs exceed 50 tokens.

This overhead is modest compared to the dominant cost of autoregressive decoding, especially for longer outputs. We thus consider the trade-off acceptable for real-world use.

Dear Reviewer 6wHJ,

Thank you for your support and helpful comments. We've tried our best to address your concerns, and we hope our responses make sense to you. Importantly, we much value your comments and would be happy to discuss more. If you have any additional questions or open discussions, please don't be hesitant to leave more comments. We are always available at all time, to actively address any concerns or be prepared for more discussions.

Your opinions are rather important for us to improve the work!

Thank you!

Sincerely,

Authors

To reviewer 1eg9

We thank the reviewer for the detailed and encouraging feedback. We're glad that you recognized the significance of the semantic hallucination issue, particularly the tendency of Vision-Language Models to rely on internal semantic priors rather than grounding their outputs in visual evidence. We appreciate your acknowledgment of our formalization and mitigation of this problem as an important step toward improving the reliability of LMMs in real-world, text-centric applications. We're also pleased that you found the introduction of the TextHalu-Bench benchmark to be a valuable resource for future research in this area.

Below, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):

[W1]: The concern of the hallucination score calculation.

[A1]: Thank you for your thoughtful comment. We agree that the wording in Line 132 may have caused some confusion. Specifically, the phrase “feed both the hallucinated token $y\_{\\text{hal}}$ and the ground-truth token $y\_{\\text{gt}}$ into the LMM's output head” was not precise.

What we intended to convey is that, at each decoding step $t$, the model computes a probability distribution over the entire vocabulary based on the prefix $x\_{\\leq t}$. From this distribution, we extract (not feed) the probabilities assigned to both $y\_{\\text{hal}}$ and $y\_{\\text{gt}}$, as they are among the candidate tokens.

This process is fully consistent with the autoregressive property of LLMs: the model’s hidden state is determined solely by the preceding tokens, and the comparison between $y\_{\\text{hal}}$ and $y\_{\\text{gt}}$ is made at the output level, not by modifying the input sequence or hidden states.

We will revise the original sentence to clarify this distinction and appreciate the opportunity to improve the presentation.

[W2]: The absence of OCRBench in the main experiment.

[A2]: Thank you for the suggestion. OCRBench was not included in the main results table because it primarily evaluates text recognition from cropped text regions, where scene text understanding plays a limited role. In contrast, our method is designed for open-world scene text spotting and understanding.

Nonetheless, to provide a more comprehensive evaluation, we have conducted additional experiments on OCRBench. The results show consistent improvements across multiple LMMs, especially in scene-text-related VQA subtasks, confirming the generalization ability of our method. We will include these results in the revised version of the paper.

[Q1]: The confusion about how to select the best layer.

[A1]: Thanks for your question. As defined in Equation (3), $A\_\\ell$ is a scalar attention score that quantifies how much the $\\ell$-th layer attends to the scene text regions.

We compute $A\_\\ell$ for each layer and collect them into a 1D array $\\{A\_1, A\_2, \\dots, A\_L\\}$. We then apply the $\\arg\\max$ operation:

to select the layer that focuses most strongly on the text regions.

Dear Reviewer 1eg9,

Thank you for your support and helpful comments. We've tried our best to address your concerns, and we hope our responses make sense to you. Importantly, we much value your comments and would be happy to discuss more. If you have any additional questions or open discussions, please don't be hesitant to leave more comments. We are always available at all time, to actively address any concerns or be prepared for more discussions.

Your opinions are rather important for us to improve the work!

Thank you!

Sincerely,

Authors

To reviewer LbB3

We thank the reviewer for the detailed and encouraging feedback. We're pleased that you found our identification of the semantic hallucination issue in multi-modal large language models (MLLMs) to be both timely and significant, especially in light of the increasing use of MLLMs for text-rich tasks. We appreciate your recognition of the benchmark we contributed as a valuable resource for future research. We're also encouraged by your positive assessment of our method’s effectiveness in addressing semantic hallucination and enhancing performance on scene-text-focused benchmarks. Furthermore, we are grateful for your appreciation of the novel ideas behind our Glimpse-Refocus strategy and your insight that these ideas may generalize to broader applications involving grounded reasoning and alignment in vision-language tasks.

Below, we respond to each Weakness (W) or Question (Q) with a corresponding Answer (A):

[W1/Q1]: Unclear Technical Soundness of the Exploration

[A1]: Thank you for your insightful question. First, we would like to clarify that the hidden states from intermediate layers are not merely underdeveloped representations. In fact, prior work has shown that intermediate layers in large language models can offer better representations for downstream tasks. For example, [1] concludes that intermediate layers provide stronger embeddings for natural language understanding tasks (see Sec. 4.1). This insight motivates our exploration in MLLMs.

Building on this hypothesis, our experiments (Table 1) demonstrate that incorporating intermediate representations significantly improves grounding and reduces hallucination.

That said, we acknowledge the potential mismatch between the intermediate hidden states and the output projection layer, since the model was only trained with final-layer outputs. To address this, we conduct an ablation study (Table 9), which shows that assigning large weights to intermediate layers can hurt performance, likely due to semantic disruption. However, a moderate fusion weight preserves the semantic capacity of the final layer while benefiting from improved visual grounding in intermediate layers.

[W2/Q2]: Impact on other capabilities.

[A2]: Thank you for your thoughtful comment. We would like to clarify that our proposed method, Grounded Layer Correction (GLC), is specifically designed to mitigate hallucination in scene text spotting and understanding by injecting localized visual grounding signals. However, this design does not compromise the model’s general reasoning or knowledge-based capabilities.

As shown in Table 5, we evaluated our method on several general-purpose benchmarks beyond scene text. Still, we observed minor improvements or comparable performance, demonstrating that our method preserves the model’s broader abilities.

To further validate this, we conducted additional ablation studies on A-OKVQA [2], a challenging benchmark that requires commonsense and world knowledge. Results show that applying our method does not degrade the model’s performance, confirming that GLC does not interfere with higher-level semantic reasoning.

We do agree that our method does not significantly boost performance on knowledge-intensive tasks, primarily because it is training-free and does not introduce new external knowledge. In future work, we plan to explore combining hallucination mitigation with external knowledge grounding to further enhance general reasoning while preserving robustness in scene text understanding.

[W3/Q3]: Limited evaluation of the effectiveness of individual components of the framework

[A3]: Thank you for the valuable suggestion. We agree that evaluating the effectiveness of each individual component would provide deeper insight into our framework.

ZoomText Evaluation:

To assess the localization accuracy of ZoomText, we manually annotated 100 samples across TextHalu-Bench, TextVQA, and ST-VQA. Each sample was labeled with quadrilateral bounding boxes indicating the ground-truth text regions of interest. Since, to our knowledge, there is no existing benchmark that simultaneously offers both VQA annotations and dense spatial annotations, this manual annotation was necessary.

We compare the following configurations on Qwen2.5-VL to measure the impact of each component in ZoomText:

(1) Baseline: Selects top-k tokens based on visual attention from the final LLM layer.

(2) + Glimpse: Incorporates query-to-image attention (Eq. 4) for token selection.

(3) + Refocus: Further filters irrelevant tokens based on relative attention dynamics (Eq. 5).

We compute the average IoU between the predicted tokens and ground-truth bounding boxes. The results show that ZoomText significantly improves text region localization, demonstrating its effectiveness in focusing on relevant visual regions.

Grounded Layer Correction (GLC):

As for GLC, it is applied uniformly across all samples during decoding—there is no conditional switching. Therefore, we cannot meaningfully report a frequency of invocation. However, its effectiveness has been validated through extensive experiments showing consistent hallucination mitigation and semantic preservation (e.g., Table 1 and Table 5), confirming its utility as a general decoding enhancement.

We will include these additional analyses and experimental results in the revised version of the paper.

[1] Does Representation Matter? Exploring Intermediate Layers in Large Language Models [NeurIPS 2024 workshop]

[2] A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [ECCV 2022]

Dear Reviewer LbB3,

Thank you for your support and helpful comments. We've tried our best to address your concerns, and we hope our responses make sense to you. Importantly, we much value your comments and would be happy to discuss more. If you have any additional questions or open discussions, please don't be hesitant to leave more comments. We are always available at all time, to actively address any concerns or be prepared for more discussions.

Your opinions are rather important for us to improve the work!

Thank you!

Sincerely,

Authors

Thank you for the author(s) detailed responsees. All of my concerns were addressed and I am willing to raise my score based on the rebuttal.