Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing

We appreciate your positive feedback about high novelty, clear presentation, and solid experiments. We address your questions below.

---

Q1: How did you arrive to the 3x3 grid division for determining the spatial proximity. I expect it might not make a major difference but did you experiment with other choices?

A1: We have indeed explored other choices. We evaluated grid sizes from 1×1 to 6×6 on Janus-Pro-7B using the Object HalBench benchmark:

The 3×3 grid achieves optimal performance, though 2×2 to 6×6 all provide reasonable results within acceptable ranges. The 1×1 setting performs poorly due to insufficient edge connectivity during graph construction, which impairs GNN learning quality by creating overly sparse co-occurrence relationships.

---

Q2: I am not entirely convinced by the need to use GNNs for extracting token representations. Did you try using a simple way of extracting node representations for clustering, for instance, just a vector summarising the edge strengths of a node? I am also not sure if line 237 references the same baseline. What representation did you use there for clustering?

A2: The GNN learning step is essential, as it is widely recognized that GNNs are effective for node clustering due to their ability to learn node embeddings that capture graph structural information. In our case, the co-occurrence graph is constructed based on spatial proximity and semantic coherence, so effectively modeling its structure with a GNN amounts to capturing visual priors. We have shown in Table 1 that trivial clustering methods like naive K-means on the original codebook representations (Cluster-Base method) yield substantially inferior performance compared to our CGC+VTD approach. This supports our decision to employ GNNs.

The "baseline" method referenced in line 237 uses naive K-means clustering directly on the original VQVAE codebook embeddings without any graph-based representation learning.

---

Q3: Line 315: Could you please describe more precisely the combination of CGC with other methods.

A3: We have provided details of how to combine CGC with other methods in Appendix B.2. Due to the page limit, we could not put them in the main body of our submission. We hope this can address your concern.

---

Q4: If you performed a simple clustering step on continuous visual embeddings to manually discretize, does it not allow the method to also easily generalise to LVMLs using continuous visual tokens.

A4: This is a great question that we have also considered. While the intuition appears straightforward, the practical implementation presents significant technical challenges. Our method fundamentally relies on identifying "absent tokens" from a finite, discrete token vocabulary. Adopting continuous visual embeddings can be understood as introducing infinite visual tokens, making it impossible to get the set of "absent tokens" anymore.

---

Limitation: I am not sure the method generalizes to architectures that use Q-Former to process vision tokens even if discrete image tokens, or LVLMs which integrate vision modality through cross attention maps at all layers.

A5: While architectures like Q-Former and intra-layer cross attention were important early milestones in LVLM development, they fall outside the scope of our work. Our work focuses on a new class of LVLMs that adopt a unified token processing paradigm, where discrete image tokens are processed directly alongside text tokens within a single transformer architecture. In these models, there is typically no need for modality-specific components such as Q-Formers (as in BLIP variants), intra-layer cross-attention(as in Flamingo), or lightweight MLP adapters (as in LLaVA), since discrete tokenization naturally supports seamless cross-modal interaction without explicit alignment mechanisms. As the design space for unified LVLMs continues to evolve, our work offers foundational insights into hallucination behavior grounded in visual priors. These insights may help inform intervention strategies across future architectures, and we would be happy to engage in further discussion if you are interested.

Thank you for the positive evaluation of the motivation and our proposed method. We will address your questions below.

---

W1: Regarding Section 3.2, please consider providing examples of the Top-K(C) object categories for each group—such as C1, C2, and C3—or examples of the tokens contained in each group, along with corresponding hallucinated objects (obj′) and segmentation annotations from the COCO panoptic dataset. The current experimental setup is somewhat difficult to follow without concrete illustrations.

A1: Consider the following example. For the third case in Figure 8, the top-5 object sets associated with $C_1$,$C_2$ and $C_3$ are: $\text{Top-5} (C_1)=('table', 'sand', 'towel','pizza','cake')$, $\text{Top-5} (C_2)=('sand','bowl','table','cake','water')$, $\text{Top-5} (C_3)=('skateboard','table','ski','knife','light')$. In here, the hallucinated object 'bowl' is the second most significant object associated with $C_2$ and does not appear in the top-5 objects associated with $C_1$ and $C_3$. This demonstrates our central finding: absent tokens from dominant clusters ($C_2$) show stronger correlations with hallucinated content (the object "bowl" in this case) than present tokens or tokens from non-dominant clusters. Please note that although $\text{Top-5}(C_2)$ also contains several ground truth objects, removing $C_2$ tokens' influence primarily eliminates hallucination-prone associations while preserving essential visual information through $C_1$ tokens. This selective intervention maintains recall for actual visual content while reducing false positive generations. We will include this example in our revision for better readability.

To supplement, we also present here how to derive the order of Top-K(C). We first construct a token-object association dictionary for each image token using the COCO panoptic dataset, where we compute the frequency of this token with which it is used to represent objects from different segmentation classes. A token is considered to represent an object if it lies within the segmented area corresponding to that object. The association dictionaries of all image tokens are then aggregated within token groups ($C_1$, $C_2$, $C_3$) to summarize how frequently tokens in each group are used to represent different segmentation classes, based on which we construct Top-K (C) sets ranked by association frequency.

---

W2: In Section 3.3, please consider using a term like v_candidate instead of v_hal, as the current terminology may be somewhat misleading. This is especially relevant since hallucinated tokens cannot always be explicitly identified in certain benchmarks (e.g., MME or Object HalBench), unlike in AMBER.

A2: Thank you for your suggestion! We will update our notations in the revision.

Thanks for your valuable feedback. We appreciate that the reviewer found that our paper proposed a novel problem and a cohesive methodology. We address your concerns below.

---

W1: Inadequate baseline fairness: Compared methods (e.g., VCD, SID, OPERA) are primarily designed for understanding-focused LVLMs, not tailored to discrete tokenizer architectures. This may lead to biased comparisons, as contrastive decoding methods, for example, may conflict with discrete token representations.

A1: To the best of our knowledge, we are the first to investigate hallucination detection and mitigation in LVLMs using discrete image tokenizers. Existing state-of-the-art methods were all developed for LVLMs with continuous image encoders, and our experiments have shown them to be ineffective when applied to models with discrete tokenizers. We believe our experimental setup is fair. As you noted, the failure of methods such as contrastive decoding may stem from a fundamental mismatch with discrete token representations. This is not a result of unfair evaluation, but rather a limitation inherent to methods designed for continuous encoders. Our findings highlight this gap and constitute a meaningful contribution, rather than a flaw. We appreciate your feedback and welcome further discussion.

---

W2: Suboptimal baseline models: Models that unify understanding and generation, such as Chameleon, Emu3, and Janus-Pro, use discrete tokenizers and perform worse than current SOTA LVLMs (e.g., Qwen2.5-VL, Intern3-VL) in visual language understanding tasks. The benchmarks only focus on understanding tasks and lack evaluation on generation tasks.

A2: LVLMs with discrete tokenizers have recently attracted rapidly growing attention, making it important to investigate their behavior with respect to hallucination. In this paper, we focus exclusively on LVLMs with discrete image tokenizers. Models such as Qwen2.5-VL and Intern3-VL rely on continuous image encoders and are therefore beyond the scope of our study.

Hallucination in VLMs, as defined by previous research [1, 2, 3, 4], is characterized by VLMs generating textual descriptions that include objects not present in or inconsistent with the target images. The benchmarks we employ (AMBER, Object HalBench) are established and well-adopted standards in the hallucination-related literature. We strictly follow previous works and use these benchmarks to ensure fair comparison.

We are not certain what you mean by “generation tasks”. If you are referring to image generation tasks, they fall outside the scope of our paper, as they conflict with how multimodal hallucination has been defined in prior work, focusing on visual understanding. If you are referring to text generation, multimodal hallucinations are identified based on the generated textual output, as defined in prior works. Our setting follows this established definition. We are happy to address any additional concerns you may have.

[1]: Rohrbach, Anna, et al. "Object hallucination in image captioning." arXiv preprint arXiv:1809.02156 (2018).
[2]: Li, Yifan, et al. "Evaluating Object Hallucination in Large Vision-Language Models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. (2023)
[3]: Wang, Junyang, et al. "Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation." arXiv preprint arXiv:2311.07397 (2023).
[4]: Liu, Jiazhen, et al. "PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2025).

---

Q1: Baseline relevance: Please verify how the proposed method compares to the SOTA model (e.g. Qwen2.5-VL) for comprehension tasks.

A3: Our experimental design follows standard practices in hallucination mitigation research, where the appropriate comparison is between different mitigation methods applied to the same base model, not across different architectures. Comparing against Qwen2.5-VL would be out of the scope of this paper. As clearly stated in our abstract and introduction, we specifically target LVLMs with discrete image tokenizers and do not cover models with continuous image encoders, such as Qwen2.5-VL.

---

Q2: Benchmark scope: Are existing benchmarks sufficient to evaluate the baseline model's hallucination in generation tasks? Please verify on a generation-centric dataset.

A4: As we clarified in the previous question, hallucination of VLM happens where models generate plausible but incorrect textual descriptions of visual content. The benchmarks we employ (AMBER, Object HalBench) are specifically designed and widely adopted for evaluating hallucinations in visual understanding tasks. Visual generation hallucinations (where models produce incorrect visual content) represent a fundamentally different research problem, which is beyond our research scope.

We appreciate your positive feedback on our motivation and solid experiments. We hope our answers can address your concerns and questions.

---

W1: The significance of the proposed method is a bit limited...

A1: Recent advances in LVLMs have increasingly focused on unifying understanding and generation within single frameworks [1,2,3], with discrete image tokenizers emerging as a particularly promising approach [4,5,6]. The recent success of BAGEL [4], which outperforms Qwen2.5-VL on 6 out of 7 image understanding benchmarks, demonstrates the potential of this unified architecture paradigm.

We acknowledge that most previous hallucination mitigation research on LVLMs is based on continuous LVLMs; however, as LVLMs with discrete image tokenizers are gaining more popularity and have shown extraordinary performance, we believe it is very important to systematically study hallucination mitigation specifically for discrete tokenizer-based LVLMs. To our knowledge, we are the first to identify and quantify hallucination sources arising from discrete image token co-occurrence patterns, providing a novel perspective for understanding visual hallucinations in unified multimodal architectures. Our proposed CGC+VTD framework directly addresses these visual priors through targeted latent space intervention, achieving superior performance while maintaining computational efficiency.

[1] Zhang, Xinjie, et al. "Unified multimodal understanding and generation models: Advances, challenges, and opportunities." arXiv preprint arXiv:2505.02567 (2025).
[2] Li, Zijie, et al. "Dual diffusion for unified image generation and understanding." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025. 
[3] Qu, Liao, et al. "Tokenflow: Unified image tokenizer for multimodal understanding and generation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025. 
[4] Deng, Chaorui, et al. "Emerging properties in unified multimodal pretraining." arXiv preprint arXiv:2505.14683 (2025). 
[5] Lin, Haokun, et al. "Toklip: Marry visual tokens to clip for multimodal comprehension and generation." arXiv preprint arXiv:2505.05422 (2025).
[6] Xie, Jinheng, et al. "Show-o2: Improved Native Unified Multimodal Models." arXiv preprint arXiv:2506.15564 (2025).

---

W3: The overall writing, organization, and illustrations could be improved to enhance clarity and readability.

A2: Thanks for the suggestion. We will improve the clarity and readability in the revision.

---

Q1: In Eq. 2 and Eq. 3, I am curious about the intuition behind why the contrastive loss alone is not strong enough...

A3: Our core intuition is that while $\mathcal{L}\_{\text{contrast}}$ uses co-occurrence weights for relative optimization, $\mathcal{L}\_{\text{pps}}$ enforces absolute similarity thresholds that ensure adequate separation between positive and negative pairs regardless of batch composition. Concretely, the gradient of $\mathcal{L}\_{\text{contrast}}$ w.r.t. $\text{sim}(\mathbf{h}\_i, \mathbf{h}\_j)$ is positively correlated to $s\_{ij}$ while negatively correlated to the number of negative sample pairs. So the pressure to improve the similarity between node i and node j towards a high $s\_{ij}$ could be diminished by a large number of negative pairs within the batch. $\mathcal{L}\_{\text{pps}}$ addresses this by setting explicit minimum similarity targets ($\beta \cdot s_{ij}$) that must be satisfied independently of other pairs in the batch. To further address your concern, we conducted additional analysis on the similarity patterns during GNN training using Janus-Pro-7B on the COCO 2017 validation set.

Without $\mathcal{L}\_{\text{pps}}$, the model achieves only 0.56 average positive similarity with a similarity gap of 0.21 between positive and negative pairs. With $\mathcal{L}\_{\text{pps}}$, positive similarity increases substantially to 0.79, while the similarity gap expands to 0.33 by 57%. This demonstrates that $\mathcal{L}\_{\text{pps}}$ successfully enforces the absolute similarity constraints, ensuring that co-occurrence relationships are preserved with sufficient fidelity for downstream clustering and hallucination mitigation. Furthermore, the ablation study in Table 4 consistently shows performance degradation when $\mathcal{L}\_{\text{pps}}$ is removed, confirming that $\mathcal{L}\_{\text{pps}}$ is crucial for our method.

---

Q2: In Figure 3, group C1 refers to tokens from the dominant cluster(s) that appear in the image. How do these tokens cause hallucinations if they are indeed present in the image?

A4: C1 tokens do not directly cause hallucinations, but they can be associated with hallucinated objects due to the limited vocabulary size of discrete tokenizers. Since discrete tokenizers use a fixed vocabulary to represent diverse visual content, each token must encode multiple visual concepts and objects. Consequently, any given token—including those present in the image (C1)—will be associated with various objects, encompassing both those that genuinely appear in the current image and those that do not. Our analysis demonstrates that while C1 tokens may indeed correlate with some hallucinated objects, the absent C2 tokens from the same dominant cluster show significantly stronger correlations with hallucinated content. This finding reveals that hallucinations primarily arise from the model's inappropriate activation of absent tokens that frequently co-occurred with present ones during training, rather than from the present tokens themselves. Therefore, C1 tokens are not the direct cause of hallucinations—they legitimately represent visual content in the image. The hallucinations stem from the learned co-occurrence priors that cause the model to inappropriately reference their absent cluster partners (C2 tokens). This is why our Visual Token Decontamination method specifically targets C2 tokens for suppression while preserving the contributions of present C1 tokens.

---

Q3: Section 3.2 should be described more clearly....

A5: For the analysis in Section 3.2, we only consider the AMBER objects that are covered by COCO segmentation classes. We do this analysis as follows. We first construct a token-object association dictionary for each image token using the COCO panoptic dataset, where we compute the frequency of this token with which it is used to represent objects from different segmentation classes. A token is considered to represent an object if it lies within the segmented area corresponding to that object. The association dictionaries of all image tokens are then aggregated within token groups (C1, C2, C3) to summarize how frequently tokens in each group are used to represent different segmentation classes, based on which we construct Top-K (C) sets ranked by association frequency. To compute HitRate@K, we extract hallucinated objects identified by the AMBER and use string matching against COCO segmentation classes to determine whether each hallucinated object appears within the Top-K(C) set.

Although this approach may overlook the hallucinations linked to AMBER objects not covered by COCO classes, it still ensures methodological consistency and does not compromise our core findings. The dramatic HitRate difference between C2 and other groups remains statistically significant, validating our hypothesis about visual priors. To further address your concern, we conduct another HitRate analysis using 500 images from the COCO dataset with Janus-Pro-7B. We can observe that C2 has a much higher HitRate compared with using AMBER, and the huge gap between C2 and other groups (C1 and C3) remains.

Moreover, it i is important to emphasize that visual priors extend beyond object-level associations—they also capture spatial relationships where tokens frequently appear in proximity within images. Our CGC method deliberately incorporates both spatial co-occurrence (tokens appearing within the same local regions) and semantic co-occurrence (tokens representing the same segmented objects) to comprehensively model these visual priors at the token level.

The HitRate analysis serves as an illustrative proxy to demonstrate the effectiveness of our clustering approach. Our method fundamentally operates on token-level co-occurrence patterns rather than being limited to object-level associations. This token-centric approach enables our method to capture richer visual relationships that go beyond simple object categories. The strong performance of our method across different datasets also validates its generalizability. As shown in Table 1, our approach trained on COCO data effectively mitigates hallucinations on AMBER, which has a different data distribution. This cross-dataset effectiveness demonstrates that our method captures fundamental visual co-occurrence patterns rather than dataset-specific artifacts, making it broadly applicable to diverse evaluation scenarios.

---

Q4: In Eq. 4, since $g^{(l)}(v_\text{hal})$ has been normalized using L2-Norm according to Line 255, can I understand that the denominator in Eq. 4 will always be 1?

A6: Yes, your understanding is correct. We will simplify this equation in the revision.