Thank you for your thoughtful and detailed review. We appreciate the time and care you took in evaluating our work, as well as your recognition of its key strengths. We address your main concerns point by point below.

Relevance

The practical relevance of this finding is still a bit unclear to me.

Between lines 323 and 333, we discuss several potential practical implications of our findings, showing applications where the narrow gate can be exploited proactively (e.g., steering via patching) and propose how to use it to reduce memory and computing usage.
We also suggest that a possible way to make the models more robust and “encourage a more distributed communication can be masking the text-on-[EOI] attention during the last part of training” (lines 325-326). To validate experimentally the last claim and address the reviewer's concern, we fine-tuned Chameleon and Emu3, masking the [EOI] tokens on an instruction dataset composed of 10k samples from the LLaVA training set and 5k samples from the VQAv2 training set. In this case, we can significantly reduce the detrimental effect of [EOI] ablation on performance.
The results evaluated on 50 validation samples each from VQAv2 and COCO are summarized below:

• Chameleon:

  • VQAv2 accuracy improved from 50% to 62% (base). With ablation of [EOI], it improves from 0.05% to 52%.
  • COCO-CIDER score increased from 0.48 to 0.72 (base). With ablation of [EOI], the score increased from 0.06 to 0.66.

• Emu3:

  • VQAv2 accuracy improved from 44 to 50% (base). With ablation of [EOI], it improves from 35% to 46%.
  • COCO-CIDER score increased from 0.73 to 0.87 (base). With ablation of [EOI], the score increased from 0.40 to 0.82.

To validate the importance of keeping [EOI] masked during training, we performed a standard fine-tuning run without masking [EOI]. Then, we repeat the ablation experiment in Tab. I and we measured the effect of ablation [EOI] after the fine-tuning.

• Chameleon: Final VQAv2 accuracy was 53% (base) and 35% with [EOI] ablation. COCO-CIDER was 0.75 (base) and 0.05 with [EOI] ablation.

• Emu3: Final VQAv2 accuracy was 45% (base) and 43% with [EOI] ablation and COCO-CIDER was 0.82 (base) and 0.48 with [EOI] ablation.

These results show that the training with [EOI] masked is an effective strategy to remove almost completely the gating mechanism of [EOI]. Notably, masked fine-tuning also shows improved performance wrt the non-masked standard, except for COCO-CIDER in Chameleon. The masked-finetuning, targeted on [EOI], is an example of how the findings of this work can be used to improve the model's performance and robustness.

This is in particular the case as the results are also not overly surprising [...] Specifically, with no loss on the image tokens, there is no pressure on the EOI token to capture much semantics about the image, as the image tokens themselves can directly be used. Why is this finding of importance?

Through our experiments, we show that having multimodal output, i.e., having a loss on the image tokens, is not sufficient to present a narrow gate: the Janus model (and the Liquid model we have integrated during the rebuttal phase) are multimodal in output, but they do not present a narrow gate. Our results not only show the (hitherto unreported) existence of a multimodal narrow gate, but also shed light on its possible causes.

The fact that native multimodal-output models, as Chameleon and Emu3, construct an efficient communication strategy despite exhibiting a wide modality gap across all the layers (Fig. 3) is rather surprising. Indeed, the modality gap can create challenges for performance in multimodal tasks [1] and can be used to quantify the quality of the training in VLMs [2]. Understanding and characterizing how native multimodal-output models develop an effective bottleneck-style communication strategy is an important insight for future studies in multimodal models, especially given their recent implementation in several state-of-the-art multimodal training pipelines [3,4].

Interpretation of results.

The authors argue that the results show that...

The reviewer offers an interesting interpretation of why semantic content appears diffuse in models that do not exhibit a narrow-gating mechanism (e.g., Llava and Pixtral).

However, our work specifically focuses on models like Chameleon and Emu3, which show the opposite behavior, where semantics are localized in a single token (the [EOI] token). In these models, we observe a significant embedding gap between image and text modalities throughout the hidden layers (see Fig. 1), much larger than in models where the semantic content is distributed, which suggests that this [EOI] may function as a modality-bridging messenger token. This hypothesis can be motivated by the very low cosine similarity between [EOI] with both text and image embeddings (<0.1), indicating it doesn't belong to either modality. Instead, it seems to act as an intermediary to enable cross-modal communication.
The structural difference, diffuse versus localized communication, might stem from how the models handle the alignment between modalities, and we agree that further experiments (e.g., token ablations or alternative pretraining approaches like DINO) could provide deeper insight.

Secondly, I have difficulties with the argument that in Llava-7B (or other text-only models) the “image embeddings and text embeddings align increasingly throughout the layers”

Our definition of image embedding is adopted following the convention in the literature [e.g. 5]. We simply refer to the embeddings of the tokens from the image. Moreover, previous work on the Llava-style models [5] showed that “activations in the late layers at each visual token position correspond to token embeddings that describe its original patch object in its corresponding image token”. Image embeddings at late layers, therefore, do preserve interpretable visual information of the input, even when no loss is applied to those positions' output.

Additional minor weaknesses.

Figure 1 (right) is cumbersome to read, as 3 dimensions are shown in 2 dimensions without the 3rd axis being clear.

We are not sure we understand the reviewer’s comment. Figure 1 (right) shows how different models perform after ablation of the two possible communication channels: image-text (x-axis) and [eoi]-text (y-axis). There isn’t a third axis; the colors of the markers simply denote different models.

The authors evaluate a custom version of Emu3 (fine-tuned from Emu3-Gen) — why is this relevant and how do results differ between models?

We fine-tuned the Emu3-Gen model since we wanted a model capable of both generating images and text.

Minor typos [...]

We thank the reviewer. These typos will be corrected in the final version.

[1] W. Lieng et al., Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS, 2022

[2] Q. Huang et al., Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate, ICCV, 2025

[3] openai.com/index/introducing-4o-image-generation/

[4] M. Shukor et al., Scaling Laws for Native Multimodal Models, 2025, arXiv:2504.07951

[5] Neo et al. Towards Interpreting Visual Information Processing in Vision-Language Models.

Thank you again for your comments and suggestions. If we manage to address your main concerns, we politely ask you to consider raising your score. Please don’t hesitate to let us know if you have further questions. We are happy to answer your questions during this rebuttal discussion period.

Thank you for your valuable feedback and the thorough analysis of our manuscript. We will use your suggestions to improve the quality of our work. We address the concerns you raised below.

Weaknesses

1. The conclusions presented in the Abstract and Introduction seem to conflict with those in Section 4. [...]

We agree with the reviewer. We will modify the Abstract and Introduction to be more consistent with our discussion and conclusions. Indeed, there is an impact of the diversity of the architectures (visual encoder) and training protocols (training from scratch vs fine-tuning of the LLM backbone) on the narrow gate phenomenon.

To shed light on this point, in an experiment we performed after the submission deadline, we analyzed LIQUID [1]. LIQUID is a unified multimodal-output model that differs from Chameleon only because its LLM backbone was fine-tuned from Gemma-7B, rather than being trained from scratch. In particular, LIQUID encodes the images with the same VQ-VAE as Chameleon. In the table below, we report the ablation analysis for the VQAv2 task.

This experiment shows that having a VQ-VAE encoder is not sufficient to produce the narrow gate, suggesting that the natively multimodal training process and objective function play crucial roles.

2. [...] the contributions are somewhat difficult to assess. The narrow gate token [...] is a widely observed phenomenon in both LLMs and VLMs.

We discuss the memory tokens in lines 912-921 and briefly cite the analogy with the narrow gate in lines 331-333. While both the narrow gate and memory tokens are known to store global input information, our study goes further. Specifically, we show that the [EOI] token in Chameleon and Emu not only stores global information but also emerges as a unique cross-modal communication channel in these multimodal models. Previous studies focusing on registers [2] only addressed VLMs trained for image understanding, where several register tokens store global information. The phenomenon we describe in this work is novel both in the model we analyze and in the qualitative picture that emerges on such multimodal-output VLMs.
Secondly, unlike memory tokens that are often manually introduced to capture global information, either through explicit design [3] or training objectives [4], the [EOI] token becomes a “narrow" gate for the information flow naturally during the autoregressive training, without targeted intervention. To our knowledge, this emergent role of [EOI] has not been found in existing work in either unimodal or multimodal output VLMs. If the reviewer can suggest relevant literature, we will be happy to compare and discuss it further in the camera-ready version of the work.

After identifying the narrow gate token pattern, I believe the paper should go further by proposing ideas to mitigate the drawbacks of this pattern and support them with experimental validation—even small-scale experiments would be valuable. [...]

Following the reviewer’s suggestion, we experimentally validated our proposed mitigation strategy by fine-tuning both Chameleon-7b and Emu3. We trained the models on a dataset composed of 10k samples from the LLaVA training set and 5k samples from the VQAv2 training set. We fine-tuned the model for one epoch using LoRA with a batch size of 128, weight decay of 0.1, and a cosine-annealed learning rate starting at 2e-5. During training, we masked the attention to the [EOI] token.

To evaluate performance, we used 50 validation samples each from VQAv2 and COCO. Results are summarized below:

• Chameleon:

  • VQAv2 accuracy improved from 50% to 62% (base). With ablation of [EOI], it improves from 0.05% to 52%.
  • COCO-CIDER score increased from 0.48 to 0.72 (base). With ablation of [EOI], the score increased from 0.06 to 0.66.

• Emu3:

  • VQAv2 accuracy improved from 44% to 50% (base). With ablation of [EOI], it improves from 35% to 46%.
  • COCO-CIDER score increased from 0.73 to 0.87 (base). With ablation of [EOI], the score increased from 0.40 to 0.82.

To validate the importance of keeping [EOI] masked during training, we performed a standard fine-tuning run without masking [EOI]. Then, we repeat the ablation experiment in Tab. I and we measured the effect of ablation [EOI] after the fine-tuning.

• Chameleon: Final VQAv2 accuracy was 53% (base) and 35% with [EOI] ablation. COCO-CIDER was 0.75 (base) and 0.05 with [EOI] ablation.

• Emu3: Final VQAv2 accuracy was 45% (base) and 43% with [EOI] ablation and COCO-CIDER was 0.82 (base) and 0.48 with [EOI] ablation.

These results show that the training with [EOI] masked is an effective strategy to remove almost completely the gating mechanism of [EOI]. Notably, it can also improve performance versus standard finetuning except for COCO-CIDER in Chameleon.

3. The experiments in the paper are limited to the image classification task [...]

The main findings of the paper are summarized in Table I of Section. 3.3 and support the central role of the narrow-gate ([EOI]-token) in the Chameleon and Emu models across diverse tasks: ImageNet classification, visual question answering (VQAv2), and caption generation (Flicker-30k and MS-COCO). Therefore, our experiments are not limited to image classification tasks.

To extend the experimental evidence given for ImageNet in Sec. 3.2, we analyzed the attention patterns of Fig. 3 for visual question answering (VQAv2) and captioning (Flickr-30k), finding the same characteristic attention pattern.
The experiment depicted in Figure 4 requires instead the selection of a set of representative concepts associated with a dataset. This selection was simple for Imagenet, where the classes give the meaningful semantics, but it is less trivial in Flickr30k (or VQAv2).

To address the reviewer's concern, we extend the experiments to a subset of 10000 samples taken from the Flickr30k dataset, analyzing the Chameleon-7b representations. To extract the essential concepts contained in the captions, we used a gte-Qwen1.5-7B-instruct [5] (a Sentence Transformer [6]) to map each caption to a vector encoding its semantics. We then measure the similarity between the hidden representations of [EOI] and the Flickr30k embeddings from the Sentence Transformer with the Neighborhood Overlap [7]. If an internal representation of Chameleon is similar to the embeddings from the gte-Qwen1.5-7B-instruct this means that these representations encode similar abstractions of the image. The result of this experiment is consistent with the one on ImageNet: the similarity between [EOI] and the captions embedding increases from 0.0 to 0.16 between layers 5 and 9, and it remains approximately 0.16 for all the remaining layers. For the average image embedding, and all the other image embeddings, it remains close to zero.

In addition, to further validate the generality of our results within the classification domain itself, we have introduced two additional benchmarks: Caltech101 [8] and Describable Textures Dataset [9]. Also these cases exhibit trends consistent with those shown in the paper in Fig. 3 for ImageNet.

4. The paper focuses exclusively on the information flow from vision to text. [...]

This is a relevant and interesting question. However, exploring the narrow gate in image generation tasks involves significant additional experimentation and analysis. It would be best addressed in a separate, future study.

[1] Wu, Junfeng, et al. "Liquid: Language models are scalable and unified multi-modal generators."

[2] Neo et al. Towards Interpreting Visual Information Processing in Vision-Language Models.

[3] Wen et al. Efficient vision-language models by summarizing visual tokens into compact registers.

[4] Burtsev et al. Memory transformer.

[5] huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct

[6] Reimers et al., Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

[7] Doimo et al., Hierarchical nucleation in deep neural networks

[8] Li et al, Caltech 101, doi.org/10.22002/D1.20086

[9] M. Cimpoi, et al. Describing Textures in the Wild

Thank you again for your detailed suggestions. If we manage to address your main concerns, we politely ask you to consider raising your score. If, on the other hand, there are some remaining questions, we are happy to address them during this rebuttal discussion period.

Thank you for your feedback and the analysis of our manuscript. We will address below the concerns you raised.

Weaknesses

1. The work is primarily an empirical analysis...

We respectfully disagree with the characterization that empirical contributions lack value at NeurIPS. The NeurIPS call for papers explicitly states: 'We also encourage in-depth analysis of existing methods that provide new insights in terms of their limitations or behaviour beyond the scope of the original work." While our contribution is empirical, we address a well-recognized problem of characterizing the differences in information flow across modalities in VLMs. By exposing a single-token bottleneck that can both steer and break multimodal models, our study provides actionable knowledge for robustness, safety, and efficiency research. We believe empirical contributions are fundamental to scientific progress and provide essential foundations for future algorithmic developments.

2. The paper omits discussion of prior studies...

We thank the reviewer for the suggestions. We will add the relevant literature in the related work section. However, all of these papers deal with VLMs that output only text. The main finding of our paper is about VLMs, which can perform jointly in image understanding and image generation tasks and, as such, remain entirely novel.

3. The paper studies only limited models

In our work, we explore the cross-modal communication in a diverse set of models. We study i) VLMs which output images and text (Emu3, Chameleon, Janus) and output only text (Pixtral, Llava) ii) VLMs trained from scratch (Emu3, Chameleon) and fine-tuned on pre-trained LLM backbones (Pixtral, Llava, Janus), and iii) models with discrete, low-level VQ-VAE vision encoders (Chameleon, Emu3) and high-level, text-aligned CLIP-like encoders (Janus, Pixtral, Llava). Both Qwen2.5-VL and InternVL2.5 are multimodal-in-input but unimodal in output, like Llava and Pixtral. As we write in lines 54-55 and 319-321, our findings show that the narrow gate allows for selective ablation and steering, but “none of these properties hold for Llava and other unimodal-output VLMs”, and the ability to “produce multimodal responses is crucial” for the emergence of this phenomenon. For these reasons, based on the experiments we performed on LLava and Pixtral, we do not expect these models to present a narrow gate.

4. The paper appears somewhat rushed and underdeveloped...

The reviewer previously remarked that “the conclusions are supported by consistent evidence across different models and tasks” and this point of view is also shared by reviewer R1ay, who recognises that we base our claims on extensive statistics across several VLMs. Our conclusions stem organically from the experimental results. 1) The high attention weight and large semantic content indicate [EOI] to be a possible narrow gate for Chameleon and Emu, and 2) the selective ablation proves this to be correct, since the removal of a single token effectively destroys performance over a diverse set of downstream tasks.

If the reviewer has more specific concerns about any of these points, we would be happy to provide clarification in the author-reviewers discussion phase. Otherwise, we politely ask them to consider raising their score.

The authors' response is too brief and does not provide useful information. In particular, some important conclusions in the paper have already been presented in the works I cited, and the authors have not included the additional experimental results that I consider necessary. However, after seeing the discussions from other reviewers, I believe there is no longer a need for me to participate in the discussion. If the authors clearly explain how they will address the relationship with strongly related prior work in the final version, I will consider raising my score.

We thank the reviewer for their reply. Below, we address the remaining concerns:

The authors' response is too brief and does not include the experiments I requested.

While we did not run experiments on Qwen2.5-VL and InternVL2.5, we did conduct experiments on models that belong to the same class, and our findings are relevant for the models suggested by the reviewer. Specifically, both Qwen2.5-VL and InternVL2.5 share the following characteristics:

1. Training: both are optimized with a next-token language-modeling loss on their text streams.
2. Architecture: each builds on a large, pre-trained transformer-based language model backbone and a MLP connector.
3. Visual encoders: all employ a Vision Transformer (ViT) to extract image features.

All of these elements can be found in Pixtral-12b and LLaVA-Onevision, which we experimentally analyze in our work and show no evidence of a narrow gate. As we discuss, the necessary conditions for the narrow gate phenomenon are: a) models capable of generating multimodal output, b) models trained from scratch on multimodal data, and c) models with a low-level vision encoder. None of these three criteria are met in Qwen2.5-VL or InternVL2.5.

Thus, while we did not directly experiment on Qwen-2.5-VL and InternVL2.5, our analyses on closely related models provide strong evidence that the narrow gate phenomenon does not occur in this class of models.

If the authors clearly explain how they will address the relationship with strongly related prior work in the final version, I will consider raising my score.

We briefly expand on the merits of these works in connection with our findings:

1. Mitigating Multimodal Hallucination from an EOS Decision Perspective. This work studies information flow in LLaVA on captioning tasks, observing the distributed integration of visual and textual information when predicting EOS. This work does not directly probe the semantic content or localization of information in EOS, nor its role in cross-modal communication, a key question in our study.

2. Mitigating Object Hallucination via Concentric Causal Attention. It analyzes custom LLaVA-style models and shows that the information flow from distant visual tokens is weaker due to a decaying effect in the RoPE positional embedding, causing hallucination when the objects are far away from the referring textual part of the prompt. We discussed a related phenomenon in lines 314 (Discussion) and 224-225 (Appendix). We will explicitly discuss this work in our revised paper.

3. Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. It shows that in text-only output VLMs, highly attended textual tokens of little semantic information tend to aggregate the content of a previous sentence, similar to register tokens ([a,b] already cited). We will expand our discussion of this connection in the Appendix section “Special Tokens, Memory Tokens, Registers.”

4. Controlmllm: Training-free visual prompt learning for multimodal large language models proposes a post-hoc method for enhancing text-to-image attention. This is distinct from our work, as the narrow gate phenomenon we describe arises naturally in model architecture and training, not via inference-time interventions.

5. Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs is very similar to 3. We also note that this work is concurrent with our submission accordign to the NeurIPS 2025 policy (see [c]). We will include this work in the same place as 3.

6. An image is worth 1/2 tokens after layer 2. We have already discussed this work in our section on “Information Flow in Text-Only VLMs and Special Tokens” in the related work (Appendix). However, since reviewers are not required to read the appendix in detail, we understand it may not have been visible. In the final version, we will additionally reference it at the end of our Discussion (l. 330–333) when connecting to memory tokens and computational efficiency strategies.

We hope these clarifications address your concerns.

References

[a] Darcet et al., Vision Transformers Need Registers.

[b] Neo et al., Towards Interpreting Visual Information Processing in Vision-Language Models

[c] neurips.cc/Conferences/2025/PaperInformation/NeurIPS-FAQ

We thank the reviewer for their comments that our results and analysis are both well structured and surprising, and that they could help design better ways to train natively multimodal models. In the section below, we will address your concerns.

Weaknesses

a. Ambiguity cross-modal attention is computed:

• 1: The cross-attention pattern reported in Fig. 3 remains quantitatively consistent if we analyze only the answer tokens as the reviewer suggests. To answer the reviewer's concern, we repeated the experiment described in Sec. 3.2 on Chameleon-7b, measuring the attention patterns only on the token generated after the prompt “ <image> This animal is a _”. Also in this setup, the highly attended image tokens are the 32nd, [EOI], and last image ones, with [EOI] receiving 44% on the total cross modal attention between layers 1 and 8, the 32nd receiving 54% of the cross modal attention between layer 5 and 12 and the last token 73 % of the total cross modal attention after layer 12. Therefore, the highly attended tokens do not change if we restrict the analysis to the answer tokens.

• 2: In Llava and Chameleon-7b, the unimodal “text->text” attention makes up from 12% to 24% of the total attention throughout the model, much smaller than the cross-modal part. The general patterns of the cross-modal part remain unchanged when we include the unimodal one. We’d like to highlight, however, that the primary focus of our work is on the communication of information between image and text in Vision-Language Models. The answers can be grounded on the visual part of the input only through the cross-modal part of the attention, regardless of how strong it is relative to the unimodal attention. For this reason, in Fig.3, we report where the information flows in the cross-modal attention block.

• 3: We performed additional experiments to verify that the patterns of Fig. 3 are task agnostic and remain qualitatively unchanged in the captioning COCO, Flickr, and visual understanding VQAv2 tasks. The experiment depicted in Figure 4 requires the selection of a set of representative concepts associated with a dataset. This selection is simple for Imagenet, where the meaningful semantics is given by the classes. It is less trivial in Flickr30k (or VQAv2). To address the reviewer's concern, we extend the experiments to a subset of 10000 samples taken from the Flickr30k dataset. To extract the important concepts contained in the captions, we used a gte-Qwen1.5-7B-instruct[1] (a Sentence Transformer [2]) to map each caption to a vector encoding its semantics. We then measure the similarity between the hidden representations of [EOI] and the Flickr30k embeddings from the Sentence Transformer. We compute the representation similarity with the Neighborhood Overlap. The result of this experiment is consistent with the one on ImageNet: the similarity between [EOI] and the captions embedding increases from 0.0 to 0.16 between layers 5 to 9, and it remains approximately 0.16 for all the remaining layers. For the average image embedding, and all the other image embeddings, it remains close to zero.
  We also extended the analysis to two other classification tasks: Caltech101[3], comprising 101 classes of different objects, and DTD Dataset [4] with 47 classes of textures inspired by human perception. Also in these cases, the overlap profiles exhibit trends consistent with COCO captions and ImageNet.

b. Are attention patterns predictive

As we write in lines (165, 166), a candidate's narrow gates must "(i) have a large weight in the text-image attention, and ii) have a rich semantic knowledge of the image". Therefore, we never implied that attention patterns alone are predictive of tokens acting as narrow gates. The analysis of Sec. 3.2 is aimed at discussing both these two necessary conditions, with the section paragraph “Probing the semantic content of visual tokens” showing that in Emu3 and Chameleon, only [EOI] retains visual semantic information and satisfies condition (ii) and can thus be considered a candidate narrow gate. The final evidence that [EOI] acts as a narrow gate is given by the ablation results shown in Table 1. By addressing your previous concern a) we also showed that the analysis of Sec. 3.2 generalizes across datasets (Caltech101, DTD) and tasks (COCO-captioning).

c. Actionability of the findings

Between lines 323 and 333, we discuss several potential practical implications of our findings, suggesting that a possible way to make the models more robust and “encourage a more distributed communication can be masking the text-on-[EOI] attention during the last part of training” (lines 325-326). To validate experimentally this claim and address the reviewer's concern, we fine-tuned Chameleon-7b and Emu3 on a dataset composed of 10k samples from the LLaVA training set and 5k samples from the VQAv2 training set, masking the attention to [EOI] during training. To evaluate performance, we used 50 validation samples each from VQAv2 and COCO. Results are summarized below:

• Chameleon:

  • VQAv2 accuracy improved from 50% to 62% (base). With ablation of [EOI], it improves from 0.05% to 52%.
  • COCO-CIDER score increased from 0.48 to 0.72 (base). With ablation of [EOI], the score increased from 0.06 to 0.66.

• Emu3:

  • VQAv2 accuracy improved from 44% to 50% (base). With ablation of [EOI], it improves from 35% to 46%.
  • COCO-CIDER score increased from 0.73 to 0.87 (base). With ablation of [EOI], the score increased from 0.40 to 0.82.

To validate the importance of keeping [EOI] masked during training, we performed a standard fine-tuning run without masking [EOI]. Then, we repeat the ablation experiment in Tab. I and we measured the effect of ablation [EOI] after the fine-tuning.

• Chameleon: Final VQAv2 accuracy was 53% (base) and 35% with [EOI] ablation. COCO-CIDER was 0.75 (base) and 0.05 with [EOI] ablation.

• Emu3: Final VQAv2 accuracy was 45% (base) and 43% with [EOI] ablation and COCO-CIDER was 0.82 (base) and 0.48 with [EOI] ablation.

These results show that the training with [EOI] masked is an effective strategy to remove almost completely the gating mechanism of [EOI]. Notably, it can also improve performance versus standard fine-tuning except for COCO-CIDER in Chameleon. The masked-finetuning, targeted on [EOI], is an example of how the findings of this work can be used to improve the model's performance and robustness.

d. Influence of the visual encoder

We agree with the reviewer's observation that the models analyzed can not fully disentangle the impact of the vision encoder from that of the training protocol. To shed light on this point, in an experiment we performed after the submission deadline, we analyzed LIQUID [5]. This model is unified multimodal in output, differing from Chameleon only because its LLM backbone was finetuned from Gemma-7B and rather than being trained from scratch. In particular, LIQUID encodes the images with the same VQ-VAE as Chameleon. In the table below, we report the ablation analysis for the VQAv2 task. The results indicate that LIQUID behaves similarly to Janus and the other unimodal-output models: removing the [EOI] token does not affect performance, whereas removing the internal image tokens does degrade it. This experiment shows that having a VQ-VAE encoder is not sufficient to produce the narrow gate, suggesting that the natively multimodal training process and objective function play crucial roles.

Questions

How would Fig. 2 look like for [EOI]?...

We have measured the median cosine similarity between the [EOI] embeddings and i) random text token embeddings and ii) random image token embeddings chosen from 200 random samples of the Flickr dataset. For the Chameleon 7b model, both profiles start similarly, with an initial peak at 0.10 at layer 2. After layer 10, the cosine similarity with image embeddings drops to 0.01, and the cosine similarity with text remains constant at 0.10. Both these values are well below the average text-text and image-image similarity.

Have you analyzed the attention pattern of [EOI] wrt the rest of the image tokens for the different models?...

This seems not to be the case. To evaluate the sparsity of the attention pattern of the [EOI] tokens into the image, we calculated the (normalized) entropy of the [EOI] row of the attention pattern. We have done so for the Chameleon 7b and Llava models. In the former - even if we exclude the anomalous contribution from the 32nd token - Snorm fluctuates between 0.5 and 0.82. However, Snorm is much higher in the Llava model, with values fluctuating between 0.73 and 0.99, indicating that in some layers the attention from [EOI] in Llava is almost uniform.

Paper formatting concerns

Fig. 3 and Fig. 4 change the legend...

Thank you, we will update the legend in the camera-ready version.

What is in Eq. 2?...

Thank you for raising this clarity issue. We will add an example in the method section of the paper after Eq.2 is defined.

[1] huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct

[2] Reimers et al., Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

[3] Li et al, Caltech 101, doi.org/10.22002/D1.20086

[4] M. Cimpoi, et al. Describing Textures in the Wild

[5] Junfeng, et al. "Liquid: Language models are scalable and unified multi-modal generators."

Thank you again for your detailed suggestions. If we manage to address your main concerns, we politely ask you to consider raising your score. If, on the other hand, there are some remaining questions, we are happy to address them during this rebuttal discussion period.

a-b. I would suggest to integrate them in a revised version of this submission

We are pleased to hear that our replies addressed your concerns in a-b. We will add the plots of the experiments in the appendix and discuss them in Sec. 3.2

Clarification of fine-tuning scope and related questions of point c.

Scope. As a general comment, we would like to emphasize that our fine-tuning experiment was done primarily to show that we can effectively remove the narrow gate, making the cross-modal communication more distributed. The performance increase was not the main goal, only a side remark we made.

Questions:

c. why would masking [EOI] attention during SFT result in increased performance? [...] Any intuition?

A possible intuitive explanation is as follows:
When EOI is masked, the model is forced to learn a new way to communicate the visual information without relying on EOI. During inference, EOI is unmasked and will also communicate its own visual information, useful for the downstream task. If the model can combine redundant useful information to solve a task, this generally helps generalization [1, 2]. This can be intuitively seen as a form of (implicit) ensembling of predictors that is known to improve the performance of ML models.
However, this is only a speculation, and more evidence and analysis are needed to validate it.

c. not giving access to it should make SFT harder?

If by harder we mean that the training loss is higher at the beginning of training and the convergence is slower, then yes, the masked training takes more iterations to converge. However, even though masking [EOI] may slightly delay convergence, the model can still learn to route information effectively through different pathways.

c. It also seems that even without the masking of [EOI], fine-tuning ameliorates the drop in performance

This is not the case for all tasks. In COCO captioning, fine-tuning without masking does not reduce the drop in performance:

• Chameleon: performance gap increases from 0.48 → 0.06 (before) to 0.75 → 0.05 (after)

• Emu3: remains similar, from 0.73 → 0.40 (before) to 0.82 → 0.48 (after)

For VQAv2, we fine-tuned on some examples from that dataset. This likely helped the model learn task-specific conventions (e.g., short, one-word answers), which likely made the validation drop smaller even without masking EOI.

Generally speaking these results and analysis would have made the original submission way more interesting so I woudl suggest to add them to a revision with a proper comment.

We will add the fine-tuning experiments in a dedicated section (3.4) at the end of the results.

d. I'm not sure this addresses my concern. If LIQUID is equivalent to chameleon wrt the visual encoder, it does not show whether the distribution of the attention comes from the pre-training of the vision encoder or the training objective?

In our previous answer, we tackled the question of whether the main discriminating factor for the narrow gate is the training objective or a powerful (high-level) image feature extractor. We used LIQUID because we believe it can better disentangle three factors:

1. training objective (multimodal output vs text-only output);

2. training “history” (backbone trained from scratch vs fine-tuned on pre-existing LLM);

3. vision encoder (low-level tokens vs text-aligned “semantic rich” tokens);

LIQUID, Metamorph, Chameleon, and Emu all share the same training objective in that they can generate multimodal outputs within a unified architecture.

LIQUID differs from Chameleon/Emu only in its training “history”: LIQUID’s backbone is fine-tuned from an existing LLM, while Chameleon/Emu are trained from scratch. Our previous reply shows that this is sufficient to remove the modality gap and the narrow gate in this model.

Metamorph has two elements that differ from Chameleon/Emu: training history and vision encoder. First, its backbone is fine-tuned from an existing LLM (as is the case for LIQUID). Second, it uses a SigLIP encoder, which produces more “text-aligned” embeddings due to the contrastive pretraining. Given that the first condition is sufficient to suppress the narrow gate, and further, all the models trained with strong vision encoders we analyzed (LLaVA, Pixtral, Janus) had distributed image-text communication patterns, we do not expect that Metamorph will have a narrow gate token.

Please let us know if we addressed your concerns.

[1] Doimo et al., Redundant representations help generalization in deep neural networks, NeurIPS 2022

[2] N Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 2014

Does this mean that the two reported results above are trained for a different number of iterations?

No, both training runs were done on the same 15k examples for one epoch for a total of 117 gradient updates.
Inspecting the training loss of intermediate checkpoints, we noted that when the training is masked, more gradient updates were needed to converge.

This type of experiments would have made a way more comprehensive submission

Thank you for appreciating the value of this experiment. We are happy to include it in the camera-ready version as an interesting application of our main finding about the narrow gate in VLMs.

We are also grateful for the time you spent on this review and your comments. Both helped us improve the quality of this work.

We thank the reviewer for their recognition of the originality of our findings and the possible future applications for a better understanding of MMLMs. Below, we address your concerns.

Weaknesses

Limited scope of findings: the paper [...] focuses almost entirely on demonstrating the gate’s existence and lacks any further substantive or surprising conclusions.

We believe that our paper goes beyond demonstrating the existence of the “narrow gate” at the [EOI] token. First of all, this finding is tightly related to the general, and counterintuitive, observation that native multimodal-output models maintain a large modality gap across all layers (Fig. 3), a property generally considered to be a drawback for cross-modal reasoning [1,2].

We support our findings by analyzing the impact of diverse architectures and training protocols on the narrow gate phenomenon. These insights are of practical importance. Localizing and characterizing how these models develop an effective communication mechanism despite the modality gap is a finding central and timely in the multimodality community, since this class of models is being employed in several state-of-the-art multimodal training pipelines [3,4], motivated by advances in autoregressive image generation [5].

We discuss the potential applications where the narrow gate can be exploited proactively in Sec. 3.4 (e.g., semantic steering via activation patching) and propose how to use it to reduce memory and computing usage. We also suggest that a possible way to make the models more robust and “encourage a more distributed communication can be masking the text-on-[EOI] attention during the last part of training” (lines 325-326).

To validate experimentally the last claim and address the reviewer's concern, we fine-tuned Chameleon and Emu3, masking the [EOI] tokens on an instruction dataset composed of 10k samples from the LLaVA training set and 5k samples from the VQAv2 training set. In this case, we can significantly reduce the detrimental effect of [EOI] ablation on performance. Results are summarized below:

• Chameleon:

  • VQAv2 accuracy improved from 50% to 62% (base). With ablation of [EOI], it improves from 0.05% to 52%.
  • COCO-CIDER score increased from 0.48 to 0.72 (base). With ablation of [EOI], the score increased from 0.06 to 0.66.

• Emu3:

  • VQAv2 accuracy improved from 44% to 50% (base). With ablation of [EOI], it improves from 35% to 46%.
  • COCO-CIDER score increased from 0.73 to 0.87 (base). With ablation of [EOI], the score increased from 0.40 to 0.82.

To validate the importance of keeping [EOI] masked during training, we performed a standard fine-tuning run without masking [EOI]. Then, we repeat the ablation experiment in Tab. I and we measured the effect of ablation [EOI] after the fine-tuning.

• Chameleon: Final VQAv2 accuracy was 53% (base) and 35% with [EOI] ablation. COCO-CIDER was 0.75 (base) and 0.05 with [EOI] ablation.

• Emu3: Final VQAv2 accuracy was 45% (base) and 43% with [EOI] ablation, and COCO-CIDER was 0.82 (base) and 0.48 with [EOI] ablation.

These results show that masking [EOI] during training is crucial for removing almost completely the gating mechanism, distributing the communication of visual information across multiple tokens, and removing the issues associated with a constrained information flow (see lines 324-327). Notably, it can also improve performance versus standard fine-tuning, except for COCO-CIDER in Chameleon.

Our results, therefore, provide two main contributions. First, a new empirical finding and a mechanistic explanation for how native multimodal training impacts information flow, advancing the understanding of VLM internals. Secondly, they suggest practical, actionable strategies for exploiting or mitigating the issues associated with the presence of the narrow gate.

Questionable validity of the zero‐ablation method [...].

The zero-ablation approach may distort the model's internal representations. However, for all the models, the zero-ablation on any single token (besides [EOI] in Chameleon and Emu3) does not cause a performance drop despite the distribution shift caused by the ablation (see Tables 1 and A2). This suggests that the performance drop observed in Emu3 and Chameleon when [EOI] is zero-ablated is not due to OOD effects, but instead to the gating mechanism of [EOI].

We agree with the reviewer that milder ablation techniques exist in the literature, the mean-ablation being one of the most common alternatives [6]. In this case, the ablated token is substituted with its mean value taken over the dataset. We applied the mean ablation to [EOI] in the COCO dataset in Chameleon-7b, and also in this case, we observed a degradation in CIDER from 0.46 to 0.08.

No causal analysis of narrow‐gate emergence [...].

In our study, we analyzed VLMs composed of a different set of architectures (visual encoders, connectors) and trained with different pipelines (from scratch or fine-tuned on existing LLM backbones). This wide variety of models allowed us to pin down the conditions to observe a narrow gate: the model must be trained from scratch on multimodal data, both for understanding and generation tasks, and must possess a low-level vision encoder (VQ-VAE). Studying how the narrow gate forms during training is beyond the scope of our work due to the lack of intermediate training checkpoints for Chameleon and Emu3.

Questions

At lines 288–290, [...] one must examine the value of the last token’s ($\chi^{l,gt}$). Have you checked this?

We agree with the reviewer on this observation. We measured $\chi^{l,gt}$ also on the last token, and its value grows from 0 to 0.45 between layer 0 and 24 in Chameleon-7b, from 0 to 0.58 between layer 0 and 44 in Chameleon-34, and from 0 to 0.37 between layer 0 and 11 in Emu3. In all cases, the growth of the $\chi^{l,gt}$ in the last token comes after the one on [EOI]. When [EOI] is ablated, also the $\chi^{l,gt}$ in the last token drops (see Table 1). Therefore, the claim at lines 288-290 is fully supported by the experiment.

The notation ($A_{\text{img}\rightarrow\text{text}}$) on line 88 is ambiguous. [...]

$A_{i,j}$ is a matrix which we introduce in line 87 as the “attention from token i to token j”. Given that in our setup the image comes before the text (line 85), if we set “$N_{[EOI]} < i ≤ N, 0 < j < N_{[EOI]}$”, $A_{\text{img}\rightarrow\text{text}}$ is the rectangular block of the attention matrix in which the text tokens at positions $N_{[EOI]} < i ≤ N$ attend to the image tokens $0 < j < N_{[EOI]}$. We denote this block of the attention matrix as cross-modal attention (see end of line 86) and analyze it in Sec. 3.2. We use $A_{\text{img}\rightarrow\text{text}}$ in Sec. 3.4 at lines 233 - 250 - 255 to reference the part of the attention matrix set to zero when we ablate the communication between image and text tokens.

In the Chameleon model, what is your hypothesis for why the 32nd token serves as the narrow gate? [...]

The 32nd token in chameleon-7b is not a narrow gate. In lines (165, 166), we write that a token must “(i) have a large weight in the text-image attention, and ii) have a rich semantic knowledge of the image”. The analysis of Sec. 3.3 shows that condition (ii) is not met for token 32nd, which has a $\chi^{l,gt}$ with the imagenet concepts close to zero in all the layers (see Fig. 4 left). In a side experiment, not reported in the paper, we verified that the norm of the token 32 at layers $5-12$ is $\times100$ larger than all the other image tokens. Massively activated tokens in LLMs can occur as artifacts in LLMs, and their origin is the subject of current research [7].

[1] W. Lieng et al., Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.

[2] Q. Huang et al., Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate.

[3] https://openai.com/index/introducing-4o-image-generation/

[4] M. Shukor et al., Scaling Laws for Native Multimodal Models

[5] P. Sun et al, Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

[6] Wang, et al. "Interpretability in the wild: a circuit for indirect object identification in GPT -2 small."

[7] Sun, M. et al, Massive Activations in Large Language Models

Thank you again for your comments and suggestions. If we manage to address your main concerns, we politely ask you to consider raising your score. Please don’t hesitate to let us know if you have further questions. We are happy to answer your questions during this rebuttal discussion period.

We are deeply grateful for your timely response and the appreciation of our replies.

Response to W1

Yes, we can and will incorporate this experiment after section 3.4 as section 3.5.

Response to W2

The value 0.46 reported in this rebuttal was measured on 300 MS–COCO examples. In Table 1, the experiment was run on 2000 examples. This was done to speed up the experiment in the rebuttal phase.

Response to W3

Our hypothesis for the emergence of the narrow gate stems from the strong modality gap observed in native multimodal models.

This gap is due to the VQ-based image tokenizer, which is known to encode low-level information from the images, while the tokenized words capture higher semantic information [1, 2]. This different degree of abstraction between text and image tokens is, by construction, present also in the last hidden representations, which are decoded back into pixels and words via the VQ-decoder and the unembedding matrix.
Our first finding, shown in Fig. 2, is that this modality gap is also evident throughout all hidden layers in models like Chameleon and Emu3. We observe that image and text embeddings form distinct homogeneous clusters, with low cross-modal cosine similarity, indicating that the modalities follow separate paths occupying different regions in the embedding space.

Given this separation, some mechanism or mediator is needed to communicate the visual information to the text embeddings. This role cannot be played by tokens within the images or text sentences. Due to the autoregressive nature of the transformer, image tokens only generate the following image token, and text tokens do the same for text. Since the residual stream maintains the modality-specific separation, and the attention heads themselves must have at least some modality-specific role, the cross-modal interactions are limited.

This leaves the end-of-image special token [EOI] as a unique candidate for cross-modal mediation. Since [EOI] is not aligned with either image or text embeddings, it can serve as a bridge. Supporting this, we find that the cosine similarity between [EOI] and both image and text embeddings is below 0.1, much lower than typical intra-modal similarities.
This suggests that [EOI] occupies a neutral space and can transfer information between modalities.

Response to Q1 & Q3

Thank you!

Response to Q2

We will make explicit that A_{img->text} refers to a submatrix and not to a scalar using the slicing notation, which will make it more transparent which columns are selected. This is an update of line 88:

We define $A_{\text{img} \to \text{text}} = A_{\left[N_{\text{EOI}} + 1: N, 0: N_{\text{EOI}} -1 \right]}$, where the notation $A_{[a:b,c:d]}$ denotes the submatrix consisting of rows from a through b, and columns c through d inclusive.

[1] Ge et al., Planting a SEED of Vision in Large Language Model;

[2] Wu et al., Towards Semantic Equivalence of Tokenization in Multimodal LLM.