Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models

5. Relative position of text and image

In the standard LLaVA model, we have explored putting visual embeddings at different locations and observed no noticeable performance difference. Specifically, we try placing the input image before both system prompt and text query, after system prompt and before text query, as well as after both system prompt and text query. We do not observe substantial performance differences. This may imply that the position of an image within text does not play a crucial role.

In the case where the relative position between visual and textual embeddings is absolutely necessary, it is easy to re-introduce positional info in our D-Attn framework while retaining the nice property of debiased positional embedding. Specifically, as shown in the figure below:

• Debiased positional encoding

we can set the position of all visual embeddings to a constant (0 here), and set the position of textual embeddings as usual (1, 2,...,5 here). In this setup, each textual embedding has unbiased positional weights toward all visual embeddings, while textual embeddings at different positions are aware of the positional difference with visual embeddings.

6. Naming of V2V Diagonal-Attn

Applying two $fc$ layers is the result of diagonalizing the standard V2V Self-Attn. This motivates us to reuse and share the $fc_v$ and $fc_o$ from the pre-trained attention module. We also inherit other designs in the self-attention module, such as pre-layernorm, post-layernorm, and skip connection. Overall, V2V Diagonal-Attn is more than just applying two fc layers. We therefore name this design V2V Diagonal-Attn as a hint of its origin and motivation.

1. V2V Diagonal-Attn for single image and multi-images

In the case of single image, V2V Diagonal-Attn is different from standard attention. Note that |V| represents the number of visual embeddings, not number of images as defined in L19, L74, L136, etc. For example, an image will be encoded into 576 visual embeddings by CLIP ViT-L/14-336. Standard Self-Attn in LVLM computes the attention between each pair of visual embeddings to model the contextual information. For our proposed V2V Diagonal-Attn, as described in Section 2.2, since the visual encoder has already modeled this contextual information, we propose not to re-learn it within the LLM, thereby saving substantial computations.

In the case of multiple images, since V2V Self-Attn is for capturing the contextual information between visual embeddings, when diagonalized, we may expect weakened modeling of this contextual information. In practice, there is no noticeable performance degradation. Specifically, our model follows LLaVA's anyres strategy, where a high resolution image is divided into several patches, each encoded by CLIP independently. In this case, each patch has no such contextual information about other patches. However, as we can see in Table 2 of the paper, incorporating V2V Diagonal-Attn does not degrade performance.

To further confirm this observation, we train a VideoQA model following LLaMA-VID [1] , where the input is multiple frames (images) and the model has to model the contextual information across frames to answer the question. In this experiment, we again observed no performance degradation by incorporating V2V Diagonal-Attn.

Our explanation of this phenomenon is that visual embeddings could exchange information indirectly via textual embeddings. Concretely, each textual embedding gathers visual information via T2V Cross-Attn at the L-th decoder layer. Therefore, at the (L+1)-th layer, visual embeddings indirectly exchange information via text embeddings in T2V Cross-Attn.

2. Computational advantage of V2V Diagonal-Attn for high-resolution images

V2V diagonal attention is most advantageous in the case of high |V| such as high resolution images. Although it is straightforward to reduce |V| by downsampling the input high-resolution images, it has been widely shown in previous works such as LLaVA [1] that a high-resolution input image is necessary for better performance. Simply downsampling images for reducing computation may degrade model performance. In contrast, our proposed V2V Diagonal-Attn reduces the overall computational complexity without compromising model performance as shown in Table 2 of the main paper.

3. Novelty and contribution of proposed $\alpha$-weighting

$\alpha$ weighting is a novel way of merging visual and textual information from T2V Cross-Attn and T2T Self-Attn, respectively. In cross-attention-based LVLMs, merging visual and textual information is an inevitable and critical challenge. As we point out in Section 2.4 and Section 4 of the paper, existing methods involve significant architectural changes or introduce additional parameters, which can break the integrity and degrade the performance of pre-trained LLMs. Retaining equivalence with conventional LVLM attention preserves the pretrained LLM’s capabilities and thus leads to superior performance. Compared with other existing merging strategies, alpha weighting achieves superior performance as shown in Table 3 of the paper.

4. Visual and textual modality gap

Thanks for pointing out the relevant work. Similar to referenced paper, we did not try to prove whether this modality gap is good or bad, nor propose closing this gap. Instead, this observation of modality gap shown in Figure 5 challenges the common belief in modern LVLM works such as LLaVA, where the authors claim that visual embeddings are projected into the same space as textual embeddings and thus could be processed homogenously within an LLM. This observation motivates us to process visual and textual embeddings differently within the LLM. We thereby propose the D-Attn framework to achieve this goal. We will revise the writing in the latest revision to make it more clear.

Thank you for your detailed response. I will respond to the numbered points in turn below.

1, 2, 5. Thank you for clarifying the use of the “anyres” strategy, which is important for understanding your results. This should be explicitly mentioned and properly cited in the paper. At the moment the cited Liu et al. “Visual Instruction Tuning” is cited but not Liu et al “Improved Baselines with Visual Instruction Tuning” which discusses this strategy as LLaVa-1.5 (vs. LLaVa-1, which does not use it).

Your observations regarding multiple visual embeddings (including multiple images) in context are interesting and will strengthen the paper. I am still confused by L144—150 which motivate V2V attention by saying that “each visual embedding already encapsulates contextual information from other visual embeddings”, which seems to contradict the fact that separate images (or separate patches using anyres) are encoded separately. Can you please clarify this?

Regarding relative positioning of images and text, this is a useful clarification and I encourage the authors to add this to the revised paper.

I will appreciate further clarification here, as I think this does not answer my original question (also asked by reviewer tS4a). If alpha-weighting is mathematically equivalent to LVLM attention, what is being done differently (and what is being ablated in Table 3)?

Regarding the modality gap, I’m not sure I find this argument compelling. LVLMs such as LLaVa apply a projection from visual tokens to be processed similarly to textual tokens, but this does not necessarily assume that a modality gap does not exist after the projection — just that they can then be processed with the same decoder for empirically satisfying results. Hence, the presence or degree of this gap does not seem obviously relevant.

While I understand the motivation behind the name “V2V Diagonal-Attn”, the use of fully-connected layers, layer norm, and residual connections were already standard in pre-transformer architectures, and normally the use of the term "attention" implies the use of the attention mechanism built from keys, queries, and values. Hence I still find this name somewhat misleading.

Thanks for the prompt reply and detailed comments, we address the follow-up questions below.

1. Clarification of the term "single-image" used in the paper

Yes, that is correct. Here "single-image" refers to a single low-resolution image that is fed into CLIP as a whole. A single high-resolution image divided into patches following the anyres strategy should be considered as "multiple-images" because each divided patch is encoded independently with CLIP.

We will clarify this and clearly define what "single-image" means in the paper. We will also include the additional experimental results for multiple images in previous response to further justify that our D-Attn model also works well in multi-image scenarios.

2. Difference between proposed D-Attn and conventional S-Attn in LLaVA

No, that is incorrect. The proposed D-attn is different from LLaVA's self-attention mechanism:

1. LLaVA applied one self-attention to the concatenated visual and text tokens, while D-attn decompose this architecture and applied three attention operations (V2V, V2T, T2T) with a merging strategy,
2. we replace the V2V self-attention with Diagonal-Attn/FCs so that the computational complexity is reduced from $\mathcal{O}(|V|^{2})$ to $\mathcal{O}(|V|)$.
3. We replaced the original position encoding with the proposed debiased positional encoding which boosts model performance.

D-attn would degenerate to LLaVA's self-attention only if 2) and 3) are both NOT applied. Given that 1), 2), 3) are all applied, the proposed D-attn is significantly different from LLaVA's attention mechanism. Although LLaVA claimed alternative designs of incorporating visual information, no experiments were presented to verify the effectiveness of these designs. In fact, when comparing the $\alpha$-weighting strategy to existing merging strategies for LVLM with multiple attention operations (e.g., Flamingo), the proposed merging strategy is novel and it performs better than other merging strategies as shown in Table 3.

We really appreciate your feedback and your patience for the discussion. If there is further question about this contribution or the difference between D-Attn with other methods like S-Attn, Flamingo, LLaVA, etc, we are happy to provide further clarification and context.

1. The main contribution and novelty of this paper

$\alpha$ weighting is a novel way of merging visual and textual information from T2V Cross-Attn and T2T Self-Attn, respectively. We agree with the reviewer that using cross-attention between visual and textual embeddings has been explored in, e.g. Flamingo, and did not claim these parts as novel or our contribution in the main paper. We discussed the nuance difference between our D-Attn framework and other cross-attention-based LVLMs in L486-499. Compared with other existing merging strategies, $\alpha$ weighting achieves superior performance as shown in Table 3 of the paper.

In addition to the novel $\alpha$-weighting strategy, in this paper we propose the novel V2V Diagonal-Attn and debiased positional encoding. V2V Diagonal-Attn substantially reduces computation from $\mathcal{O}(|V|^2)$ to $\mathcal{O}(|V|)$ without compromising performance. Debiased positional encoding removes undesirable positional bias between textual and visual embeddings, leading to consistent performance improvements. Ablations for these two major contributions are presented in Table 2 of the paper.

2. Motivation of decomposed attention

Thanks for pointing out the relevant work. Similar to referenced paper, we did not try to prove whether this modality gap is good or bad, nor propose closing this gap. Instead, this observation of modality gap shown in Figure 5 challenges the common belief in modern LVLM works such as LLaVA, where the authors claim that visual embeddings are projected into the same space as textual embeddings and thus could be processed homogenously within an LLM. This observation motivates us to process visual and textual embeddings differently within the LLM. We thereby propose the D-Attn framework to achieve this goal. We will revise the writing in the latest revision to make it more clear.

The improvement is driven by our novel designs, including V2V Diagonal-Attn, debiased positional encoding, and alpha weighting, to process visual and textual embeddings differently. The decomposition per se does not directly bring performance improvement. Rather, it gives us more control and flexibility, thus enabling us to derive and incorporate the novel designs mentioned above for performance improvement.

3. Analysis of alpha-weighting

Thanks for the suggestion. We show $\alpha_V$ on these benchmarks across attention heads and LLM layers in the following links:

• SQA-I
• GQA
• VQA-T

For each layer, we sort $\alpha_V$ across heads for better visualization. We can see that GQA and VQA-T both have high $\alpha_V$ across heads and layers, while SQA-I has much lower $\alpha_V$ . This observation is in consensus with MM-Star [1], where the authors point out that many questions in SQA do not require visual information to answer. We will add these figures to the paper in the latest revision.

Thanks for your prompt reply and clarification. We explain more about the merging strategies as follow.

In terms of the cascade variant, the concept of merging weight between T2T Self-Attn and T2V Cross-Attn is not applicable. In the cascade variant (described in L412-413), input textual embeddings first perform T2T Self-Attn, and then the output of the T2T Self-Attn performs T2V Cross-Attn. These two operations were performed in series and there is no such weighting as our alpha-weighting strategy in D-Attn or as standard Self-Attn.

Nevertheless, we analyze the sigmoid variant, where T2V Cross-Attn and T2T Self-Attn are weighted summed with learnable gates $\sigma$ as described in L414-415. Different from our $\alpha$-weighting, the merging weights of $\sigma$ gate are fixed after training and are independent of data. Therefore, it is unable to adaptively adjust the weighting between visual and textual embedding depending on the task. Furthermore, in our latest analysis, the learned sigmoid gates have a mean and standard deviation of 0.620 and 0.00255, respectively, across heads and layers, unlike our alpha weighting strategy that has greater expressibility across heads, layers, and datasets as shown in our previous reply.

1. Novelty and contribution of proposed $\alpha$-weighting merging strategy

$\alpha$ weighting is a novel way of merging visual and textual information from T2V Cross-Attn and T2T Self-Attn, respectively. We agree with the reviewer that the operations of T2V Cross-Attn and T2T Self-Attn remain the same and did not claim these parts as novel or our contribution in the main paper. Compared with other existing merging strategies, $\alpha$ weighting achieves superior performance as shown in Table 3 of the paper.

2. Disadvantage of other merging strategies

Significant architectural changes are disadvantages in LVLM as these changes can compromise the integrity of the pre-trained LLM, potentially degrading its inherent capabilities. As we show in Table 3 of the paper, other merging strategies involving more architectural changes lead to suboptimal performance compared to our $\alpha$-weight strategy, which introduces minimal architectural/operational changes and retains equivalence with the native LVLM attention. Thanks for pointing this out, and we will revise the writing to make it more clear.

3. Analysis of the proposed method's effectiveness

We present the broken-down scores from MME, SEED, and MMB in the tables below.

Overall, we found that D-Attn model is particularly strong on tasks involving spatial and relational reasoning, such as (1) "position" in MME, (2) "Spatial Relation" in SEED, and (3) "object localization" and "spatial relationship" in MMB. It also performs well on tasks involving OCR and document understanding, such as (1) "OCR" in MME, (2) "Text Understanding" in SEED, and (3) "ocr" and "structuralized image-text understanding" in MMB.

4. Modalities misalignment

In this paper, we did not try to prove whether this modality gap is good or bad, nor propose closing this gap. Instead, this observation of modality gap shown in Figure 5 challenges the common belief in modern LVLM works such as LLaVA, where the authors claim that visual embeddings are projected into the same space as textual embeddings and thus could be processed homogenously within an LLM. This observation motivates us to process visual and textual embeddings differently within the LLM. We thereby propose the D-Attn framework to achieve this goal. We will revise the writing in the latest revision to make it more clear.

5. Writing and presentation issues

The purpose of Figure 4 is to highlight the performance gain of proposed D-Attn models against their S-Attn counterparts, while Table 1 is for the comparison with other SoTA methods with detailed numbers across a wide range of image benchmarks. We consider revising the presentation or replacing Figure 4 with other analyses in the latest revision. For other writing issues, we greatly appreciate the reviewer's efforts in reading the paper in detail. We will fix the inconsistencies in the latest revision to make the presentation more clear.

Thanks for the prompt reply and detailed comments, we address the follow-up questions below.

1. Contribution of $\alpha$-weighting merging strategy

In our previous response, the complete sentence of the statement is:

"We agree with the reviewer that the operations of T2V Cross-Attn and T2T Self-Attn remain the same and did not claim these parts as novel or our contribution in the main paper."

In the sentence, "these parts" refers to the operation of T2V Cross-Attn and the the operation of T2T Self-Attn. This statement does not contradict our claim of contribution in the paper regarding alpha-weighting, which is a merging strategy for merging the outputs from T2V Cross-Attn and T2T Self-Attn.

If there is any question about this contribution, we are happy to provide further clarification and context.

2. Figure 5 and the analyses of detailed scores from MME, SEED, and MMB

Thanks for the suggestions. We have removed Figure 5, and replaced it with the analyses of detailed scores from MME, SEED, and MMB in the manuscript.

3. More Context

To help understand the novelty and contribution of $\alpha$-weighting merging strategy, we provide more context about major LVLM architectures as follows:

There exist two ways of incorporating visual information in modern LVLMs: (1) self-attention for concatenated visual and textual embedding, and (2) Separate T2T Self-Attn for textual embeddings and T2V Cross-Attn for visual embeddings.

1. Self-attention only: Most modern LVLMs fall into this category, such as LLaVA, InternVL, QwenVL, etc. This line of work applies only one self-attention on concatenated visual and textual tokens without any explicit merging operation. Therefore, there is NO need to consider the strategy for merging T2T Self-Attn and T2V Cross-Attn. This line of work lacks the flexibility to process visual and textual embeddings differently. The lack of flexibility hinders the opportunity to process visual embeddings differently, such as V2V Diagonal-Attn for reducing computation, and debiased positional encoding for enhancing model performance.
2. Separate T2T Self-Attn & T2V Cross-Attn: Only a few LVLMs fall into this category, such as Flamingo, LLaMA 3, and our proposed D-Attn. This line of work explicitly separates T2T Self-Attn and T2V Cross-Attn. Therefore, it is inevitable and critical to consider the strategy for merging T2T Self-Attn and T2V Cross-Attn. This line of work provides the flexibility to process visual and textual embeddings differently.

In this paper, in order to process visual and textual embeddings differently, we adopt approach (2) with separate T2T Self-Attn & T2V Cross-Attn. We found in Table 3 of the paper that the merging strategies in existing literatures (e.g., flamingo) often hurt performance. The key to superior performance is maintaining the mathematical equivalence to LLM's self-attention ($\alpha$-weighting), which is novel and non-trivial in the field of LVLMs. To the best of our knowledge, this is the first work to mathematically derive $\alpha$-weighting for this class of architecture, and experimentally (Table 3) demonstrate its advantage over existing merging strategies.

Thank you for the detailed explanation and revisions made to the paper. I partially agree with the authors regarding their contributions. I was impressed by their active incorporation of the review comments into the paper. I have raised the score by 1 point.

1. Textual embeddings placed before visual embeddings

In modern LVLMs such as LLaVA, textual embeddings are always placed after visual embeddings in both training [1] and inference [2]. Due to the causal attention in LLM, visual embeddings will not act as query or attend to the textual embeddings appended afterward. Following this design, we derive our D-Attn framework assuming visual embeddings placed in front of textual embeddings as shown in Figure 1 of the paper.

If placing textual embeddings in front of visual embeddings is absolutely necessary, our D-Attn framework is flexible enough to deal with this scenario while maintaining its performance and computational advantages. Consider the scenario where visual embeddings are inserted between two segments of textual embeddings, the causal attention is illustrated in the figure below:

• Placing text before image

Following our D-Attn framework, we can merge the information from blocks 2 and 3, as well as blocks 4, 5, and 6 via our proposed alpha-weighting strategy. Similarly, V2V-Attn in block 3 can be diagonalized to save computation. This is a straightforward extension of our D-Attn framework, which maintains causal attention property.

We have explored the design of placing textual embeddings before visual embeddings in our early stage of development and found no noticeable performance gain. We therefore decide to place the visual embeddings at the beginning following LLaVA's design.

2. Bi-directional cross-modal attention

To the best of our knowledge, modern LVLMs (models developed after LLaVA) typically do not adopt bi-directional attention design. It is likely because bi-directional attention breaks the causality characteristics inherent to modern pre-trained LLMs, leading to potential performance degradation.

Nevertheless, our D-Attn is flexible enough to incorporate bi-directional attention. Consider the scenario where we apply bi-directional attention to textual embeddings of question and visual embeddings while maintaining causal self-attention on textual embeddings of answer to not break causality during inference, the attention operation can be illustrated in the figure below:

• Bi-directional attention

Following our D-Attn framework, we can merge the info from blocks 1 and 2; blocks 3 and 4; blocks 5, 6, and 7; via our proposed alpha-weighting strategy. V2V Attn in block 1 can also be diagonalized. Compared with vanilla uni-directional D-Attn in our main paper, this design incurs extra computation in block 2 V2Q Attn with $\mathcal{O}(|V||Q|)$. Thus, the overall computational complexity is still linear with respect to |V|.

3. Visual embeddings as query

To the best of our knowledge, there aren't many successful applications of LVLM for vision-centric tasks such as visual search and face matching. Embedding learning [1][2][3][4] may still be the SoTA in these scenarios.

Nevertheless, it is possible to extend our D-Attn framework to support such scenario. Similar to the bi-directional attention described above, the attention between image1 $V$, image2 $U$, and text can be illustrated in the figure below:

• Multi-images

We use bi-directional attention between $V$ and $U$. Following our D-Attn framework, we can merge the info from blocks 1 and 2; blocks 3 and 4; blocks 5, 6, and 7; via our proposed alpha-weighting strategy. $V2V$-Attn in block 1 and $U2U$-Attn in block 4 can both be diagonalized. The interaction between $V$and $U$ happens in blocks 2 and 3, where one image acts as query to gather information from another image.

4. Dynamically assign query, key, and value

The idea of dynamically assigning query/key/value is very interesting and we did not see any LVLM literature implementing this idea. Our D-Attn architecture can be extended with a dynamic assignment module to support this scenario. Switching the role of query/key/value does not change the computation complexity within the LLM, still resulting in a linear complexity with respect to |V|. However, the additional dynamic assignment module may introduce extra computational cost.