Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

Thank you for your detailed and positive feedback. We greatly appreciate your recognition of our contributions in patch-level localization and multi-semantic alignment. Below, we respond to the remaining concerns individually and clarify the corresponding points.

Q1: However, C-Abstractor has a higher entropy than both MLPs and linear projectors. This contradicts the reasoning and claim made here.

A1: Thank you for this observation. In our paper, we primarily compare the Von Neumann Entropy $H(\mathbf{V}\_{after})$ of linear and MLP projectors since they yield similar values when randomly initialized. We don't directly compare with the C-Abstractor because a random initialized C-Abstractor has a much higher baseline entropy ($7.4913$) compared to Linear ($4.8197$) or MLP ($4.8245$). This demonstrates that absolute entropy is influenced by the inductive bias of the model architecture. The C-Abstractor uses a deep CNN-based architecture with residual connections (ResNet blocks), which—even when randomly initialized—produces more diverse outputs. This higher initial entropy reflects the architectural prior of the network, not its training quality.

To fairly assess the projector’s training effect (i.e., information compression caused by learning), we can measure the entropy drop ΔH, which removes the influence of architecture priors: $\Delta H(V) = \frac{H_{random}(V) - H_{trained}(V)}{H_{random}(V)} \in [0,1]$. where we get: $\Delta H_{C-Abs} = 0.729> \Delta H_{MLP} = 0.578 > \Delta H_{Linear} = 0.485$. This demonstrates that the C-Abstractor achieves higher entropy reduction after training. Thank you for these insightful comments. We will include this discussion with a comprehensive comparison of the C-abstractor in our revision.

Q2: Additionally, the drop in entropy cannot be claimed as being simply due to the removal of "redundant information"; potentially useful visual information may have been lost. Certainly, compression is occurring, and does not detract from the central claims of the paper.

A2: Thank you for the comments. As briefly discussed in the conclusion, we agree with the reviewer’s perspective that over compression may potentially lead to useful visual information loss. There exists a tradeoff between redundancy removal and semantic loss when the compression level continues to increase. Within a proper compression range, more compression improves performance by removing redundant information and enhancing alignment. However, over-compression causes semantic information loss and performance degradation. To empirically validate this tradeoff, we conducted a controlled ablation where we varied the patch loss weight $\beta$ in the joint objective: $L = L_{caption} + \beta L_{patch}$. Larger $\beta$ encourages stronger patch-level alignment and induces more compression. As shown in the table, performance first improves then drops as $\Delta H$ increases, which validates our conclusion.

In the revision, we will clarify this in section 3 and Lines 281-282 and include these experiments in the appendix.

Q3: The choice of linearly increasing beta (hypeparameter for the align loss weight) is not justified or ablated in the paper.

A3: Thanks for the suggestions. Due to the limited space, we omit the ablation study in our main paper. We will now present our ablation studies on the hyperparameter $\beta$ in two aspects:

1. Linear increasing vs. fixed schedule: The linear schedule gradually increases $\beta$ to impose patch-level alignment progressively. This design choice stabilizes early training and prevents premature over-compression. As shown in the table, when comparing fixed $\beta=5$ versus linearly increasing $\beta$ from $0$ to $5$, the linear schedule showed better performance. Therefore, we adopted the linear increasing schedule for all subsequent experiments.

2. Optimal final $\beta$ value: Since $\beta$ in the objective function balances global-level and patch-level alignment, a larger $\beta$ emphasizes patch-alignment loss over caption loss. We examined the impact of $\beta \in \{0, 2, 5, 10\}$, where the baseline LLaVA-1.5 corresponds to $\beta = 0$. As shown in the table, when $\beta$ is too small ($\beta = 0$ or $2$), patch-level alignment remains insufficient, leading to suboptimal performance. Conversely, when $\beta$ is too large ($\beta = 10$), the model overly focuses on local regions, compromising its ability to establish comprehensive correspondence between the entire image and sentences. This imbalance degrades global-level alignment and ultimately harms overall performance.

We will include the complete ablation study in the appendix.

Q4: It would be helpful to see ablations with other LLM families (beyond LLAMA), as well as other projector types (such as transformer projectors).

A4: Thank you for the suggestion. We have ablations on different LLMs (Vicuna and LLAMA) and projector types (linear, MLP, C-Abstractor). Due to time constraints, we were not able to include additional ablations, which we plan to explore in future work. Specifically, we aim to incorporate LLMs with different parameter sizes or architectures such as Phi-3 (3.8B) and Mixtral-8x7B (MoE), and to investigate transformer-based projectors, including Q-Former[1] and D-Abstractor[2].

[1] Li, J., Li, D., Savarese, S., & Hoi, S. (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning (pp. 19730-19742). PMLR.

[2] Cha, J., Kang, W., Mun, J., & Roh, B. (2024). Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13817-13827).

Q5: Averaging embeddings of tokenized words seems noisy, since the word parts likely carry very different semantic meaning compared to the whole. It would be interesting to understand the effect on the align loss and multi-semantic alignment when it comes to word parts vs whole words.

A5: Thank you for pointing this out. Ideally, we would prefer that each word is tokenized into a single unit by the LLM tokenizer. However, in practice, many words—especially longer or less frequent ones—are split into multiple sub-tokens. Let a visual patch embedding be $\mathbf{v}$ and a word w be decomposed by the LLM’s tokenizer into $n$ sub‑tokens with embeddings $[ \mathbf{w}\_{1}, \mathbf{w}\_{2}, … \mathbf{w}\_{n} ]$. In that case, we aim to maximize total similarity between the patch and all sub‑tokens: $\max \sum_{i = 1}^{n} \left <\mathbf{v}, \mathbf{w}\_i \right>$, which is equivalent to $\max \left <\mathbf{v}, \frac{1}{n}\sum_{i} \mathbf{w}\_i \right >$, i.e., the alignment between the vision token and the average embedding of sub-tokens.

Empirically, Trager et al. [3] show that vision-language models exhibit approximately linear semantic structure, where meanings of complex concepts can be composed by summing component representations. This also supports our choice of averaging sub-token embeddings. That being said, as briefly discussed in the conclusion, we acknowledge that averaging embeddings over sub-tokens might not be the optimal choice and could potentially dilute semantic meaning. If the reviewer has alternative suggestions, we’d be happy to test and incorporate them.

[3] Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., & Soatto, S. (2023). Linear spaces of meanings: compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15395-15404).

Q6: The choice of measuring Von Neumann entropy is deliberate, and Line 141-142 does not clarify why this particular metric was chosen, as opposed to Shannon's entropy, to help quantify the degree of "compression" in visual tokens after projection.

A6: Thank you for the notes. As reviewer suggested, we will provide stronger motivation for our choice of Von Neumann entropy. Von Neumann entropy is calculated from the eigenvalues of the normalized covariance matrix and remains invariant under orthogonal or unitary transformations. This means it captures the intrinsic "spread" of information regardless of the coordinate system used. In contrast, Shannon entropy is defined for discrete distributions or requires discretization in continuous cases. It lacks basis independence—changing the coordinate axes or binning scheme can significantly alter the measured value, creating unwanted variability when comparing embeddings before and after projection. We will clarify this in the final revision.

We sincerely thank the reviewer for recognizing the novelty of our work, the effectiveness of our proposed patch-aligned training, and its impact on both alignment and downstream MLLM performance. Below, we respond to the remaining concerns individually and clarify the corresponding points.

Q1: The idea of enforcing token-level alignment between visual features and text embeddings is quite similar to SEA, even though the implantation is different.

A1: We thank the reviewer for pointing out the related work. While SEA shares the motivation of improving alignment between visual tokens and language embeddings beyond caption-level supervision, our work differs in several significant ways:

1. Fine-grained alignment measurement: SEA focuses primarily on introducing a new training loss, whereas our work provides a systematic analysis of existing projectors using novel metrics, including Von Neumann Entropy (quantifying information compression) and $\mathrm{Align}(\mathbf{V}, \mathbf{W})$ (measuring fine-grained alignment). These analyses reveal the structural limitations of current projectors and provide actionable insights.

2. Multi-semantic alignment: SEA treats each patch as corresponding to a single label. In contrast, we introduce a Matching Pursuit-based decomposition to extract multiple aligned tokens per patch, capturing fine-grained multi-token semantics—crucial for complex understanding.

3. Vocabulary coverage: SEA relies on a fixed, predefined vocabulary, which inherently limits semantic granularity. We use RAM to recognize a broad and dynamic vocabulary of visual concepts, allowing our patch supervision to adapt to diverse real-world entities and contexts.

4. Loss formulation and performance: While SEA adopts a contrastive loss with handcrafted negative sampling, we use a simpler cosine loss that eliminates the need for sampling strategies. Since SEA doesn’t release their code, we implement the contrastive loss on our own. Our ablation studies show that our approach outperforms contrastive loss in downstream tasks: | Dataset | Original LLaVA | SEA (contrastive loss) | Ours (cos loss) | |-------------|----------------|-------------------------|------------------| | GQA | 61.93 | 60.95 | 62.99 | | Science QA | 66.8 | 67.58 | 68.67 | | OKVQA | 53.42 | 51.59 | 58.29 |

We will incorporate these key distinctions and comparisons with SEA in the revision.

Q2: Since the patch-aligned training uses additional labeled data, the performance gains after SFT—especially on fine-grained recognition tasks—are somewhat expected. A deeper discussion of this limitation is needed.

A2: Thank you for your thoughtful comment. We agree with the reviewer that our method introduces additional supervision through bounding boxes. However, the core contribution of our work isn't adding strong supervision, but rather designing a lightweight mechanism to improve fine-grained visual–language alignment in multimodal large language models (MLLMs).

Our performance gains don't come from new module design or additional training data, but from addressing the misalignment in current MLLMs trained solely with caption-level objectives. Compared to the baseline LLaVA model, our method:

• maintains exactly the same architecture,
• uses the same pretraining dataset, with labels generated by our automatic detection pipeline (no additional datasets or human-annotated labels),
• adds only a minimal patch-level cosine loss to guide alignment.

This demonstrates that our approach doesn't rely on rich supervisory information, but rather serves as a method that directly addresses a fundamental alignment gap in the existing training pipeline.

Importantly, the supervision signal is automatic, scalable, and practical. It requires no human annotation. Our pipeline generates annotations directly from existing image data. Using standard hardware (A5000 GPUs), the pipeline annotates 558K samples in ~13 hours, demonstrating real-world scalability.

As suggested, we will incorporate the above discussion in the revision.

We sincerely thank the reviewer for the feedback. Below, we respond to each of the main concerns individually and clarify the corresponding points.

Q1: Grounding DINO uses NMS and confidence-score thresholding to remove overlapping or low-quality boxes, but the quality of this filtering and the effectiveness of the automatic annotations have not been evaluated.

A1: Thank you for your suggestion. The Mask-Label Annotation Pipeline serves primarily as an automated method to quickly obtain bounding box and segmentation information from the original LLaVA pretraining dataset to support our patch-aligned training. Since our main contributions are the analysis of multimodal projector and the patch-aligned training method, due to the limited space, we omitted the presentation of evaluation of the Mask-Label Annotation Pipeline in the main paper. We will now present the details of our evaluation.

To optimize our automatic annotation pipeline, we conducted a thorough ablation study using the coco_val_2017 dataset, which provides ground truth bounding boxes. We focused on two key hyperparameters:

1. Score threshold: Only boxes with confidence scores above this threshold are selected.
2. NMS threshold: During non-maximum suppression (NMS), this determines the maximum allowed overlap between two boxes—if their IoU exceeds this threshold, the box with the lower confidence score is removed.

We evaluated performance using F1 score at IoU of 0.5, which classifies predictions as true or false positives based on an IoU threshold of 0.5. First, we tested various score threshold values:

Next, we fixed the score threshold at 0.4 and evaluated various NMS threshold values:

Based on this analysis, we selected the optimal hyperparameters (Score threshold = 0.4 and NMS threshold = 0.8) for our final implementation. Following the reviewer's suggestion, in the revision, we will include this evaluation in the appendix.

Q2: There is a similar work SEA: Supervised Embedding Alignment, which aligns each visual token to the LLM word embedding to solve the problem that caption-loss can only bring coarse-grained semantics.

A2: We thank the reviewer for pointing out the related work. While SEA shares the motivation of improving alignment between visual tokens and language embeddings beyond caption-level supervision, our work differs in several significant ways:

1. Fine-grained alignment measurement: SEA focuses primarily on introducing a new training loss, whereas our work provides a systematic analysis of existing projectors using novel metrics, including Von Neumann Entropy (quantifying information compression) and $\mathrm{Align}(\mathbf{V}, \mathbf{W})$ (measuring fine-grained alignment). These analyses reveal the structural limitations of current projectors and provide actionable insights.

2. Multi-semantic alignment: SEA treats each patch as corresponding to a single label. In contrast, we introduce a Matching Pursuit-based decomposition to extract multiple aligned tokens per patch, capturing fine-grained multi-token semantics—crucial for complex understanding.

3. Vocabulary coverage: SEA relies on a fixed, predefined vocabulary, which inherently limits semantic granularity. We use RAM to recognize a broad and dynamic vocabulary of visual concepts, allowing our patch supervision to adapt to diverse real-world entities and contexts.

4. Loss formulation and performance: While SEA adopts a contrastive loss with handcrafted negative sampling, we use a simpler cosine loss that eliminates the need for sampling strategies. Since SEA doesn’t release their code, we implement the contrastive loss on our own. Our ablation studies show that our approach outperforms contrastive loss in downstream tasks: | Dataset | Original LLaVA | SEA (contrastive loss) | Ours (cos loss) | |-------------|----------------|-------------------------|------------------| | GQA | 61.93 | 60.95 | 62.99 | | Science QA | 66.8 | 67.58 | 68.67 | | OKVQA | 53.42 | 51.59 | 58.29 |

We will incorporate these key distinctions and comparisons with SEA in the revision.

Q3: Could you provide evaluation results comparing with manual annotations in terms of IoU/mAP, so as to quantify the true impact of false positives and false negatives on training?

A3: Thank you for the question. Directly creating manual annotations for the entire LAION dataset would require an impractical amount of human effort and time, so we are unable to provide a direct comparison with fully manual annotations. However, we used the coco-val 2017 dataset, which has high-quality annotated bounding boxes, to evaluate our pipeline with optimal hyper-parameters and compare it with the original Grounding DINO. The results are as follows:

AP@[IoU=0.50:0.95] is the mean precision across IoU thresholds from 0.50 to 0.95 (stepped by 0.05). A prediction is considered correct if its overlap with ground-truth exceeds the IoU threshold. As the results demonstrate, our pipeline consistently outperforms the baseline. Following the reviewer's suggestion, we will include this evaluation in the appendix of the revision.

Q4: In the RefCOCO and instruction-following sections, is there a side-by-side comparison with models of the same parameter size?

A4: Thank you for your question. In both Table 5 (referring expression comprehension) and Table 7 (instruction-following), our comparisons between Patch-aligned LLaVA (7B) and LLaVA-1.5-7B are conducted using exactly the same models (the same architecture and parameter size (7B)), ensuring a fair evaluation. We additionally compare with LION (Dual-Level vIsual knOwledge eNhanced Multimodal Large Language Model) [1] because it reports zero-shot performance on RefCOCO—without training on RefCOCO—consistent with our evaluation setting. In contrast, many other models (e.g., QwenVL) only report results after fine-tuning on RefCOCO, which leads to higher scores. Although LION uses a larger 12B model, as shown in the table, our 7B model still achieves higher average performance on referring expression grounding tasks, highlighting the effectiveness of our method.

[1] Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024). Lion: Empowering multimodal large language model with dual-level visual knowledge.

Q5: Von Neumann Entropy may indicate better alignment, but is there a ΔH versus task performance curve that reveals the tipping point where over-compression causes semantic information loss?

A5: Thanks for the question. Yes, there exists a tipping point in $\Delta H$ versus task performance curve. As shown in the table, as $\Delta H$ increases, the overall performance first improves then declines. Before the tipping point, redundant information is removed, which improves performance compared to the original LLaVA with $\beta = 0$. However, after the tipping point, performance drops rapidly as over-compression causes semantic information loss.

Here, the change of entropy is measured as normalized one: $\Delta H(V) = \frac{H_{before}(V) - H_{after}(V)}{H_{before}(V)} \in [0,1]$. Task performance is measured by taking the average of the performance over the QA datasets (GQA, Science QA, VizWiz VQA and OKVQA). The compression level is controlled by the weight of the patch-aligned loss in the aligned training: $L = L_{caption} + \beta L_{patch}$, where larger $\beta$ leads to larger $\Delta H$, indicating more compression.

Due to space limitations, we omitted this ablation study in the current version. We will include it in the revised version, either in the appendix or in the main paper, depending on the available space.

Dear Reviewer ZSmz,

Please take a moment to carefully read the authors’ rebuttal and add your comments.

Thank you for your time and contribution.

We sincerely thank the reviewer for the feedback. Below, we respond to each of the main concerns individually and clarify the corresponding points.

Q1: The proposed method simply adds a patch loss, which is computed based on automatically generated patch-text pairs. The technical contribution is quite limited.

A1: Thank you for your comments. We appreciate the opportunity to clarify our contributions. We want to first note that the patch loss only serves as one component of our contributions. In this work, we develop a comprehensive framework for both analyzing and improving fine-grained vision-text alignment in MLLMs. We highlight our key contributions as follows: (1) examining the projector's role in compressing vision embeddings and aligning them with word embeddings from an entropy perspective, (2) proposing metrics that quantitatively reveal how projectors trained only with caption loss (the standard approach) achieve limited patch-level alignment, resulting in weak and coarse-grained alignment, (3) using matching pursuit to find the proper word embeddings to preserve multiple semantic meanings, and (4) finally proposing a patch-alignment method to enhance patch-level alignment.

With respect to the proposed patch loss, as mentioned by Reviewer zak5, it offers several key advantages: (1) Efficient and simplicity. The cosine similarity between visual tokens and LLM’s embedding tokens is computationally efficient. Since the patch-level loss is independent only requires the LLM's embedding matrix and operates independently of the LLM's forward pass, the training time of PatchAligned LLaVA remains comparable to the original LLaVA. (2) Effectiveness. Despite being a straightforward modification, the proposed align loss significantly improves the projector's fine-grained visual-text token alignment, yielding substantial improvements: $16$% on referring expression grounding tasks, $4$% on question-answering tasks, and $3$% on modern instruction-following benchmarks. (3) Adaptable to any MLLM framework. Our approach requires no specialized architecture and can be easily integrated into various MLLMs to enhance fine-grained alignment between different LLM and vision encoders.

Q2: The findings in Sec 3 reveals that existing projectors show limited multi-semantic alignment and patch-level alignment. Those findings, however are not quite innovative and can be inferred from the performance of MLLMs. E.g., the poor localization capability indicates the poor patch-level alignment.

A2: We respectfully disagree that these findings lack novelty. To the best of our knowledge, no prior work has formally studied patch-level alignment and multi-semantic alignment. If the reviewer is aware of prior work that directly quantifies these alignments, we would be happy to cite it. While poor localization performance of MLLMs may hint at weak patch-level alignment, it can also arise from several other factors, such as limitations of the vision backbone or the language model’s ability to interpret visual tokens. Without a formal metric and analysis, these factors cannot be disentangled. To bridge this gap, Section 3 introduces a quantitative metric ($\mathrm{Align}(\mathbf{V}, \mathbf{W})$) for patch-level alignment and proposes an algorithm to analyze multi-semantic alignment. Using our $\mathrm{Align}(\mathbf{V}, \mathbf{W})$ metric, we show that the original LLaVA projector exhibits weak alignment, improving $\mathrm{Align}(\mathbf{V}, \mathbf{W})$ only from $0.065$ to $0.142$ compared to a random projector. Our proposed patch-align training further improves $\mathrm{Align}(\mathbf{V}, \mathbf{W})$ to $0.279$, leading to a $16$% gain on referring expression grounding tasks. We will incorporate this point at the beginning of section 3.2.2. in the revision.

Q3: The presentation should be improved. Many symbols are used with different meaning, e.g., V_before, and V_after show different dimensionalities in Sec. 3.1.

A3: The $\mathbf{V}\_{\mathrm{before}}$ and $\mathbf{V}\_{\mathrm{after}}$ could have different dimensionality. As explained in Eq. (1), $\mathbf{V}\_{\mathrm{before}}$ and $\mathbf{V}\_{\mathrm{after}}$ refer to features before and after the projector, in MLLMs such as LLaVa which have a structure Vision Encoder $\rightarrow$ Projector $\rightarrow$ LLM. For example, the vision encoder CLIP-ViT-Large-Patch14-336 has an embedding size of $1024$ (the dimension of $\mathbf{V}\_{\mathrm{before}}$), whereas Vicuna 7B has an embedding size of $4096$ (the dimension of $\mathbf{V}\_{\mathrm{after}}$). We will add a clarifying comment after Eq. (1) to make this point explicit. If the reviewer has noticed any other issues with the notation, we would be happy to address them.

Dear Reviewer UhGf,

Please take a moment to carefully read the authors’ rebuttal and add your comments.

Thank you for your time and contribution.