Thank you for your constructive feedback and acknowledging the robustness to noise of our Align connector. Below we address your concerns.

Concern 1: Scope and Additional Results on General Vision-Language Tasks

Document understanding is challenging due to its inherent multimodal nature (integrating complex visual and textual information) and has numerous critical real-world industry applications. Thus, it serves as an ideal domain for evaluating our proposed AlignVLM approach. Our Align Connector maps visual features into convex combinations of the LLM’s text embeddings which are regions of the latent space that the LLM is inherently trained to interpret as text. As shown in Tables 2 and 3, the Align connector yields substantially larger improvements over existing connectors (e.g., MLP) on document benchmarks (Table 2) compared to general vision-language tasks (Table 3), supporting its design motivation.

We agree with the reviewer that testing additional baselines on general vision-language tasks would further validate the generalizability of our method. To that end, we conducted ablation experiments using SigLIP-400M and Llama-3.2-3B with different connector designs, all trained on the LLaVA-Next dataset [1]. These models were trained under identical configurations: for pretraining, we used a learning rate of 1e-3, batch size of 64, and 1 epoch; for instruction tuning, we set the LLM and connector learning rate to 1e-5, vision encoder learning rate to 2e-6, batch size of 8, and 1 epoch. We evaluated the resulting models on standard general vision-language benchmarks including MMMU-dev, SeedBench, MMVet, POPE, and GQA. The results, summarized in the table below, demonstrate that the ALIGN connector outperforms all other connector variants on these general tasks on average, proving its effectiveness beyond the document domain.

Concern 2: Comparison with HReducer and HoneyBee

We would like to clarify that all models in the lower half of Table 1 (including mPLUG-DocOwl) were instruction-tuned on the same datasets (BigDocs-7.5M and DocDownstream). Table 1 is intended to provide a comparison between AlignVLM and state-of-the-art models at the full model level, under a consistent instruction-tuning regime.

We also agree with the reviewer that a connector-level comparison with HReducer from mPLUG-DocOwl and HoneyBee (C-Abs) is necessary for a fair and direct architectural evaluation against our proposed connector (Align), similar to our comparisons with MLP, Perceiver Resampler, and Ovis in Table 2.

To ensure a fair comparison under a compute-efficient yet controlled setup, we reused the same experimental configuration described in Concern 1 using SigLIP-400M and Llama-3.2-3B trained on the LLaVA-Next dataset with identical pretraining and instruction tuning settings. We evaluated all connectors, including HReducer and HoneyBee (C-Abs), on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, SeedBench, MMVet, POPE, GQA). As shown in the tables above (general vision tasks) and below (documents tasks), the Align connector consistently outperforms these strong baselines, highlighting its robustness across domains. Notably, HoneyBee is the second-best connector, likely due to its ability to preserve local visual context. We attribute ALIGN's superior performance to its unique design, which projects visual features into the convex hull of the LLM's text embeddings (a region in the latent space that the LLM is inherently pretrained to understand)

Concern 3: Additional Analysis of the Align Connector Internal Mechanism

We conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the llm’s text embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.

To further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that ALIGN effectively leverages these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively understand. It also shows that the Align connector can benefit from further pruning to improve its efficiency significantly.

Unfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript to clarify the inner workings of the Align connector.

Thank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to consider raising your score to support the dissemination of our work within the ML community.

[1] Llava-NeXT: Improved reasoning, OCR, and world knowledge

Thank you for acknowledging the effectiveness of the align connector and its robustness to noise. We address your concerns and questions below.

Concern 1: Experimentation with other LLMs

We understand your interest in evaluating the Align Connector using other LLMs. However, due to computational constraints, we primarily experimented with the open-source SOTA Llama 3 models with different sizes (1B, 3B, and 8B parameters) to highlight Align’s scalability and consistent performance. Nonetheless, our Align module is LLM-agnostic and applicable in principle to any LLM. We have launched our training experiment with Qwen2.5-3B and will report the results in the camera-ready revised manuscript once it’s completed.

Question 1: Do we keep the embedding layers frozen during training?

No, we train all weights of the Align connector in all stages, as detailed in Table 5 of the appendix. In initial experiments, freezing the LM head and the LLM embedding matrix in the Align connector resulted in unstable training. This is primarily due to the distribution shift in the LM head layer's inputs (in the Align Connector), from the previously encoded textual outputs of the LLM to the current visual features from our vision encoder.

Question 2: Runtime comparison between Connectors

We acknowledge that the Align connector includes an additional LM head layer, which slightly increases the parameter count. However, this addition has a negligible impact on runtime efficiency due to its straightforward design, involving simple matrix multiplication operations (as shown in Equations 1 and 2 in the paper), rather than stacking multiple complex layers requiring sequential processing like the deep fusion methods.

To empirically support this, we benchmarked the runtime and memory usage of models using different connectors (MLP, Align, Ovis, and Perceiver), corresponding to the experiments reported in Table 2 of our paper. The results below show minimal differences in inference speed and GPU memory usage between the connectors, despite the Align connector's notably superior performance as demonstrated previously in Table 2.

Thank you for your thoughtful review. We hope our response has addressed your concerns.

Thank you for your constructive feedback and acknowledging the novelty of our Align connector and its robustness to noise . We address your comments below.

Concern 1: Low-resource regime training

We agree that experimenting with a low-data regime can further demonstrate ALIGN's efficiency. To address this, we conducted additional experiments using SigLIP-400M and Llama3.2-3B under a low-data regime by finetuning on the LLava-Next [1] dataset (779K data points). Specifically, we followed the LLava-Next recommended training configurations for both pretraining (freezing the llm and vision encoder, learning rate: 1e-3, batch size: 64, epochs: 1) and instruction tuning (LLM and connector learning rate: 1e-5, vision encoder: 2e-6, batch size: 8, epochs: 1).

We evaluated models on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, SeedBench, MMVet, POPE, GQA). Our findings indicate that the Align connector significantly outperforms other connectors under the low-data regime, especially on document understanding tasks. This makes the Align connector very valuable in resource-constrained environments such as Academic labs.

Document Understanding

General VisionTasks

Concern 2: Runtime and Memory Comparisons with Deep Fusion Methods

We compared Align against the deep fusion mechanism used in LLaMA-3.2 Vision [2]. We use the same backbone vision encoder and llm (SigLIP-400M, LLaMA-3.2-8B) to isolate the impact of the connector itself on inference runtime and memory usage. As shown in the table below, our Align connector reduces runtime and memory consumption, confirming our claim regarding its computational efficiency compared to deep fusion methods. We will include these details in the revised manuscript.

Concern 3: Visual Semantic Leakage and Mappings

To further understand the mapping process for Align, we conducted a deeper analysis of the token distributions described in Section 5.3. We observed that ALIGN does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted all the llm text embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.

To further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages and combines these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively understand and interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency significantly.

Unfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript to clarify the inner workings of the Align connector.

Concern 4: Comparison with LayoutLMv3 and Pix2Struct.

We believe that LayoutLMv3 and Pix2Struct are not directly comparable to our models due to fundamental architectural differences. LayoutLMv3 is an encoder-only model (similar to BERT), while Pix2Struct follows an encoder-decoder design. In contrast, AlignVLM is built on a vision-language architecture with an LLM decoder.

Moreover, both LayoutLMv3 and Pix2Struct are relatively small in scale (typically 100M–300M parameters), whereas our models range from 1B to 8B parameters, making direct comparisons less meaningful and potentially misleading.

We understand the importance of comparing AlignVLM with strong document-specific baselines. For this reason, Table 1 includes results for DocOwl-1.5, a recent and powerful document understanding model with a similar architecture and size. This offers a more appropriate and fair comparison. As shown in Table 1, AlignVLM consistently outperforms DocOwl1.5 across various benchmarks when instruction tuned on the same datasets, highlighting its effectiveness in the document domain.

Concern 5: Differences between BigDocs and Proprietary datasets

Detailed datasets used by models like Qwen2-VL and InternVL remain undisclosed, making a completely unbiased comparison difficult. However, our deliberate selection of the BigDocs-7.5M and DocDownstream datasets was guided by their focus on document understanding which is the main focus and scope of our paper.

Furthermore, open models (qwen2-vl, Intern-vl2) are anticipated to have been trained on significantly larger datasets (trillions of tokens and billions of images) and benefit from far more extensive GPU resources. We expect that this already places AlignVLM at a disadvantage, as larger-scale training typically leads to stronger generalization and performance improvements. Despite these constraints, AlignVLM remains competitive within the scope of multimodal document understanding. Our paper focuses explicitly on architectural improvements rather than extensive dataset curation.

Suggestion on Adding an analysis section on non-document tasks.

Our paper already includes results on several widely used general vision-language benchmarks (MMMU, MMVet, POPE, GQA, and SeedBench) as presented in Section 5.4, demonstrating the generalizability of the Align connector beyond document understanding.

We also agree with the reviewer that evaluating Align on additional diverse domains, such as fashion, scene understanding, or scene graph generation, would further strengthen the paper’s contribution. We appreciate this suggestion and will incorporate such analyses in the revised manuscript.

Thank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to consider raising your score to support the dissemination of our work within the ML community.

[1] Llava-Next: Improved reasoning, OCR, and world knowledge [2] The Llama 3 Herd of Models

Thank you for your thoughtful comments and acknowledging the novelty of our Align idea. We address your comments below.

Concern 1: Design of Alignment Method

The initial linear projection layer in Align is essential to match the dimensionality of the visual features with the LLM’s embedding space. This projection is effectively learned during training. The core innovation of the Align connector lies in the subsequent step, where visual features are mapped to a convex combination of the LLM’s text embeddings. This ensures that all visual inputs remain strictly within the convex hull of the LLM’s embedding latent space. This design naturally reduces the risk of noisy or out-of-distribution (OOD) representations. This is also supported by empirical results in Section 5.5; When Gaussian noise was added to the visual features, the Align connector exhibited only a 1.67% drop in performance, in contrast to a 25.54% drop observed with an MLP connector, highlighting the regularization benefits of our approach.

Concern 2: Align Connector Representation Learning

We conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.

To further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages and combines these select embeddings to guide visual features into meaningful semantic regions within the LLM's latent text space that the LLM can effectively interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency.

Unfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings/figures and revise our manuscript to reflect them. .

Concern 3: Slightly Lower Performance Compared to SOTA Models like Qwen2.5-VL-Instruct

Detailed datasets used by models like Qwen2-VL and InternVL remain undisclosed, making a completely unbiased comparison difficult. However, our deliberate selection of the BigDocs-7.5M and DocDownstream datasets was guided by their focus on document understanding which is the main focus and scope of our paper.

Furthermore, open models (qwen2-vl, Intern-vl2) are anticipated to have been trained on significantly larger datasets (trillions of tokens and billions of images) and benefit from far more extensive GPU resources. We expect that this already places AlignVLM at a disadvantage, as larger-scale training typically leads to stronger generalization and performance improvements. Despite these constraints, AlignVLM remains competitive within the scope of multimodal document understanding. Our paper focuses explicitly on architectural improvements rather than extensive dataset curation.

To further justify our dataset choice and evaluate Align’s performance under low-data regime, we experimented with an alternative popular dataset, LLava-NeXT [1] which consist of 779K data points. We observed two critical findings: (1) Models trained on BigDocs-7.5M significantly outperform those trained on LLava-NeXT, and (2) Align significantly outperforms the MLP connector in a low-data regime, underscoring the strong inductive bias added through leveraging the LLM’s text embeddings for representing visual features. This makes the Align connector very valuable in resource-constrained environments such as Academic labs.

Concern 4: Performance of Comparable Llama Vision Models in Table 1

Official Llama Vision models are limited to two versions only: Llama-3.2-90B (outside our scope) and Llama-3.2-11B which is reported in our manuscript (Table 1). Notably, the Llama-3.2-11B uses the exact LLM backbone (Llama-3.1-8B) as our AlignVLM-8B. The increased parameters from 8B to 11B in Llama-3.2-11B are primarily due to deep fusion connectors (additional cross-attention/feed-forward layers). The results in Table 1 shows that our AlignVLM-8B model achieves superior performance despite using a smaller connector, clearly demonstrating its efficiency and effectiveness compared to deep fusion methods.

Thank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to consider raising your score to support the dissemination of our work within the ML community.

[1] Llava-Next: Improved reasoning, OCR, and world knowledge

Thank you for your response. We're glad our rebuttal has addressed some of your concerns. Below, we address your remaining point regarding Figure 3 and 4.

We agree that Figure 4, in its current form, does not provide sufficient evidence to fully support our claim in lines 278–282 that the Align connector helps the model map visual features into semantically meaningful embeddings. Due to the very high-density nature of the token distribution (as shown in Figure 3, where the top token receives only a 0.0118 probability out of 128K tokens), inspecting only the top tokens in Equation 1 is not very informative. Since the final visual embedding is computed as a convex combination (weighted sum) over all 128K token embeddings (Equation 2), the resulting vector can differ significantly from any individual top token. This makes interpreting the one-to-one token mappings in Equation 1 potentially misleading (Figure 4).

Instead, we believe the strength of AlignVLM lies in constraining all visual features to lie within the convex hull of the LLM’s text embeddings, ensuring that the visual inputs to the LLM remain “in-distribution” similar to its original pretraining data inputs (text). Other connectors like MLP do not impose such constraints and may generate out-of-distribution representations for this reason. This design of the Align connector significantly eases the learning process, especially in low-data regimes (llava-next dataset), as supported by our earlier results in the rebuttal and Table 2 in the paper. It also makes the connector more robust to noise as shown in Section 5.5.

We will revise our claims in lines 278–282 to reflect these new analyses (Previous Response to Concern 2) instead which provide better insights into the internal mechanism of the Align connector.

We will also replace Figure 4 with the PCA visualization of the 3.4K tokens over the LLM vocab we discussed in the previous rebuttal response results (Concern 2).

Please let us know if you have any additional questions or feedback. We sincerely appreciate your constructive suggestions, which have improved the presentation of our work. If our responses have addressed your critical concerns, we kindly ask you to consider revising your score to reflect your final evaluation of our work.

Thank you for your thoughtful comments and acknowledging that our Align idea is elegant and grounded in strong inductive bias. We address your comments below.

Concern 1: Experimenting with Other Vision Encoders

In the paper, we focused on SigLip-400m due to its superior performance over CLIP [1]. We agree that experimenting with various vision encoders can further demonstrate ALIGN's generalizability. To address this, we conducted additional experiments using SigLIP-400M and CLIP vision encoders under a low-data regime setup for computational efficiency. Specifically, we finetune on the LLava-Next dataaset [2] and follow their recommended training configurations for both pretraining (freezing llm and vision encoder, learning rate: 1e-3, batch size: 64, epochs: 1) and instruction tuning (LLM and connector learning rate: 1e-5, vision encoder: 2e-6, batch size: 8, epochs: 1).

We evaluated the models on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, MMVet, GQA). Our findings indicate that Align still produces strong and comparable performance despite using a weaker vision backbone. This proves that the Align connector generalizes well across different vision encoders.

Concern 2: Comparisons with Lightweight encoder-free VLMs (Fuyu-8B)

We evaluated Fuyu-8B on the same document, understanding benchmarks and observed significantly limited performance compared to AlignVLM as shown in the table below. Although encoder-free models simplify architecture, they typically lack performance due to not leveraging pretrained visual components [3]. We will include these results in Table 1 and revise the discussions in our revised manuscript.

Concern 3: Runtime and Memory Comparisons with Deep Fusion Methods

We compared Align against the deep fusion mechanism used in LLaMA-3.2 Vision [4]. We use the same backbone vision encoder and llm (SigLIP-400M, LLaMA-3.2-8B) to isolate the impact of the connector itself on inference runtime and memory usage. As shown in the table below, our Align connector reduces runtime and memory consumption, confirming our claim regarding its computational efficiency compared to deep fusion methods. We will include these details in the revised manuscript.

Concern 4: Further Analysis of the Mapping of Visual Features over Embeddings.

We conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.

To further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency.

Unfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript.

Thank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to consider raising your score to support the dissemination of our work within the ML community.

[1] Sigmoid Loss for Language Image Pre-Training (arxiv) [2] Llava-NeXT: Improved reasoning, OCR, and world knowledge [3] PaliGemma: A versatile 3B VLM for transfer [4] The Llama 3 Herd of Models