DS-VLM: Diffusion Supervision Vision Language Model

However, the default input space of the Text Adapter (CLIP features) and the LLM backbone (word embeddings) are different. Why are the newly learned information assumed to be able to leveraged by the LLM backbone?

A1: In response to the reviewer’s question, our DS-VLM framework leverages a diffusion model to supervise both the connector outputs and multi-level visual encoder features, effectively shortening the long knowledge propagation chain often seen in MLLM training.

As for the difference between the Text Adapter’s input (CLIP features) and the LLM’s backbone (word embeddings), our key view is that the connector’s goal is to semantically align visual features to serve as effective textual embeddings. The diffusion model provides strong supervision and helps align feature distributions across modalities, allowing the connector to produce features that carry both visual detail and semantic meaning. These features, once projected into the LLM’s embedding space, can be naturally understood and used by the language model.

Our experiments show that DS-VLM significantly improves multimodal task performance. We will further clarify this mechanism’s rationale and contributions in the final version.

The original LLaVA framework freezes the vision encoder during training. However, this work fine-tunes the vision encoder. It is unclear whether the so-called "LLaVA" in Table 1 freezes the vision encoder or not. If yes, comparing the proposed method to LLaVA with a trainable vision encoder is necessary to verify the effectiveness.

A2: Thank you for your suggestion. In Table 1, the LLaVA baseline freezes the visual encoder during training. Following your advice, we have conducted additional experiments comparing DS-VLM with a version of LLaVA where the visual encoder is trainable. As shown in the Table 1 below, DS-VLM still demonstrates clear advantages even when compared with LLaVA using a trainable visual encoder.

Table1: Comparative experiments with LLaVA with trainable encoder

It would be appreciated if the paper can analyze qualitative results of the model and summarize on what types of questions the proposed method improves the performance. This might be helpful to support that the diffusion module enhances the learned semantic visual information.

A3: We agree that analyzing performance across question types helps validate the diffusion module’s impact. DS-VLM shows strong results on tasks requiring structural understanding and fine-grained perception, with notable gains on datasets like MMMU, MMB, and MMS. Specifically, MMMU covers multi-disciplinary visual reasoning (e.g., medicine and engineering), requiring accurate interpretation of complex visuals and domain-specific symbols—areas where DS-VLM excels due to its enhanced visual representations learned via pixel-level reconstruction. MMB evaluates broad multimodal capabilities, including structured diagrams and spatial layouts, where DS-VLM’s multi-layer supervision improves detail sensitivity. MMS further emphasizes semantic alignment between modalities, which benefits from our diffusion-enhanced visual features. We will include qualitative examples and analyses in the final version to better demonstrate DS-VLM’s effectiveness across these question types.

Additional slashes in Line 242-243.

A4: Thank you for pointing this out. We will remove the unnecessary slashes in lines 242–243 in the final version to avoid confusion.

Figure 3 shows that the LLM backbone is finetuned with LoRA. It would be better to also mention it in the Implementation Details in Section 4.1.

A5: Thank you for pointing out this issue. We will include the relevant information in the final version to ensure consistency between the text and figures, thereby improving the completeness and readability of the paper.

In Line 302-303, Section 4.1. The MLLM frameworks that use SigLIP as the vision encoder should be cited.

A6: We will include citations to MLLM frameworks that use SigLIP as the vision encoder in the final version to improve the discussion of related work.

It would be better to also include the performance of LLaVA in Table 2.

A7: Thank you for your suggestion. We will include the performance of LLaVA in Table 2 of the final version to enable a more comprehensive comparative analysis.

Thanks for your feedback. We’ll address each point in detail.

The paper reports good results across various test sets. Section 4.3 shows accuracy gains from different supervision features and Section 4.5 show typical reconstruction process for diffusion models. However, the connection between proposed method and performance improvements are still not clear, and more discussions are recommended.

This paper shows result improvement on multiple benchmarks. However, only the numbers are discussed in Section 4.2. More discussions on the improvement of model ability are recommended.

A1: Thank you for pointing this out. The performance gains come from our diffusion-based pixel-level supervision, which creates a short gradient propagation chain from image pixels to visual features, greatly enhancing the visual encoder and connector’s representation capabilities.

Our method incorporates multi-level visual feature supervision (high, middle, and low layers) along with an MOE-based fusion mechanism, allowing the model to comprehensively perceive semantic details and structural information within the image—effectively compensating for the semantic sparsity of traditional text-only supervision in VLMs. Furthermore, the visualization results in Section 4.5 clearly demonstrate how DS-VLM progressively reconstructs the original image, providing intuitive evidence of the effectiveness of supervised features in guiding the visual understanding process.

The 1.9% gain on MMMU highlights our method’s strength in structured image understanding and semantic reasoning. We’ll clarify this causal link in the final version.

More ablation studies for different modules are recommended, such as the effect of adapters, the layer selection, cross-attentions.

A2: As shown in Table 3 (No.2–4) of the paper, we explored layer choices in our ablation study. Joint supervision across visual layers significantly boosts performance, as the layers complement each other and enhance supervision quality, improving model understanding and generalization.

Following your suggestion, we conducted ablation studies on Adapter types (Table 1 below) and Cross Attention mechanisms (Table 2 below). Our proposed Adapter and decoupled Cross Attention show clear performance advantages. The multi-adapter structure enables finer adjustment of features from different modalities compared to the Q-Former-style Adapter. The decoupled Cross Attention avoids interference across feature levels, enabling each to fully express its semantic information through its dedicated attention path. These results validate the strength of our architectural design.

Talbe1: Adapter type comparison

Table2: Comparison using shared Cross-Attention mechanism

Visualizations of model outputs are recommended to clearly show the difference between proposed method and baseline.

A3: We agree that visual comparisons are important for validating our method. In a fine-grained description task (e.g., “Describe the accessories worn”), the baseline missed the earring detail, while DS-VLM correctly generated “a silver hoop earring on the left ear.” This shows the benefit of our diffusion-based supervision in capturing fine details. We will add visual examples to highlight DS-VLM’s strength in fine-grained understanding.

Line 324 mentions the Mini-Gemini dataset, where is the result?

A4: The Mini-Gemini dataset is included in the "Expanding to larger training datasets" section of Table 1 in the paper. We'll clarify this in the final version.

Does the proposed method achives comparable results if trained on partial training data (such as 2/3) of LLaVa?

A5: As shown in Table 3 below, reducing training data leads to performance drop, highlighting the importance of full data for effective diffusion-based supervision. This analysis will be added in the final version.

Table3: Comparison using partial training data

The suggestions for additional analyses (e.g., statistical significance, computational overhead) would further strengthen the paper and provide a better evaluation of the proposed method.

The training process involving diffusion models is computationally intensive. The paper does not provide any analysis on the training time and resource requirements, which are critical for practical applications. This lack of information raises concerns about the feasibility of deploying DS-VLM in real-world scenarios.

A1: Thank you for your constructive suggestion. We conducted full training of DS-VLM on 8 H100 GPUs. As shown in Table 1 below, compared to the original LLaVA-1.5, DS-VLM increases training time by approximately 15%. However, during inference, the diffusion module is not involved, so there is no additional inference overhead compared to the baseline. Given the performance gains on multiple benchmarks (e.g., a 2.1% improvement on MMMU and 1.8% on MMS), we believe this moderate training cost is justified. We will include this analysis in the final version to better illustrate the practicality of our method.

Talbe1: Comparison of two-Stage training time consumption under different settings.

The paper provides a good review of related work. However, there are a few related works and concepts that are relevant but not currently cited or discussed in the paper:

A2: Thank you for highlighting the missing references. We appreciate your suggestions and will include and discuss them in the final version.

While the idea of using diffusion models to supervise vision-language models is interesting, it appears to be an incremental extension of existing work rather than a new approach. For instance, the integration of diffusion models in VLM has been extensive explored in previous studies such as DiffTPP, and also been explored in other contexts (e.g., text-to-image generation), and the application to VLMs seems like a straightforward adaptation.

A3: Thank you for your valuable comments. We understand the concern about the novelty of using diffusion to guide VLMs. Our approach is not a direct application of existing methods but an innovative integration of diffusion into VLMs.

Unlike methods such as DiffTPP, which uses diffusion for generation, we first use it as a supervision signal during training. This enables the construction of a short-path gradient propagation chain from the image pixel space to visual features. Such a mechanism effectively mitigates the supervision information loss caused by long gradient paths in current VLM training. At the same time, the image reconstruction task provides fine-grained, multi-level optimization signals for both the visual encoder and the connector.

Furthermore, our proposed Multi-Adapter Diffusion architecture flexibly receives supervision features from different modalities and employs MOE Cross Attention to aggregate information across multiple layers—a design that, to the best of our knowledge, is unprecedented in prior work.

Therefore, we believe this work introduces substantial and original contributions in coupling diffusion models with VLMs, rather than being a generic application of diffusion techniques to VLMs. We will clarify this point further in the final version of the paper.

The core components of DS-VLM, such as the Multi-Adapter Diffusion model and the MOE Cross Attention mechanism, are built on well-established concepts. The paper does not introduce any new mechanisms or architectures that advance the field.

A4: Thank you for your thoughtful review. We sincerely acknowledge that while the Multi-Adapter Diffusion and MOE Cross Attention mechanisms in DS-VLM are inspired by some existing foundational concepts, our method introduces key innovations in how these components are integrated and applied. Specifically, Multi-Adapter Diffusion is the first to leverage a diffusion model for supervising intermediate features during the training phase. It receives features from multiple layers of the visual encoder as well as from the connector, and reconstructs the image through a unified U-Net framework. Meanwhile, the MOE Cross Attention mechanism processes these heterogeneous features through separate attention pathways and dynamically fuses them via expert routing, enabling the injection of fine-grained supervision signals. This design not only establishes a short supervision path from image pixels to multi-level visual features but also significantly enhances the structural coherence and semantic expressiveness of the learned representations. We believe this constitutes a substantial advancement over traditional VLM optimization paradigms. We will further highlight the uniqueness and novelty of these mechanisms in the final version of the paper.

The improvements are marginal ~1-2% and not "substantial" as claimed by the authors. Also for many benchmarks there is no improvement at all in Table 1.

The authors need to analyse this inconsistency in performance a cross the different benchmarks.

A1: Thank you for pointing out the modest improvements and performance inconsistency. We understand your concern and clarify that DS-VLM is not just about fine-tuning gains, but proposes a new optimization paradigm: diffusion-based image reconstruction supervision. It delivers dense semantic signals from pixel space to the encoder and connector, shortening the gradient path in traditional text supervision and mitigating semantic sparsity, thus enhancing structural and semantic representation. This is particularly effective on datasets like MMMU, MMBench, and MMStar. In OCRB tasks requiring high-res text recognition, our reconstruction focuses on structure and semantics, lacking detail for small text. We plan to explore enhanced text-area reconstruction in future work. DS-VLM adds no extra inference cost, ensuring practicality and scalability. We will include this analysis in the final version.

Do the authors roll out the diffusion process during training to backpropagate gradients. This needs to be clarified in the paper.

A2: The gradient backpropagation path in DS-VLM follows: Image Reconstruction Loss → Diffusion → Adapter → Vision Encoder. Below, we derive the backpropagation process of the model. For simplicity, we set the number of diffusion iterations to 2. We begin by defining:

$z_e=E(f;a)$

$z_a=A(z_e;b)=A(E(f;a);b)$

$z_{d1}=D(x,z_a;c)=D(x,A(E(f;a);b);c)$

$z_{d2}=D(z_{d1},z_a;c)=D(D(x,A(E(f;a);b);c),A(E(f;a);b);c)$

$L=L(z_{d2})$

Here, $E$ denotes the encoder with parameters $a$, $A$ denotes the adapter with parameters $b$, and $D$ denotes the diffusion module with parameters $c$. $L$ denotes the reconstruction loss.

According to the chain rule, the gradient of the Adapter parameter can be deduced as:

$\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z_{d2}}[\frac{\partial D(z_{d1},z_a;c)}{\partial z_{d1}}\cdot \frac{\partial D(x,z_a)}{\partial z_a}+\frac{\partial D(z_{d1},z_a)}{\partial z_a}]\cdot \frac{\partial A}{\partial b}$

The gradient of the Encoder parameters is：

$\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z_{d2}}[\frac{\partial D(z_{d1},z_a)}{\partial z_{d1}}\frac{\partial D(x,z_a)}{\partial z_a}+\frac{\partial D(z_{d1},z_a)}{\partial z_a}]\frac{\partial A(z_e;b)}{\partial z_e}\frac{\partial E(f;a)}{\partial a}$

I hope the above can answer your questions. We look forward to further communication.

The authors add a bunch of new layers to feed the multi-level features into the diffusion model. I assume these layers are not used during inference since their only purpose is to adapt the features for diffusion model. Seems to me that this could result in the primary encoder not learning anything useful since the new layers(unused during inference) maybe doing most of the heavy lifting. Have the authors figured out a way to solve this issue ?

A3: To address concerns about the Adapter dominating learning, we ran a controlled experiment (Table 1 below) comparing: (1) freezing the encoder and (2) training both. Training both gave lower reconstruction loss, confirming the encoder is effectively optimized. The lightweight Adapter acts as a bridge to the diffusion model, supporting—not replacing—the encoder, ensuring efficient training and better visual representation.

Table1

In Table 2 the authors show the best results using the proposed framework + Qwen2-7B but they do not show results on Qwen2-7B by itself

A4: The experimental results for Qwen2-7B are presented in the third row of the "Expanding to other LLMs" section in Table 1 in the article. We will add these results to Table 2 in the final version.

This method is not particularly new except the proposed use of Diffusion models, as illustrated by the authors in Fig 2.

A5: Thank you for your interest. Instead of simply using diffusion for supervision, we propose a novel design to tackle long supervision paths and sparse textual signals in VLM training. Our Multi-Adapter Diffusion fuses multi-level visual and connector features, while MOE Cross Attention dynamically routes multimodal inputs for fine-grained fusion. Together, they create a compact, semantically rich supervision path that enhances visual representation. We will further clarify these contributions in the final version.

line 242-243, "/////" added for no reason.

A6: Thanks for pointing this out. The slashes will be removed in the final version.