LaViDa: A Large Diffusion Model for Vision-Language Understanding

W1. Fixed-Length Generation Constraints of Diffusion Models

Overall, we recognize that fixed-length generation is a major constraint for diffusion language models. This limitation applies broadly across all such models. Notably, recent studies published after the submission deadline have begun to address this issue. For example, DreamOn [1] introduces special tokens that allow for expansion of masked tokens or deletion of unused ones. We will explore incorporating these advances in future work to address this limitation.

W2. Limited Use Cases for Text Infilling

We incorporate additional results on text-infilling across three tasks shown in Fig. 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sample 100 images from the LAION-2B dataset and use GPT-4o to generate task-relevant questions for each image. We also ask GPT-4o to produce optimized prompts for VLMs for these tasks. For the text editing and script completion tasks, the prompts instruct the VLMs to edit a specific sentence or complete a specific line while leaving the rest of the text unchanged. For JSON completion, we validate schema correctness using Python's json package. For text editing and script completion, we use string matching to verify whether the required prefix and suffix are preserved in the generated text. The results are reported in the table below.

These results highlight the advantage of diffusion-based VLMs, which can provably achieve 100% constraint satisfaction without relying on careful prompt engineering. They also demonstrate the broader potential use cases for diffusion VLMs. We will incorporate more details of these evaluations in the final version.

W3. Performance Gap with SOTA

We acknowledge that our model currently shows a performance gap with state-of-the-art VLMs, particularly on OCR tasks. One contributing factor is that diffusion language models typically require more training steps to converge than autoregressive (AR) models, as shown in a recent study [2]. To test this in our setting, we conducted an additional two epochs of training, which resulted in substantial improvements on both OCR tasks (e.g., ChartQA) and conventional vision tasks (e.g., MME). These results suggest that additional training can significantly improve OCR performance. We hope that future work on diffusion language models will explore strategies for improving training efficiency. It is also worth noting that state-of-the-art VLMs are generally trained on more data and for more steps, which presents opportunities for further scaling.

Q1. Additional Speed-Quality Tradeoff

As requested, we provide additional performance–latency results on MathVista in the table below. We report the maximum generation length (L), number of sampling steps, average number of non-padding tokens generated, per-sample latency, and throughput of non-padding tokens. Per review policy, we will include a visualization of these results similar to Figure 6 in the camera-ready version. To aid interpretation, we bold all metrics that outperform the AR baseline in the table. With optimal parameter choices, our model achieves up to a 2.5× speedup over the AR baseline.

Q2. Context Length

The context length is 4096 tokens. After accounting for vision and prompt tokens (~1000), the maximum possible generation length is around 3000 tokens. However, over 95% of our training data has generation lengths under 1000, which suffices for most tasks. As shown in the Q1 table above, even when AR models are allowed unlimited generation length, they typically produce only ~200 tokens on complex math reasoning tasks. When we set L = 1024 for LaViDa, it generates around 400 non-padding tokens. In summary, the current generation length is sufficient for most tasks, and within this range, the diffusion model demonstrates a speed advantage over AR models. We acknowledge that long-context generation remains underexplored for diffusion language models and hope that future work will address this area.

Q3. Bagel

Bagel uses a continuous flow-matching objective with an MSE loss for image tokens, enabling parallel image generation. However, it uses autoregressive next-token prediction with a causal mask for language modeling (see Appendix Fig. 15 of their paper). Therefore, it does not support parallel language generation. We also note that it was posted on arXiv on May 20th, after our submission deadline.

Q4.Why Not Fully Mask the Answer Segment?

During inference, we decode a fully masked sequence into clean text tokens in a gradual manner. To support this process, it is important to train the model with varying mask rates. In practice, we sample a random mask rate between 0 and 1 during training. For mask rates less than 1, we must adopt a masking strategy that makes the best use of the training data. We already include an ablation study comparing complementary vs. random masking in Table 4a, with further discussion in Appendix B.4. We did not experiment with always masking the full sequence because that essentially reduces to learning a one-step generation model, which is extremely difficult.

Q5.If the number of visual tokens is increased to match state-of-the-art models, would the inference speed advantage still hold ?

We extend the speed comparisons in Table 3a and Figure 4b to include an older AR model (LLaVA-1.5-7B), which uses low-resolution inputs (and thus fewer visual tokens). Our results show that LaViDa remains faster than AR models, even though it uses more visual tokens.

Q6(a). Intern-VL Vision Encoder

Small models such as Mini-InternVL achieve superior performance by distilling knowledge from stronger models. For example, their 300M vision encoder is distilled from InternViT-6B, a significantly larger model. Therefore, this is not a fair comparison. We plan to explore distilled vision encoders in future work.

Q6(b). Could the performance improvements observed in this paper be primarily attributed to the visual encoder?

We include ablations of different vision encoders in Appendix C, Table 11. We selected SigLIP for the final model due to its strong overall performance, although it is not the best on math tasks. For example, LaViDa-CLIP slightly outperforms LaViDa-SigLIP on MathVerse in our small-scale ablations (20.05 vs. 19.67). However, the full-scale model using the SigLIP encoder outperforms the baseline model using a CLIP encoder (27.2 vs. 14.6 in Table 1), suggesting that performance improvements do not stem solely from the vision encoder. In fact, our LaViDa-CLIP model trained on small-scale data still outperforms the AR baseline (19.67 vs. 14.6) using less data.

[1] Zirui Wu*, Lin Zheng*, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, Lingpeng Kong (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond Fixed-Size Canvas.

[2] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.

W1 Additional Performance-Latency Results

As requested, we provide additional performance-latency results on MathVista in the table below. We report the maximum generation length (L), the number of sampling steps, the average number of non-padding tokens generated, the latency per sample, and the throughput of non-padding tokens. In accordance with review policy (no image allowed), we will include an additional visualization of these results similar to Figure 6 in the camera-ready version. For clarity, we highlight all metrics that outperform the AR baseline in bold in the table. With an optimal choice of parameters, our model achieves up to a 2.5x speedup compared to the AR baseline model.

W2 Evaluation of Text-Infilling Tasks

We create a constrained satisfaction test set that requires the model to generate four-line poems, where the first word of each line must begin with two specific letters. The first letter must be a consonant from the set {'b', 'c', 'd', 'f', 'g', 'h', 'n', 'p', 'r', 's', 't'}, and the second letter must be a vowel from the set {'a', 'e', 'i', 'o', 'u'}. The task asks the model to generate a poem with specific starting syllables (e.g., "ca") for each line. Sentence-level and sample-level satisfaction are computed by string matching to check if each line begins with the specified characters. Sample-level satisfaction is achieved if all lines satisfy the sentence-level constraints. The prompt template is shown exactly in Figure 4. We note that this is a reasonable prompt and that better prompting may further improve the performance of AR models. However, one of the key advantages of diffusion models is their ability to achieve 100% provable satisfaction of arbitrary constraints without the need for prompt tuning.

To further validate this claim, we include additional results on text-infilling for three tasks shown in Figure 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sample 100 images from the LAION-2B dataset and ask GPT-4o to generate a relevant question of each category for each image. We also ask GPT-4o to generate optimized prompts for VLMs for these tasks. For text editing and script completion, the prompts specifically ask the VLMs to edit a specific sentence or complete a specific line while leaving the rest of the text unchanged. For JSON completion, we verify schema compliance using the Python json package. For text editing and script completion, we use string matching to verify that the prefix and suffix are preserved in the output. The results are reported in the table below.

These results highlight the strength of diffusion-based VLMs, which consistently achieve a 100% constraint satisfaction rate without the need for careful prompting. We will provide additional details on these evaluations in the final version.

Q1 Complementary Masking

We observe that encoding and backpropagating through the vision encoder introduces considerable overhead. Since we split each image into 5 views, adding one more conversation effectively increases the batch size of the vision encoder by 5. We conducted additional measurements on 8 A100 GPUs. Using a per-GPU batch size of 2 with complementary masking results in 4.16 seconds per step, while a per-GPU batch size of 4 without complementary masking results in 7.01 seconds per step, highlighting the high overhead of vision encoding.

Another contributing factor is that we pad each batch to the longest sequence. The expected maximum length of 4 randomly selected sequences is greater than that of 2 randomly selected sequences.

Q2 GPU Hours

Total training time is 902 A100 (80GB) hours for LaViDa-LLaDa and 1,768 A6000 (48GB) hours for LaViDa-Dream.

Q3 Generation Length

We already report the values of L in Appendix B, Table 6. We will provide detailed average generation lengths for each task in the final version. In general, the generation length for short-answer tasks is typically fewer than 10 tokens. Changing the maximum length does not affect performance or the number of generated tokens, as the prompt often includes format instructions such as "answer with a few words." However, when evaluating our reasoning model on MathVista, we found that maintaining a generation length greater than 200 tokens generally yields stable performance, whereas aggressively reducing it leads to performance degradation.

As an example, we include the results of varying generation length for AI2D and MathVista in the tables below.

[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.

[2] Wu, Z., Zheng, L., Xie, Z., Ye, J., Gao, J., Feng, Y., Li, Z., Wu, V., Zhou, G., & Kong, L. (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond a Fixed-Size Canvas.

Dear Reviewer Y6e3

This is a friendly reminder that the end of discussion period is fast approaching. We have conducted several additional experiments to address your concerns. We would appreciate it if you could let us know if those concerns have been successfully address. If you have any last-minute questions, feel free to let us know as well.

Best

Authors

We thank the reviewer for the constructive feedback.

W1 Performance on OCR Tasks

We acknowledge that our model’s performance on OCR tasks leaves room for improvement. One contributing factor is that diffusion language models generally require more training steps to converge than autoregressive (AR) models, as shown in a recent study [1]. To validate this hypothesis in our setting, we conducted an additional 2 epochs of training. This resulted in substantial improvements on both OCR tasks such as ChartQA and conventional vision tasks such as MME. Hence, OCR performance can be improved through additional training. We hope that future work on diffusion language models will further enhance training efficiency.

W2 Fixed-Length Generation Constraints of Diffusion Models

We recognize that fixed-length generation is a significant constraint for diffusion language models. This limitation applies to all current diffusion-based language models. However, recent studies released after our submission deadline have begun to address this issue. For example, DreamOn [2] introduces special tokens that allow for expanding masked tokens or deleting unused ones. We plan to explore incorporating such advances in future work to overcome this limitation.

W3 Memory Footprint

We use prefix caching primarily to improve inference speed (see Table 3.a). On the A5000 GPU, we do not observe a significant difference in peak memory usage between the two settings. For example, memory usage is 22,811 MB with caching and 21,668 MB without caching.

Q1 Memory Footprint

Please refer to W3 above.

Q2 OCR Tasks

Please refer to W1 above.

Q3 Multi-Modality

As part of an ongoing follow-up project, we have integrated a VQ-tokenizer and additionally trained the model on text-to-image generation tasks. We achieved a GenEval score of 0.75, outperforming previous state-of-the art masked generative model Meissonic[3] (0.54) and a current work MMaDa[4] (0.63) . These are preliminary results and will be part of a separate future submission. Nonetheless, these results indicate that it is entirely feasible to extend our setup toward building a unified model for both understanding and generation using a single diffusion objective.

References

[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.

[2] Zirui Wu*, Lin Zheng*, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, Lingpeng Kong (2025). DreamOn: Diffusion Language Models for Code Infilling Beyond Fixed-Size Canvas.

[3] Bai, J., Ye, T., Chow, W., Song, E., Chen, Q. G., Li, X., ... & Yan, S. (2024, January). Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations.

[4] Yang, L., Tian, Y., Li, B., Zhang, X., Shen, K., Tong, Y., & Wang, M. (2025). Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809.

Dear Reviewer kZHX

We just noticed that while we included the memory footprint of LaViDa as requested in our original response, you also asked for a comparison with the memory cost of similar-sized auto-regressive models. We apologize for this oversight and included the modified version of our response as below.

W3 Memory Footprint

We use prefix caching primarily to improve inference speed (see Table 3.a). On the A5000 GPU, we do not observe a significant difference in peak memory usage between the two settings. For example, memory usage is 22,811 MB with caching and 21,668 MB without caching. This peak memory was comparable to similar-sized AR model. Specifically, the AR model Open-Llava-Next-8B has a memory usage of 20,112 MB. Both results are obtained with batch size 1 on MathVista benchmark with a max generation length of 256, using the Flash-Attention-2 kernel.

We thank the reviewer for the constructive feedback.

W1 Additional Performance-Latency Results

As requested, we provide additional performance-latency results on MathVista in the table below. We report the maximum generation length (L), the number of sampling steps, the average number of non-padding tokens generated, the latency per sample, and the throughput of non-padding tokens. In accordance with review policy (no image allowed), we will include an additional visualization of these results similar to Figure 6 in the camera-ready version. For clarity, we highlight all metrics that outperform the AR baseline in bold in the table. With an optimal choice of parameters, our model achieves up to a 2.5x speedup compared to the AR baseline model.

W2 OCR Performance

As requested, we provide additional results on OCR-Bench in the table below. We acknowledge that our model’s performance on OCR tasks has room for improvement. One contributing factor is that diffusion language models typically require more training steps to converge than AR models, as demonstrated in a recent study [1]. To validate this hypothesis in our setting, we conducted an additional two epochs of training. This resulted in substantial improvements on both OCR tasks such as ChartQA and conventional vision tasks such as MME, confirming that OCR performance can be enhanced with additional training. We hope future research on diffusion language models will further improve their training efficiency.

W3 Additional Quantitative Experiments on Text-Infilling

As requested, we have included additional results on text-infilling for three tasks presented in Figure 6: JSON satisfaction, text editing, and visually grounded script completion. We randomly sampled 100 images from the LAION-2B dataset and employed GPT-4o to generate relevant questions for each image category. We also utilized GPT-4o to create optimized prompts for these VLM tasks. For the text editing and script completion tasks, the prompts explicitly instruct the VLMs to edit a specific sentence or complete a specific line while preserving the remaining text. For JSON completion, we verified the schema compliance using the Python JSON package. For text editing and script completion tasks, we employed string matching to ensure that the prefix and suffix were preserved in the generated output. The results are reported in the table below.

These results highlight the strength of diffusion-based VLMs, which consistently achieve a 100% constraint satisfaction rate without the need for careful prompting. We will provide additional details on these evaluations in the final version.

Q1 Concurrent Works Discussion

We are aware of the concurrent works mentioned, which were released after our submission deadline. We will include relevant discussions and citations for these works in the revised version.

Q2 Typographical and Grammatical Errors

We thank the reviewer for identifying these editorial errors and will correct them in the final version.

[1] Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.15857.

Dear Reviewer tRob

This is a friendly reminder that the end of discussion period is fast approaching. We have conducted several additional experiments to address your concerns. We would appreciate it if you could let us know if those concerns have been successfully address. If you have any last-minute questions, feel free to let us know as well.

Best

Authors