DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression

Thank you for your thorough review and valuable feedback. We have addressed each point below. Please see the following responses for details.

1. The inference speed of DiVLA(Same as R3#1)

This issue was partially addressed in our response to R13rJ#1. Bellow, we fristly clarify why DiVLA-7B has similar number of parameters but is 8 times faster than OpenVLA-7B, followed by an explanation of how vLLM optimizes inference efficiency in VLMs.

1. Inference Speed. Diffusion-based VLA operates significantly faster than autoregressive VLA in robot action generation. In robotics, inference speed refers to the number of actions produced per second. For instance, using the Franka Emika with 7 degrees of freedom, OpenVLA must predict 7 tokens for each action. In a typical setup, the model needs to predict 30–60 actions for each incoming observational state, requiring 210–420 tokens for prediction. In contrast, diffusion-based VLA, such as DiVLA, only needs 10 denoising steps to predict 30–60 actions. This conclusion is supported by FAST [1], which finds that diffusion-based VLA is much faster than autoregressive VLA. In summary, DiVLA benefits from significantly faster inference times due to its use of a diffusion process, confirming the accuracy of our inference tests.

2. DiVLA with vLLM. Regarding our DiVLA with vLLM, the inference speed of VLM can achieve approximately 2× speedup. vLLM enhances VLM inference efficiency through PagedAttention, which prevents memory fragmentation by storing KV cache in fixed-size pages, reducing memory overhead and improving stability for long-sequence generation. Additionally, optimized CUDA kernels accelerate attention computation, resulting in significantly faster inference compared to traditional frameworks, often achieving 2–3× speedup.

2. Details architectural diagram for DiVLA

We apologize for the absence of the detailed architectural diagram. As you correctly pointed out, the “projection layer" and "action decoder" are crucial components of our framework but are not represented in Figure 1. To address this, we will include a more comprehensive diagram that clearly illustrates the entire architecture, including these specific modules and their interactions.

The projection layer consists of two MLP layers with LayerNorm. It bridges the VLM's embedding to the diffusion model, aligning their dimensions. Additionally, our action decoder follows the standard diffusion policy design [2]. It takes the embeddings from the VLM as conditions and predicts the noise based on the noisy actions, repeating this process k times to obtain the final action chunk.

3. Clarify on Language Generation and the L_NTP loss function

In Section 3.2 Model Design Choices, we state that our training loss comprises tow part: the diffusion loss L_diff and the NTP loss L_ntp. The ground truth for L_ntp is based on language supervision. Consequently, we can generate reasoning tokens (i.e. language output) during inference.

4. Failure case analysis

We have failure case analysis (page 7 lines 378-384) which systematically examines how dynamic reasoning enables action self-correction. To extend this discussion, we propose a dual-aspect failure taxonomy: erroneous reasoning chains and action execution errors. As demonstrated in Table through controlled experiments on the bin-picking task, we evaluate 4 trials per object across both seen and unseen categories. Seen objects are appeared in robot data while unseen objects are not. Our model could successfully pick up those objects when facing seen objects and we empirically observe that action always fails when the model cannot recognize the unseen objects though. We attribute the recognition degradation to robot-data overfitting as exclusive training on robotic demonstrations diminishes the base VLM's open-vocabulary recognition capacity. Thus a potential solution is co-training the VLA with robot data and vision-text data, preserving both generalizable visual concepts and action-specific skills.

5. Typo problem

Thank you for your careful review. We appreciate your feedback regarding the misreferenced table and figure. These issues have been corrected in the next version of the manuscript, which will be updated accordingly.

[1]. FAST: Efficient Action Tokenization for Vision-Language-Action Models

[2]. Diffusion policy: Visuomotor policy learning via action diffusion.

Thanks for your careful review and valuable comments. We address each question below.

1. Why DiVLA-7B has similar number of parameters but is 8 times faster than OpenVLA-7B?

Thank you for pointing out this question.

1. Diffusion-based VLA operates significantly faster than autoregressive VLA in robot action generation. In robotics, inference speed refers to the number of actions produced per second. For instance, using the Franka Emika with 7 degrees of freedom, OpenVLA must predict 7 tokens for each action. In a typical setup, the model needs to predict 30–60 actions for each incoming observational state, requiring 210–420 tokens for prediction. In contrast, diffusion-based VLA, such as DiVLA, only needs 10 denoising steps to predict 30–60 actions. This conclusion is supported by FAST [1], which finds that diffusion-based VLA is much faster than autoregressive VLA. In summary, DiVLA benefits from significantly faster inference times due to its use of a diffusion process, confirming the accuracy of our inference tests.

2. To further ablate the impact of vLLM, we report the following test on an A6000 GPU, where T_ntp and T_diff represent the time spent on the NTP and diffusion processes, respectively.

2. Clarify the role of reasoning and action tokens in DiVLA

We appreciate you highlighting this.

1). Are both tokens used as conditions for diffusion model? Yes, both action and reasoning tokens serve as different conditions on action denoising. Specifically, action tokens mainly include visual embeddings providing global observations and raw instruction tokens. On the other hand, reasoning tokens encapsulate hierarchical substep-level features that decompose complex tasks into temporally executable steps.

2). How action tokens are generated? Action tokens refer to vanilla visual tokens and tokens from raw instruction processed by VLM backbone. They are not generated autoregressively.

3. Whether DiVLA's ability to perform zero-shot bin picking is attributed to its generalization capability or spatial memory?

Thank you for your insightful question. To answer your question, the DiVLA’s ability to perform zero-shot bin-picking definitely stems from its strong generalization capability.

As outlined in page 8 line 410-420, the bin picking experiment used 102 unique objects that were entirely absent from the training set, ensuring that the model had to generalize to novel objects. In Figure 8, we shows that the objects have varying size and height. Object like white tape is only 1cm high, while the toy dragon is 10cm. Also, in Figure 7, we present some unseen objects that we used for zero-shot bin picking tasks. It can be observed that these objects exhibited substantial variations in shape, size, color, texture, and deformability, ensuring that the model could not rely on simple heuristics such as shape consistency or fixed spatial features. Our model, DiVLA, achieved a 63.7% success rate, substantially outperforming baselines (Diffusion Policy: 8.9%, Octo: 19.6%, TinyVLA: 23.5%, OpenVLA: 28.4%), underscoring its superior generalization for unseen objects.

[1]. FAST: Efficient Action Tokenization for Vision-Language-Action Models

Thank you for your thorough review and valuable feedback.

1. Simulated evaluation

Real-world evaluation is more challenging than simulation. While our work emphasizes complex real-world tasks like long-horizon bin-picking and bimanual table bussing, we also evaluate DiVLA on two standard simulation benchmarks, Calvin and Libero. Compared with baselines including Diffusion Policy, Octo, OpenVLA, RT-1, Robo-Flamingo, and GR-1, DiVLA achieves the best performance under a unified training setup (results: https://i.postimg.cc/xTybS5p3/simulation.png), demonstrating its robustness across both real and simulated environments.

2. Ablation on key modules

We do agree that ablation study is necessary for better support our claim. We have addressed this problem in response to Reviewer fKzV#3. Please refer to that section for details.

3. Clarify difference between points 1 and 6 in the introduction section

We are sorry for the confusion between point 1 and point 6. Generalizing manipulation skills on novel objects and robust to dynamic environment are two main challenges in robot learning. Point 1 focuses on DiVLA’s ability to recognize novel objects through self-generated reasoning, while Point 6 highlights its robustness to distractors and dynamic environments.

4. Citation for “Research has shown … various embodiments.”

Thanks for pointing this out. The next-token-prediction method used in OpenVLA struggles to complete tasks when adapting to new embodiments [1] and performs poorly when learning dexterous skills with high-frequency control [2].

5. Limitation of current diffusion-based VLAs and Superiority of DiVLA

Thanks for your insightful question. We have addressed this problem in response to Reviewer fKzV#5. Please refer to that section for details.

6. Prompt for gpt-4o

Thanks for pointing this out. The prompt is at here: https://i.postimg.cc/kXcDH4Dj/prompt.jpg.

7. Clarify whether the model will confuse similar objects or not

Thanks for your reminder. While DiVLA shows strong generalization, it occasionally misclassifies unseen objects, such as confusing small stuffed toys with toy cars. However, DiVLA handles shape-similar, color-distinct objects well, thanks to its robust color perception. Real-time reasoning traces help diagnose misclassifications and offer insights for further improvement.

8. Comparison on model size, inference speed, training speed with baselines

Thanks for pointing this out. We will discuss the model size and inference speed separately.

1). Model size: Our DiVLA, which uses only 2B parameters—1/3 of OpenVLA’s parameters—achieves the best performance across all tasks, demonstrating our method's model efficiency.

2). Inference speed: As shown in table, DiVLA-2B can achieve 82hz control frequency at test time with vLLM and action chunk. This is 20x faster than OpenVLA. The results highlight the inference efficiency of diffusion-based VLAs over autoregressive VLAs.

3). Training speed: We have addressed this problem in response to Reviewer fKzV#1. Please refer to that section for details.

9. Clarify details on reasoning and action tokens

Thanks for your feedback.

1). Difference between reasoning and action tokens: Action and reasoning tokens serve as different conditions on action denoising. Specifically, action tokens mainly include visual embeddings providing global observations and raw instruction tokens. On the other hand, reasoning tokens encapsulate hierarchical substep-level features that decompose complex tasks into temporally executable steps.

2). How is VLM trained to output reasoning and action tokens: Action tokens refer to vanilla vision tokens and tokens from raw instruction processed by VLM backbone while reasoning tokens are task-specific language generated by VLM. Thus, we only use standard next-token prediction loss for supervising reasoning generation.

3). Number of both tokens: As previously noted, action tokens comprise tokens from raw instruction and visual tokens, whereas reasoning tokens are autonomously generated by the VLM. Action tokens are much longer than reasoning tokens as a large number of vision tokens.

10. Typo problems

Thanks for your feedback. We will improve them in updated version.

[1]. TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation

[2]. FAST: Efficient Action Tokenization for Vision-Language-Action Models

[3]. CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

[4]. RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

[5]. π0: A Vision-Language-Action Flow Model for General Robot Control

Thanks for your careful review and valuable comments. We address each question below.

1. Training efficiency for DiVLA compared to baselines

Thank you for your valuable feedback and insightful advice. Since DP and Octo perform significantly worse across most tasks, our comparison focuses on competitive VLAs.

1). Pretraining Computations: OpenVLA requires 27 epochs (21,500 A100 GPU hours) for action token prediction on the OXE dataset, while DiVLA achieves language rationale generation in just 2.5 epochs (155 H800 GPU hours) on part of Droid dataset. DiVLA uses only 39K pretraining samples, 25 times fewer than OpenVLA's 970K, yet still delivers interpretable reasoning, demonstrating superior efficiency.

2). Finetuning Computations: All models are trained with the same batch size on 8×H800 GPUs. TinyVLA takes 12 hours to adapt, while DiVLA outperforms it by 29.8% in factory sorting and reduces finetuning time compared to OpenVLA, achieving a 20.9% improvement. DiVLA shows superior efficiency and adaptability in finetuning.

3). Model Size and Memory Usage: As shown in the table, DiVLA is smaller than OpenVLA (2B vs. 7B parameters) and requires less GPU memory, making it more reproducible and accessible.

Overall, DiVLA demonstrates superior computational efficiency in both pretraining and finetuning, achieving significant performance gains and highlighting the advantage of our approach.

2. Ablation on projection layer of action tokens

Thanks for pointing it out. We ablate two projection layer, as requested for action tokens on 3 tasks (described in page 4 line 189-199). For each variants we evaluated them for 11 trails for each task.

As shown in table, using two-MLP+LayerNorm as projection layer performs relatively better than FiLM which highlights our choice is effective.

3. Ablation on three reasoning injection modules (MLP, Q-Former, and FiLM)

We thank the reviewer so much for pointing this out. We ablate three reasoning injection modules, 1) Plain MLP like LLaVA [1], 2) Q-former as in BLIP [2], and 3) FiLM. For each variants, we evaluated on three tasks, each with 11 trials.

As demonstrated in the table, DiVLA with the FiLM module significantly outperforms plain MLP and Q-former, confirming the efficacy of our design. Intuitively, reasoning tokens act as conditioning factors, enhancing robot action generalization without dominating predictions. The FiLM architecture effectively fulfills this conditioning role.

4. Do we use 2e-5 for all module?

Yes, we use initial learning rate of 2e-5 to all layers, including newly initialized projection modules. We didn’t do hyper-parameter search in our experiments.

5. It is interest that Diffusion-VLA can generate language rationales alongside its robot action, how does it compare to OpenVLA and pi0 in terms of interpretability

Thank you for raising this point. DiVLA is the only model that generates language rationales alongside robot actions, making it interpretable. In contrast, OpenVLA and Pi0 cannot generate language. DiVLA uses reasoning for dynamic scene understanding and task planning, revealing the model's decision-making process and improving long-horizon task completion by making failure reasons visible to users.

Our experiments shows the importance of making VLA interpretable. Specifically, as illustrated in Figure 3, on the sorting task involving 4 progressively challenging settings, DiVLA can generate real-time reasoning traces (e.g., identifying objects and categorizing them). It achieves 20.9% improvement over OpenVLA across all settings. Furthermore, as shown in Figure 4, DiVLA exhibits remarkable zero-shot generalization, outperforming OpenVLA by 35.3% on 102 unseen objects. These experimental results shows that introducing interpretability to VLAs significantly enhances their ability to execute long-horizon tasks and generalize to novel objects.

6. For format, section renaming and figure annotation

Thanks for pointing this out. We will improve these formating problems, section renaming and figure annotation in updated version.

[1]. Improved Baselines with Visual Instruction Tuning

[2]. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation