HybridVLA: Collaborative Autoregression and Diffusion in a Unified Vision-Language-Action Model

[W1]. Clarification on Reasoning Capability

W1-1. LLM's Intrinsic Reasoning Prowess:

Thank you for your comments. We would like to clarify that we do not claim the reasoning capability of LLMs arises entirely from the autoregressive next-token prediction. In Lines 37–43, our intention is to convey that diffusion-based VLA methods [13, 14] leverage the pretrained VLM for multimodal understanding and use it to provide semantically rich latent features for the diffusion head. We did not mean to suggest that diffusion-based VLA completely lack contextual reasoning abilities, but rather that they may miss some degree of reasoning capacity due to (1) the newly injected diffusion head trained from scratch, and (2) the absence of iterative output generation through the next-token prediction paradigm that benefits from the LLM’s internet-scale pretraining.

As you commented, LLM reasoning stems from a combination of factors, including the attention mechanism, large-scale parameters and pretraining, and the autoregressive output paradigm. And our goal is to explore how diffusion-based generation can better utilize the inherent reasoning capabilities of pretrained VLMs. We will improve the writing in the revised version as per suggestion. For example, Lines 40–42 will be updated as follows:

  "Moreover, although diffusion-based VLA models leverage the pretrained VLM to extract semantically rich multimodal latent features as conditions for the diffusion head, the diffusion head itself has not undergone internet-scale pretraining and does not fully exploit the VLM’s original reasoning generation paradigm."

W 1-2. Diffusion Models' Utilization of Context:

Unlike general large-scale diffusion models [18, 19], current diffusion-based VLA methods struggle to benefit from large-scale pretraining due to the high cost of collecting robotic data. While they can condition on VLM-derived context, they rely on separate diffusion heads trained from scratch. To address this, we introduce a unified VLA paradigm (HybridVLA) that integrates diffusion’s Markovian denoising steps into the LLM’s prediction process via a shared token sequence. This design allows diffusion action generation to inherit the LLM’s pretrained reasoning paradiam, treating each denoising step as an iterative reasoning process.

Furthermore, we conduct additional experiments to validate HybridVLA’s advanced contextual reasoning capabilities. In robotic manipulation, contextual reasoning refers to the policy’s ability to integrate task instructions and visual inputs for closed-loop control. We evaluate this performance on tasks with unseen instructions and novel visual observations.

For unseen task instructions, we conduct experiments in the RLBench simulator to evaluate performance under diverse unseen task instructions. Each RLBench task includes multiple semantically equivalent instructions, with different ones used during training and testing. As shown in the table below, while HybridVLA experiences a slight drop in average manipulation success rate, the decrease is smaller than that of other diffusion-based VLA baselines. These results demonstrate HybridVLA’s relatively robust contextual reasoning ability when handling flexible task instructions.

For unseen manipulated objects, as presented in Table 4 of the main paper, we evaluate HybridVLA’s contextual reasoning ability under unseen visual observations. To further validate this, we conduct an additional long-horizon experiment on the “Open drawer & place inside” task. Unlike the visualization in the last row of Figure 5 (Apendix), we modify the test setting by placing not only an orange into the tray but also adding two unseen objects. As shown in the table below, HybridVLA outperforms other methods in handling novel objects, which highlights its capacity for effective reasoning under unfamiliar visual conditions.

[W2]. Explanation of Optimization Goals and Unification

We would like to clarify that HybridVLA does not claim to unify the generation process. Instead, we improve the action generation accuracy by integrating the token sequence formulation and feature representation of two generation methods.

W2-1. Distinct Mathematical Foundations:

We first illustrate why interegrating these two generation methods can boost feature representation.

(1) In HybridVLA, the autoregressive and diffusion branches aim to model the same action distribution space as the action data are normalized in the same way (range [-1,1]). Note that the discrete action is simply a quantized version of the same action distribution. Despite differing in loss formulations, the diffusion and autoregressive branches both aim to approximate the same conditional action distribution, leveraging distinct generative paradigms to model the same data. That is, they both seek to maximize the log-likelihood of the training actions. The autoregressive branch achieves this by explicitly modeling the conditional probability of each action given its preceding context. The diffusion branch models the data generation process as iterative denoising and approximates the log-likelihood of actions by maximizing the ELBO. Despite taking different modeling paths, both branches aim to produce distributional representations that align with the action distributions. Therefore, their imitation learning objectives share the same modeling target, making joint modeling and optimization under shared representations beneficial.

(2) This joint optimization strategy is also supported by recent unified VLM works, such as Diffusion Forcing [69] (NeurIPS 2024) and Transfusion [70] (ICLR 2025 Oral). In particular, Transfusion trains a single Transformer architecture that integrates both autoregressive and denoising objectives, enabling generation of discrete text and continuous images. Meanwhile, unlike these prior works, which perform joint modeling over different output modalities, HybridVLA operates in a shared SE(3) action modeling space.

(3) We further validate, using Principal Component Analysis (PCA) distribution, that jointly optimizing diffusion and autoregressive losses leads to mutual enhancement of their feature representations. Specifically, we select several trajectories from the Pick actions and Place actions, and use the corresponding frames as inputs to the model. We then extract both the diffusion-denoised tokens and the autoregressive action tokens, and project them into a 2D space using PCA. We compare the feature distributions produced by models trained with our collaborative training recipe against those from models where each generation branch is trained independently. Training independently means that we apply either the diffusion loss or the autoregressive loss exclusively on our model. As shown in the table below, jointly trained diffusion and autoregressive tokens form tighter intra-class clusters and exhibit larger inter-class distances in both Pick and Place action data. The improved feature representations demonstrate that joint optimization implicitly regularizes the model to retain latent dimensions beneficial for both generation forms.

W2-2. Parameter Sharing vs. Process Unification:

Regarding “seamlessly incorporating diffusion denoising into the next-token prediction process,” we aim to convey that we integrate the diffusion-denoised tokens into the token sequence of the LLM. This does not imply a unification of the underlying generative paradigms, but a unification of token sequence formulation of two generation methods with causal attention (L164) .

In HybridVLA, diffusion denoising steps and autoregressive token predictions are embedded within a unified token sequence and jointly processed via causal attention. Instead of treating the two branches as separate heads, the model performs joint modeling within a shared token context. Notably, the denoised action token from diffusion is explicitly used as a condition for the autoregressive prediction, directly influencing the probability distribution of the generated actions. This establishes structural coupling within the LLM’s modeling process, going beyond mere decision-level fusion.

[Q1]. Misreading of Inference Speed

The inference speed values in Table 2 may have been misread: OpenVLA-7B achieves 6.3 Hz, while HybridVLA-7B achieves 6.1 Hz. The comparison remains fair, as OpenVLA uses a KV-cache for autoregressive generation, and we extend the same mechanism to the diffusion-based iterative denoising process within our token sequence design.

[Q2]. Additional experiments on SimplerEnv

Thank you for your valuable suggestion. We additionally evaluate our model in the SimplerEnv variant aggregation setting using the Google robot. However, since other baselines (e.g., Pi0) are not originally trained on the same subset of the Fractal [86] dataset, we perform consistent fine-tuning across all baselines for fair comparison. Due to time constraints, the remaining results will be included in the revised version.

We sincerely thank the reviewer for the detailed comments and valuable time. We hope our responses address your concerns. If any issues remain, please let us know, and we will respond promptly.

Point 2. Diffusion-based VLA possesses reasoning capabilities.

We fully agree with your perspective. We also acknowledge that existing diffusion-based VLA models possess reasoning capabilities. Based on your constructive suggestion, we will revise the description of diffusion-based VLA models in Lines 40–42 of the main text (e.g., rebuttal W1-1).

In this paper, our objective is to more effectively leverage the pretrained knowledge of vision-language models (VLMs) within diffusion-based VLA architectures. Specifically, we aim to: 1) utilize an internet-scale pretrained LLM as the backbone for diffusion-based action generation, and 2) emulate the iterative output generation paradigm inherently adopted by LLMs during large-scale pretraining.

To further compare the design choices of using an LLM as the diffusion backbone versus attaching a separate diffusion head to the end of HybridVLA, we conduct an additional experiment on the RLBench simulator. As shown in the table below, we compare our original HybridVLA with a variant in which a DiT head (with 308M parameters) is appended after HybridVLA. 'HybridVLA-dif' (7B) integrates diffusion and autoregressive generation during training but relies exclusively on diffusion-based actions during inference. 'HybridVLA+DiT Head' also utilizes only diffusion-based generation during inference. The results demonstrate that HybridVLA and HybridVLA-dif converge faster and enables more robust action generation.

Point 3. The Reasoning Capabilities of Diffusion Models and Additional Contextual Reasoning Experiments

Thank you for your comments. We would like to further express our acknowledgment that diffusion-based VLA models do possess reasoning capabilities. We will revise our description of the relationship between reasoning capability and next-token prediction (NTP) to make it more precise and rigorous in the revised version.

However, regarding Bagel, we would like to clarify that it does not use a separate diffusion head as the backbone for diffusion generation. Instead, it employs either a full LLM (MoT variant) or FFN blocks from the LLM (MoE variant) as the diffusion backbone. Diffusion generation is performed iteratively on the LLM itself. This design aligns with our intuition, as we also aim to use the entire LLM as the backbone for diffusion-based action generation in order to leverage its rich pretrained knowledge and achieve more robust action generation. Meanwhile, we retain the NTP mechanism because LLaMA 2 (our LLM backbone) adopts this generation paradigm during its large-scale pretraining. Thus, we aim to fully leverage the LLM’s original generation process and its inherent capabilities.

In addition to the RLBench simulator, and to better evaluate the contextual reasoning capabilities of our method, we also conduct additional experiments on unseen task instructions and unseen manipulated objects in the rebuttal W 1-2. Our model demonstrates more robust performance than existing diffusion-based VLA methods in handling flexible task instructions and reasoning about unfamiliar visual conditions.

Point 4. Effectiveness of our integration design

Our design of integrating autoregression and diffusion action tokens in a shared token sequence are shown in Table 1 and Section 3.2:

• We carefully design the integration strategy for diffusion and autoregressive action tokens. We observe that if the autoregressive tokens are placed first, the ground truth (GT) action labels contained in the input during training (a normal learning paradigm in LLM) will leak into the diffusion model’s conditions, negatively impacting the learning of diffusion action tokens. Therefore, we arrange the sequence so that diffusion tokens are placed before autoregressive action tokens. The denoised action token from the diffusion branch is explicitly used as a conditioning signal for the autoregressive prediction, directly influencing the probability distribution of the generated actions.
• Meanwhile, we introduce special tokens <BOD> (beginning-of-diffusion) and <EOD> (end-of-diffusion) to encapsulate the diffusion action tokens, clearly distinguishing the boundaries between diffusion and autoregressive action tokens, and ensuring consistent and stable action generation.
• We further investigate how to encode the robot state and determine its optimal placement within the LLM’s token sequence to enhance temporal consistency in action prediction.

The benefit of our design:

Empirically, the effectiveness is demonstrated in Table 3. All else being equal, comparing Ex1 and Ex2 (which use only the diffusion branch for test) shows a 6% difference between training with both branches and training with only the diffusion branch. Similarly, comparing Ex3 and Ex4 (which use only the autoregressive branch for test) reveals a 5% difference between training with both branches and training with only the autoregressive branch. The results empirically demonstrate that the Collaborative Training Recipe improves the model’s ability in both diffusion-based action generation and autoregressive generation compared to training them separately.

In analysis, we would like to present our insights into the improvement gains achieved through this collaborative optimization approach:

• We enable modeling actions at different levels, namely at the action level and the DoF (degree of freedom) level, to achieve a more robust action representation. Action-level diffusion action generation: This manner models all action tokens in parallel by denoising the entire 7-DoF (e.g., single arm) action sequence simultaneously, effectively capturing the overall action dimensions as a whole. This allows for maintaining coherence and integrated action learning across all degrees of freedom. DoF-level autoregressive action generation: This manner predicts each DoF token sequentially, paying closer attention to the relationships between individual degrees of freedom within the single action. This allows the model to capture intricate dependencies and constraints between different DoFs. Since both imitation learning objectives share the same target (i.e., action), jointly optimizing them enables the model to learn more robust representations, which in turn mutually enhance the accuracy of both types of action outputs.

• It effectively impacts their feature representation. As shown in the PCA results in W2-1 (3) in rebuttal, collaborative optimization forms tighter intra-class clusters and exhibits larger inter-class distances, both in diffusion and autoregressive tokens. These refined PCA distributions suggest that collaborative training facilitates alignment and mutual enhancement between generation branches, resulting in more structured and robust action representations.

Point 5. Why did we mention Transfusion?

We appreciate your comments. Indeed, unlike Transfusion, which focuses on unifying image and text generation, HybridVLA addresses a different problem, which involves collaboratively modeling continuous SE(3) actions for robotic control. However, our motivation for mentioning Transfusion in the rebuttal was to illustrate two key similarities with our work: 1) the joint optimization of autoregressive and diffusion losses within a single Transformer model is effective, and 2) diffusion-related tokens can be embedded into the autoregressive token sequence.

While Transfusion does not fuse outputs from different generation objectives, our method integrates both autoregressive and diffusion-based action outputs. This is based on a simple observation in our paper that the two generation outputs exhibit different strengths across manipulation tasks, as shown in Table 7 of the appendix. Based on this observation, we perform adaptive ensemble of the two sets of actions in the SE(3) space, which leads to more robust robot control.

Point 6. Inference speed clarification

We will include this clarification in the revised version of the paper. The higher inference speed of HybridVLA-dif (7B)(9.4hz) compared to OpenVLA (7B)(6.3hz) in Table 2 can be attributed to its inference configuration. As described in Lines 71–73 and 274–276, HybridVLA-dif integrates both diffusion and autoregressive generation during training but relies solely on diffusion-based actions at inference, achieving an inference speed of 9.4 Hz. Specifically, it performs only 4-step DDIM denoising during testing, which results in faster inference than OpenVLA (7B).

In contrast, the full version of HybridVLA incorporates the Collaborative Action Ensemble mechanism to enhance the robustness of robot control. As you rightly noted, this version performs full autoregressive generation during inference, resulting in a lower inference speed (6.1 Hz) than OpenVLA.

Once again, thank you for all your valuable suggestions and time, which we will carefully address in the revised version. We sincerely hope that our revisions adequately resolve your concerns. If you have any further concerns or questions, please do not hesitate to let us know. We would be happy to provide a prompt and detailed response.

Point 1. Revision of the Motivation and Associated Unclear Descriptions

We sincerely appreciate your prompt response and the time. We also apologize for any remaining concerns and assure you that we had no intention of misrepresenting your comment. However, due to the rebuttal’s character limit, we are afraid that we didn’t clearly convey our intended idea. Therefore, we’d like to further explain the original intent behind our writing, as well as our motivation for combining diffusion-based and autoregressive generation tokens into a single shared pretrained LLM.

We completely agree with your perspective that “reasoning capability is not exclusively dependent on, or solely manifested through, autoregressive next-token prediction.” This is exactly what we intend to convey in both the main paper and the rebutal.

Revision of related sentences L4 and L31: However, since this ambiguity has now arisen and your thought means a lot to us, we want to address any potential concerns. To avoid misunderstandings caused by unclear writing, we plan to revise our sentences in Lines 4 and 31 as follows:

Recent autoregressive vision-language-action (VLA) methods have inherited the pretrained parameters of vision-language models (VLMs) and adopt their next-token prediction output paradigm.

Reiteration of our motivation: In this paper, our goal is to improve the ability of VLA models to better inherit the pretrained knowledge of VLMs, especially in diffusion-based VLA settings where the VLM serves as the backbone and a DiT module is used as the diffusion generation head, as in Pi0 and CogAct.

We aim to enable diffusion action generation to more fully leverage the pretrained knowledge of LLMs, rather than just utilizing the multi modal understanding latent feature from VLMs. Therefore, given the pretrained knowledge of LLMs, we first explore whether directly using large-scale pretrained LLMs as the backbone for diffusion-based action generation is beneficial. Meanwhile, since the iterative output paradigm (NTP) is the original and effective output paradigm of LLMs, we secondly explore whether simulating this output paradigm across the shared token sequence can improve the overall action representation. We will follow your suggestion and improve the related motivation description in the revised version.

The rationale of our motivation:
Through extensive simulation, real-world experiments, and ablation studies in the main paper, we have observed that our approach is indeed effective, providing experimental evidence for its rationale. Additionally, other reviewers have acknowledged the soundness of our motivation; for instance, Reviewer RfcC comments that it “makes sense and is well-motivated.” Meanwhile, Reviewer Ez2J also claims that our framework “is novel and also important for the whole community.”

In conclusion, we would like to reiterate that we are fully aligned with your perspective and strongly agree that "reasoning capability" does not solely originate from NTP. The ambiguity between our views may have arisen due to our unclear writing, which will be carefully addressed in the revised version.

Dear Reviewer emK6,

As the discussion period draws to a close, we sincerely apologize for the interruption and would like to kindly check whether our additional responses have addressed your concerns. In line with your suggestions, we will incorporate all the promised experiments and revise the writing to avoid any potential misunderstandings or controversies. Thank you once again for your valuable feedback and time.

Paper 16855 authors

Based on my review and the authors' rebuttal, my position on this work remains unchanged. The authors have failed to provide convincing explanations for my core concerns, particularly regarding the unclear motivation, lack of methodological rigor, and ambiguous comparisons with existing work. Therefore, I will maintain my score and recommend that the paper be reject.

Point 2: Reasoning Capabilities in Manipulation Tasks

First, regarding the definition of "reasoning" ability in manipulation tasks, we, along with Reviewer cjex and Reviewer Ez2J, agree that a VLA model’s capacity to integrate flexible unseen task instructions and generalize to unseen visual configurations in long-horizon tasks while still generating relatively accurate actions can reasonably be interpreted as a form of reasoning in robotic manipulation.

Moreover, in your constructive response on August 2 (lines 10–13), you commented that:
“How should we interpret 'the newly injected diffusion head trained from scratch lacks reasoning capability'? For example, do works like RDT/GR00T, which train a diffusion head from scratch, also lack reasoning capabilities? RDT's paper presents a very interesting phenomenon: when the training set only contains instructions to pour 1/3 and pour a full cup of water, the model can still follow an instruction to pour 2/3. Is this an example of reasoning?”

We completely agree with your point that RDT demonstrates the contextual reasoning ability of DiT-based policy models, using examples such as changing the task instruction from “pour 1/3 cup of water” (seen during training) to “pour 2/3 cup” (unseen). In a similar spirit, we believe our experiments, where we modify task instructions from seen to unseen variants and manipulate objects from seen to unseen ones (belonging to different semantic categories), can also evaluate the model’s contextual reasoning capability. For example, during training, our HybridVLA has never encountered tasks like “pick up the charger/strawberry and put it in the drawer.” Besides, as shown in our rebuttal (W1–2), we conducted an additional long-horizon experiment on the “Open drawer & place inside” task, which not only introduces unseen objects during training but also increases the overall number of steps and atomic subtasks.

Point3: Misinterpretation of Bagel

Thank you for your further comments. We apologize if our response gave the impression that we were discussing Bagel's relevance. Our response aims to clarify that Bagel employs both NTP and diffusion modeling within an unified model simultaneously, which is entirely consistent with our approach. Therefore, we would like to clarify that Bagel serves as supporting evidence for the necessity of an NTP-based method and our concept of collaborative NTP and diffusion generation

As you noted in reference to the Bagel [1] model, “works like Bagel have also shown that an LLM + Diffusion approach can enable the diffusion head to acquire some reasoning capabilities during image generation.” As demonstrated by Bagel and other studies [2], jointly modeling autoregressive and diffusion-based outputs within a unified backbone can both fully leverage the pretrained knowledge of the LLM and enable the diffusion branch to acquire nontrivial reasoning abilities.

This is precisely the foundation of our work. Our goal is to maintain the autoregressive pathway's capacity for action generation while integrating the diffusion-based module to enhance continuous action modeling. Critically, we explore a collaborative training paradigm and token-sequence design that ensures the two branches do not interfere with each other, but instead reinforce one another. More importantly, as shown in the ablation study (Table 3, Ex1 and Ex2), we empirically validate that collaborative NTP-based action generation and diffusion-based action generation during training can enhance the accuracy of the diffusion branch’s action generation during testing.

Finally, we respectfully offer a gentle clarification: Bagel is not a model that combines an LLM with an external diffusion head; instead, it adopts a unified architecture that integrates both autoregressive and diffusion-based generation.

[1] Emerging Properties in Unified Multimodal Pretraining

[2] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

Point 1: Further Clarifying the Core Motivation and Addressing Misinterpretations Regarding Reasoning Abilities in Manipulation Tasks

B. The collaborative "action ensemble" strategy is a post-processing strategy designed to produce more accurate action predictions when multiple candidate actions exist at the same timestamp during testing. We introduce this strategy as a natural extension based on the empirical observation that the two action generation branches (autoregressive and diffusion) exhibit varying strengths across different tasks. This insight led us to combine their outputs adaptively, and we further validate its effectiveness with an 8% improvement in accuracy, as shown in Table 2 of the main paper. Importantly, the action ensemble operates independently of the model’s common-sense reasoning capabilities. Its purpose is solely to refine the final action outputs at inference time, not to enhance or contribute to reasoning. We would like to clarify that at no point in the paper or rebuttal do we claim that this ensemble method has anything to do with common-sense reasoning.

C. Though other reviewers stated these problem during the first round rebutal, they all claim now that we have solved their concerns, meaning these problems no longer exists after our rebuttal. Specifically：

The concern raised by Reviewer RfcC is primarily due to our Figures 1 and 2, as well as insufficient detail in our description of diffusion token modeling. The reviewer initially misunderstands our method as “feeding the entire diffusion chain as tokens into the LLM.” However, once we clarify that only a single noisy sample from the current denoising step is fed into the LLM to predict the corresponding noise, the reviewer responds that this explanation “addressed my major concerns.” Therefore, we believe this is a misunderstanding caused by our unclear presentation, not an issue of mathematical soundness or justification. At the same time, Reviewer RfcC’s comment will help us refine the description of diffusion token modeling.

The "contextual reasoning" concern raised by Reviewer cjex has been addressed through our clearer definitions and the addition of new experiments involving unseen instructions and unseen objects from different semantic categories (long-horizon task). Notably, Reviewer cjex responds that “most of my concerns are well addressed.” We sincerely appreciate Reviewer cjex for raising this important point, as it motivates us to further validate the effectiveness of our method and demonstrate its performance in contextual reasoning within manipulation tasks.

The concern raised by Reviewer Ez2J has also been addressed by providing additional experiments demonstrating the effectiveness of both autoregressive and diffusion-based action generation, along with the corresponding PCA analysis. Reviewer Ez2J responds, “I think this paradigm indeed achieves the goal of a generalist model.” We are also sincerely grateful to Reviewer Ez2J for raising this point, which prompted us to further validate the effectiveness of our motivation and method through PCA distribution analysis.

Finally, we would be deeply grateful if you could consider revisiting your evaluation based on our responses, as well as the rebuttal feedback and positive final ratings from the other reviewers. Meanwhile, our writing may have unintentionally implied viewpoints that we did not intend to express (e.g., as you noted, “implying that diffusion-based VLA methods, which do not use NTP, inherently lack this ability”), which led to your main concern. Please rest assured that we will revise the relevant parts of the manuscript in the revised version to address the issues you pointed out.

[W1]. Contextual Reasoning Comparison with Baseline Methods in Manipulation Tasks

A. Definition of context reasoning for VLA:

Thank you for your constructive comments. In robotic manipulation tasks, contextual reasoning for VLA models refers to the policy’s ability to integrate task instructions and robotic observations in order to generate closed-loop action sequences. HybridVLA improves this by leveraging a unified token sequence and a corresponding collaborative training recipe that integrates the Markovian denoising steps of diffusion into the next-token prediction process, where each denoising step is treated as a reasoning iteration. This design not only enables HybridVLA to support both diffusion-based and autoregressive action generation, but also inherits the reasoning paradigm of large-scale pretrained VLMs.

B. Additional context reasoning experiments:

Following your valuable suggestion, to empirically evaluate contextual reasoning, we conduct two additional experiments, including inference on unseen task instructions and unseen visual observations.

(1) For unseen task instructions, we conduct additional experiments in the RLBench simulator to evaluate performance under diverse unseen task instructions. Each task in RLBench is associated with multiple semantically similar instructions, but during training, both HybridVLA and all baselines are only exposed to the longest version. At test time, we randomly select one of the previously unseen instructions for each rollout. As shown in the table below, although HybridVLA shows a slight performance drop compared to its results in Table 2 of the main paper, the decrease is significantly smaller than that of other VLA baselines. This demonstrates HybridVLA’s more robust contextual reasoning ability for diverse task language instructions.

(2) For unseen manipulated objects, as shown in Section 4.4 and Table 4 of the main paper, we assess HybridVLA’s contextual reasoning ability when faced with unseen observation configurations. To further validate this, we conduct an additional long-horizon experiment on the “Open drawer & place inside” task, in which three objects are placed on the plate, two of which belong to unseen categories during training. Specifically, unlike the visualization in the last row of Figure 5 in the Appendix, we modify the test setting by placing not only an orange into the tray but also adding two unseen objects, including a charger and a strawberry. Since visualizations are not allowed in the rebuttal, a similar experimental procedure can be referenced around the 5:10 timestamp in the submitted supplementary video. As shown in the table below, HybridVLA outperforms other methods in handling novel objects, especially in the long-horizon task, highlighting its ability to reason effectively under unfamiliar visual conditions.

[W2]. Discussion and Comparison with Recent VLA Models [1,2]

Thank you for your suggestion. We will add the discussion and comparison in the revised paper as follows.

A. Discussion with [1, 2]:

Pi0.5 [2] aims to build diffusion-based VLA models with a separate diffusion head, while OpenVLA-OFT [1] similarly adopts an independent MLP head for either regression-based or diffusion-based action generation. However, our HybridVLA adopts a unified LLM to perform collaborative diffusion and autoregressive action generation, which differs from these approaches in both implementation strategies and optimization objectives.

In detail, Pi0.5 extends the modular architecture of Pi0 [13] by adding an action expert module following the VLM, and introduces additional training data with intermediate language instructions. This enables the model to first generate low-level commands and subsequently predict actions conditioned on them, offering greater flexibility in decoupling high-level semantics from low-level execution.

OpenVLA-OFT proposes a parallel decoding strategy with action chunking, which generates continuous actions via the MLP head. This design substantially enhances inference efficiency, enabling the model to operate at high frequencies.

Unlike [1, 2], HybridVLA proposes using a shared LLM as the backbone to construct a unified token sequence that integrates both diffusion and autoregressive action generation paradigms. We integrate the Markovian denoising steps of diffusion into the LLM’s next-token prediction process, treating each step as a reasoning iteration. This results in a new action modeling paradigm in which the two generation mechanisms can mutually enhance each other. The following is a summary table comparing the three methods.

B. Comparison with [1]:

As shown in the table below, since Pi0.5 has not been open-sourced, we conduct quantitative comparisons only with OpenVLA-OFT on the RLBench simulator. Using the same dataset, we follow the official simulation fine-tuning setting provided by OpenVLA-OFT for training. Our method outperforms OpenVLA-OFT in manipulation accuracy by fully leveraging the two generation mechanisms and the pretrained knowledge of the LLM.

[W3]. The selection of parameter θ

To determine the value of the parameter θ (confidence threshold), we have conducted large-scale empirical experiments. As shown in Lines 249–251, we find that for our autoregressive action generation, over 80% of successful trajectories had an average action token confidence exceeding 0.96. Meanwhile, as shown in Appendix B.2 and Table 8 (Lines 792–803), we conduct an ablation study on different threshold values. The results confirm that setting the threshold to 0.96 yields the most effective fusion between the two action branches, leading to more robust robot control.

Moreover, the choice of this hyperparameter remains effective in real-world robot experiments. Since HybridVLA is pre-trained on extensive real-world robotic data, we observe that when the confidence of an autoregressive action token exceeds 0.96, the corresponding action is highly likely to succeed during real-robot deployment.

[W4]. Additional Generalization Experiments in Simulation

Thank you for your valuable suggestion. We additionally evaluate our model in the SimplerEnv variant aggregation setting using the Google robot. As shown in the table below, our method achieves satisfactory generalization performance. However, since other baselines (e.g., Pi0) are not originally trained on the same subset of the Fractal [86] dataset, we conduct consistent fine-tuning across all baselines using the same dataset to ensure a fair comparison. Due to time limitations, we complet only the comparative experiments for Pi0. The remaining results will be included in the revised version.

Finally, we sincerely thank the reviewer for the thoughtful comments and valuable time. We hope our detailed responses address your concerns. If any issues remain, please let us know, and we will respond promptly.

Dear Reviewer cjex,

Thank you for the time and effort you dedicated to reviewing our paper. We have provided point-by-point responses to the raised questions, including the definition and additional experiments on contextual reasoning in manipulation tasks (W1), discussions and comparisons with several recent studies [1,2] (W2), an explanation of the parameter selection rationale (W3), and generalization experiments conducted in SimplerEnv (W4).

As the rebuttal period has now passed the halfway point, we hope our responses have addressed your concerns and would be grateful if you could consider updating the score should you find our clarifications satisfactory. We remain available for any further discussion if there are unresolved concerns. Thank you again for your thoughtful feedback and positive recognition of our work.

Paper 16855 authors

[W1]. The difference between AR and Diffusion action generation

Thank you for your thoughtful comments. The statement in Lines 58–59 is based on empirical findings from the experiment in Appendix B.1 (Table 7), where diffusion-generated actions and autoregressive-generated actions exhibit varying performance across tasks. To provide a more intuitive explanation of this observation, we elaborate from three perspectives: (1) differences in generation mechanisms and conditioning features, (2) a Principal Component Analysis (PCA)-based distribution analysis, and (3) additional experiments on unseen manipulated instances.

A. Differences in generation mechanisms and conditioning features:

Unlike the autoregressive action generation approach, which requires quantizing actions into discrete bins, the diffusion branch can directly generate continuous actions. HybridVLA performs multi-step Markovian denoising within the LLM, treating each step as a reasoning iteration. This process enables HybridVLA to progressively refine action predictions by leveraging the pretrained knowledge of the LLM. As a result, diffusion-based generation is particularly suitable for producing smooth and fine-grained action sequences, which are essential for high-precision manipulation tasks. In contrast, the autoregressive branch can condition not only on task instructions and visual observations but also on the denoised tokens produced by the diffusion branch. These multimodal conditions and diffusion action latent conditions enable the autoregressive branch to reason over a broader context, making it more effective at capturing task-dependent semantic alignment between the environment and action intent.

B. Principal Component Analysis (PCA):

To more robustly evaluate the representational differences between the two generation branches across different tasks, we apply Principal Component Analysis (PCA) for dimensionality reduction and compute the intra-class and inter-class distances in the principal component space. Specifically, we select several trajectories from the Pick actions (real-world single-arm setup) and modify only the manipulated object (e.g., replacing a cube with a charger or a strawberry), while simultaneously updating the corresponding language instruction. We then extract the diffusion-denoised tokens and the autoregressive action tokens, and project them into 2D using PCA to obtain their distributions across three different object variants. We observe that the diffusion-based features show relatively blurry clustering in PCA space across different object categories. In contrast, the autoregressive action token embeddings form tighter intra-class clusters and exhibit larger inter-class margins, indicating greater robustness to the semantic differences in manipulated objects. This result supports our empirical observation that autoregressive action representations capture relatively richer semantic context. Due to rebuttal constraints that prohibit including additional visualizations, we instead report the intra-class and inter-class distances of the two generation paradigms in the table below.

C. Additional unseen instances manipulation experiments:

For additional experiments on unseen manipulated instances, we follow the Pick and Place task setting from the generalization experiment (Section 4.4) using a single-arm robot. Specifically, we modify the manipulated object to previously unseen objects, such as a charger and a strawberry. As shown in the table below, HybridVLA-AR exhibits a smaller performance drop after object replacement. This further supports our claim that the autoregressive branch is more effective at capturing semantic variations.

[W2]. Why is autoregressive discrete action generation essential in HybridVLA

First, diffusion-based and autoregressive action generation exhibit different strengths across tasks, as evidenced by the empirical findings in Appendix B.1 (Table 7). This suggests that the differences are not solely attributable to their output formats (continuous vs. discrete), but also to their fundamentally distinct generation paradigms. Specifically, as discussed in W1.B and W1.C, our PCA-based distribution analysis and additional manipulation experiments both indicate that the autoregressive branch is more effective at capturing semantic variations when handling unseen visual configurations.

Second, the combination of diffusion and autoregressive generation yields a more stable optimization objective. The two generation branches both aim to approximate the same conditional action distribution, leveraging different generative paradigms to model the same observed data. That is, they both seek to maximize the log-likelihood of the training actions. Therefore, introducing autoregressive generation during training promotes more robust action representation learning. Our ablation results (e.g., Ex1 vs. Ex2, Ex3 vs. Ex4) further validate this, showing that jointly trained diffusion and autoregressive branches outperform their independently trained counterparts. This validates that retaining autoregressive discrete action generation within the hybrid objective serves as a mutual regularizer and enhances action modeling. As per your valuable suggestion, in the future version, we will switch from the epsilon prediction paradigm to the v prediction paradigm to explore whether it can lead to more robust collaborative action generation.

[W3]. Action chunking

Good question. There are two ways to incorporate action chunking. (1) We can apply action chunking scheme to the diffusion-based action generation, while the autoregressive branch only predicts the next immediate action. In this setup, the Collaborative Action Ensemble strategy is applied only at the most critical first step, while the subsequent actions are solely adopted from the diffusion-generated action chunk. This approach preserves the entire efficiency of action generation. (2) Additionally, we can adopt parallel decoding to accelerate the generation of action chunks in the autoregressive branch as much as possible. This enables the Collaborative Action Ensemble strategy to operate over the entire action chunk.

Finally, we sincerely thank the reviewer for the thoughtful comments and valuable time. We hope our detailed responses address your concerns. If any issues remain, please let us know, and we will respond promptly.

[W1,Q1,Q2] Clarification on diffusion chain modeling in HybridVLA

A. Clarification of the implementation:

Thank you for the insightful question. We would like to clarify that HybridVLA does not feed the entire diffusion chain as tokens into the LLM. Only one noisy sample from the current denoising step is fed into the LLM to predict the corresponding noise. Below, we provide a detailed explanation of our implementation.

(1) During training, as shown in Lines 177–180, we follow the standard DDPM training scheme by randomly sampling a single denoising timestep and the corresponding noisy action, which are projected into the LLM’s token embedding space.

(2) During inference, as shown in Lines 232–234, HybridVLA adopts a step-wise DDIM denoising process, where at each step, only the current noisy sample is fed into the LLM to predict the noise. The predicted noise is then subtracted from the noisy sample to produce the input for the next step, while the token sequence does not retain any previous noise samples. This design enables multi-step Markovian denoising within the LLM, where each step is treated as a reasoning iteration. In this way, HybridVLA can progressively refine action predictions by leveraging the LLM’s large-scale pretrained knowledge.

Importantly, as you correctly pointed out (“earlier samples just introduce additional variance”), our method does not feed the entire diffusion chain into the LLM, while only processing the current noisy token at each step.

B. Clarification of Figures 1 and 2:

Figures 1 and 2 aim to illustrate that each denoised sample is reused as input for the LLM’s prediction, forming an iterative process. However, the static illustrations may misleadingly suggest that the entire diffusion sequence is input at once. Based on your valuable comments, we will revise both the figures and the corresponding descriptions to clarify the denoising mechanism.

C. Empirical justification:

To further validate our diffusion design choice, we conduct an additional experiment in which we explicitly included earlier noise samples in the token sequence. Note that this process retains only the previously input noisy sample, but does not perform parallel denoising over the entire noise chain. We follow the same collaborative training recipe as in the main paper and report results for HybridVLA under this variant.

As shown in the table below, this design setting leads to performance degradation due to unstable conditioning introduced by earlier noise samples. These results empirically confirm that retaining the entire diffusion chain introduces unnecessary variance and reduces overall model robustness.

[W2]. Different from previous works

Thank you for your detailed comments. In the robotics domain, the joint training of diffusion and autoregressive action generation in our HybridVLA is non-trivial. We carefully design a robotics-specific token sequence to model the contextual information essential for robotic manipulation and make the first attempt to demonstrate that jointly optimizing both generation branches can mutually enhance each other, leading to more robust action generation.

A. Difference with unified VLMs:

Unlike prior works [69,70] that focus on unifying image and text generation, HybridVLA targets a fundamentally different problem that collaborative model continuous SE(3) action for robotic control. Our contribution lies not just in BOD/EOD tokens, but in several key components:

(1) A robotics-specific token sequence formulation is designed to embed diffusion modeling into the LLM’s token sequence, leveraging its pretrained knowledge. This design allows the model to integrate the Markovian denoising steps of diffusion into the LLM’s next-token prediction process. Such integration enables diffusion-based action generation to benefit from the pretrained LLM backbone, treating each denoising step as a reasoning iteration analogous to those in LLMs. Additionally, the diffusion action tokens can provide explicit continuous action latent conditioning for the subsequent autoregressive generation.

(2) A new approach to jointly optimizing unified action representations using diffusion and autoregressive generation. In our HybridVLA, both generative branches aim to approximate the same conditional action distribution, thereby jointly generating distributional representations that align with the action distribution under a shared LLM backbone. This approach differs from general VLMs that generate image and text modalities separately, as it facilitates mutually reinforcing optimization and promotes more robust action representation learning.

(3) A collaborative action ensemble mechanism that adaptively fuses both generation branches at test time. This has not been explored in related work. Specifically, we empirically and innovatively observe that combining the two generation branches not only enables mutual enhancement but also reveals that each branch excels in different tasks or trajectories. As a result, this ensemble mechanism provides a simple yet effective output strategy that leads to more stable control in both simulation and real-world robotic environments.

B. Difference with diffusion-based VLA methods:

Recent diffusion-based VLA methods in the robotics domain, such as Pi0 [13] and CogAct [14], typically attach a separate diffusion head that is trained from scratch after a VLM. These methods use the multimodal latent features extracted by the VLM as static conditions. Meanwhile, existing diffusion-based VLA approaches have not explored the joint optimization of action representations using both diffusion and autoregressive generation. In contrast, HybridVLA is the first to integrate diffusion and autoregressive generation within a unified LLM backbone for collaborative action modeling. It treats each denoising step as a reasoning iteration and designs a robotics-specific unified token sequence to inherit the pretrained reasoning paradigm of the VLM.

Finally, we sincerely thank the reviewer for the thoughtful comments and valuable time. We hope our detailed responses address your concerns. If any issues remain, please let us know, and we will respond promptly.

Dear Reviewer RfcC,

First of all, thank you very much for taking the time to review our paper and for your valuable comments. We have taken your concerns seriously and provided detailed responses. Specifically, we clarified that HybridVLA does not feed the entire diffusion chain as tokens into the LLM. Instead, only a single noisy sample from the current denoising step is iteratively fed into the LLM to predict the corresponding noise. We sincerely apologize for any confusion caused, and we will revise Figures 1 and 2, as well as the corresponding description of the diffusion token modeling, in the final version. In addition, we have provided empirical evaluations to support the effectiveness of both our diffusion token modeling approach and the perspective you raised.

As the rebuttal period has now passed the halfway point, we sincerely hope our responses have addressed your concerns. If you find our clarifications satisfactory, we would greatly appreciate your consideration in updating the score, as it would be a meaningful acknowledgment of our efforts. Should you have any further questions or suggestions, we would be happy to discuss them.

Paper 16855 authors