ChatVLA-2: Vision-Language-Action Model with Open-World Reasoning

W1 The failures of prior models are primarily attributed to perception weaknesses, rather than deficiencies in the diffusion policy’s execution ability.

Traditional end-to-end approaches, due to their tightly coupled internal processes, often make it challenging to attribute failures to perception, reasoning, or execution.

Even methods incorporating intermediate language outputs ("reasoning") can encounter this challenge. The core issue is the need for reliable mappings between perception and reasoning, and between reasoning and execution. Without guarantees on the correctness of these reasoning steps or explicit modules ensuring actions follow the reasoning, definitive failure attribution remains problematic.

In contrast, our proposed method addresses these limitations. By employing a Mixture of Experts (MoE) architecture, our approach effectively preserves the knowledge from pre-trained Vision-Language Models (VLMs) to enable robust reasoning. Furthermore, our reasoning-following module and two-stage training recipe make our model's actions consistent with its internal reasoning.

Given these considerations, we suggest that attributing failures of prior models solely to perception weaknesses is not appropriate.

Q1 Experiments on forcing the correct reasoning is valuable.

We appreciate the reviewer's valuable feedback. We would like to clarify that it is infeasible for our model to explicitly enforcing correct reasoning during inference. This is because the reasoning output (e.g., textually stated digits or bounding boxes) is decoded directly from the hidden state, while the policy head takes these same hidden states—rather than explicit reasoning representations—as its input.

Given this design, it is infeasible to externally impose specific reasoning components on the model during the inference phase. We appreciate the reviewer's understanding of this architectural consideration.

Summary1 The paper calls this an "action expert" but this is confusing because it is not an expert in the MoE of the LLM.

We appreciate the reviewer's comment. To clarify, our "action expert" is an independent action output module, not an expert within an MoE framework. This terminology is consistent with its use in previous works like pi0, pi05, pi05-KI, and DexVLA.

We will include these explanations in the revised version of our paper. We hope this can addresses some of your concerns.

W1 Authors should present evaluation on long-horizon tasks or simulation environments besides the two tabletop pick-and-place tasks.

Thank you for raising these concerns. We address them point‑by‑point below.

1  Lack of suitable simulators

Current robotic simulators are tailored to short‑horizon pick‑and‑place or highly scripted scenes. None supports the open‑world generalization and multi‑step reasoning we study—namely, tasks that interleave vision, language, symbolic problem‑solving, and physical manipulation over several minutes. We therefore conducted real‑robot experiments to expose the full spectrum of perceptual ambiguities, occlusions, and physical uncertainties that a model must handle. If a simulator capable of this breadth exists, we would gladly adopt it and include the reviewer’s reference in a revision.

2  Long‑horizon difficulty—not “simple” pick‑and‑place

Some trial lasts > 2 minutes and comprises dozens of low‑level actions, multiple branching decisions, and recovery behaviors—not a one‑shot grasp. The robot must:

1. Parse instructions stated in natural language.
2. Interpret visual cues (e.g., handwritten formulas, answer cards).
3. Reason symbolically to obtain a solution.
4. Execute and, if needed, correct a manipulation plan in real time.

We also report consecutive‑run success rates, demonstrating that the behavior is reliable rather than cherry‑picked.

3 Why these tasks matter

The experiments were crafted to showcase ChatVLA‑2’s ability to translate high‑level reasoning into action—for example, solving a math problem on a whiteboard and then physically selecting the correct answer card. Such “mundane” but cognitively rich activities reflect everyday human tasks far better than isolated pick‑and‑place benchmarks.

W2 Including additional results on multimodal understanding and fine-grained visual recognition.

We sincerely thank the reviewer for their thoughtful comments and valuable suggestions. We have conducted extensive evaluations across 12 diverse multi-modal understanding benchmarks, covering tasks such as document understanding (DocVQA), chart and scientific reasoning (ChartQA, AI2D), OCR-based question answering (TextVQA, OCRBench), and real-world fine-grained recognition (InfoVQA, RealWorldQA, MMStar).

We also present the results of baseline model ChatVLA, as is shown below:

While ChatVLA adopts a static MoE design that explicitly routes VLM and robot data through separate experts, our method uses a dynamic MoE that jointly trains on both modalities with learned routing. The results demonstrate consistent improvements on 11 out of 12 benchmarks, with particularly notable gains in tasks requiring strong multi-modal understanding capabilities—such as +7.0 on TextVQA, +11.7 on ChartQA, and +6.4 on InfoVQA. These results indicate that key understanding abilities, including fine-grained recognition, OCR, and multimodal reasoning, are retained from the pre-trained VLM.

Q1 A comparison between fine-tuned versions of OpenVLA , DexVLA and the proposed ChatVLA-2 would provide a more comprehensive evaluation.

We would like to respectfully clarify that the baseline models compared in our paper were fine-tuned on robotic data of our tasks, rather than utilizing pre-trained open-source weights. This approach was taken to ensure a fair comparison, which we believe more clearly demonstrates the superior generalization capabilities of our model.

Q2-1 Analyze the additional inference overhead

We appreciate the reviewer's valuable feedback. Compared to the dense 3B model, we observe an ~60% increase in inference latency and an ~40% increase in memory usage, primarily due to the increased number of active parameters as we use 8 experts with Top-2 gating. Given the substantial improvement in out-of-distribution task performance (success rate increased from near-zero to 82.7%) in open-world scenarios, we consider this overhead acceptable. ****Moreover, the inference speed of ChatVLA-2 is 30Hz, which is still faster than conventional VLA method like OpenVLA (5Hz).

Q2-2 Provide the load balancing of MoE after fine-tuning.

We've analyzed the expert usage statistics of our Mixture-of-Experts (MoE) model. The following data details the top-2 experts per layer and their corresponding token counts:

Layer 0: Expert_3 - 981, Expert_5 - 883

Layer 1: Expert_5 - 1129, Expert_4 - 1037

Layer 2: Expert_7 - 1053, Expert_4 - 960

Layer 3: Expert_4 - 912, Expert_2 - 779

Layer 4: Expert_5 - 1047, Expert_2 - 730

Layer 5: Expert_2 - 1051, Expert_3 - 886

Layer 6: Expert_4 - 1097, Expert_1 - 747

Layer 7: Expert_7 - 1157, Expert_2 - 569

Layer 8: Expert_1 - 1187, Expert_7 - 657

Layer 9: Expert_5 - 1002, Expert_0 - 928

Layer10: Expert_0 - 1202, Expert_6 - 825

Layer11: Expert_6 - 1124, Expert_1 - 1030

Layer12: Expert_2 - 905, Expert_7 - 746

Layer13: Expert_5 - 1168, Expert_0 - 733

Layer14: Expert_4 - 889, Expert_2 - 720

Layer15: Expert_2 - 1141, Expert_3 - 844

Layer16: Expert_2 - 1099, Expert_0 - 891

Layer17: Expert_6 - 1028, Expert_1 - 941

Layer18: Expert_3 - 1273, Expert_7 - 904

Layer19: Expert_7 - 745, Expert_5 - 733

Layer20: Expert_4 - 1128, Expert_5 - 704

Layer21: Expert_1 - 1195, Expert_3 - 755

Layer22: Expert_6 - 1071, Expert_0 - 635

Layer23: Expert_1 - 1102, Expert_6 - 832

Layer24: Expert_5 - 1012, Expert_0 - 729

Layer25: Expert_3 - 1065, Expert_4 - 936

Layer26: Expert_4 - 1037, Expert_5 - 641

Layer27: Expert_3 - 947, Expert_2 - 913

Following the Noisy Top-K Gating mechanism[1] , our router uses linear layer with adaptive noise perturbation before taking the softmax function, which theoretically helps with load balancing. The average standard deviation of token counts per layer is approximately 282, and the average top-1/top-8 expert token ratio is 5.27×. The average entropy is 2.79, compared to the theoretical maximum of 3.0.

These statistics provide an overview of expert utilization across our model, and the complete data is included in Table 1 at the end. We hope this can addresses some of your concerns.

Q3 Typo: the “three-stage training process” in abstract

We appreciate the reviewer's careful attention to detail. The methodology employed is the two-stage training process consistently described throughout the main body of the paper. We sincerely apologize for any confusion this oversight may have caused and will rectify it in the final version of the manuscript.

[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER

Table 1 Expert Traffic per Layer

W1 The authors should converts action sequences into symbolic representations and use semantic similarity function to validate the reasoning-following enhancement module.

We argue that constructing a function and using it to measure reasoning-action alignment is not suitable for the robotics domain and could lead to misleading conclusions. However, we agree with the reviewer that evaluating the reasoning-following module is essential. Therefore, we propose an alternative behavior-based evaluation strategy that incorporates module ablation.

1. Mapping actions to reasoning is problematic. The suggested evaluation assumes a function maps low-level continuous robot actions to high-level discrete symbolic reasoning. We argue that this assumption is theoretically invalid in robotics. This is an ill-posed inverse problem, where solutions are not unique, stable, or guaranteed to exist. Meanwhile, low-level action data (continuous joint values) does not contain enough information to uniquely reconstruct the high-level symbolic reasoning, as is called “semantic gap”.

2. We propose an alternative evaluation that does not require mapping actions to symbolic forms. Specifically, we design an intervention experiment in the math-matching game where we swap the positions of the answer card while keeping the math equation constant. The math equation and the initial position of answer card are seen during training, while the distractor cards and the swapped position of answer card remain unseen.

   This setup is designed to achieve two primary objectives. First, by using in-domain equations, we maximize the likelihood that the model generates correct reasoning. This allows our evaluation to focus exclusively on the model's reasoning-following module. Second, the introduction of unseen distractor cards and the dynamic change in the answer's position prevent the model from relying on pre-existing positional biases. Third, any observed performance difference observed before and after the card swap thus directly indicates the model’s reasoning-following capability . We evaluate both the full ChatVLA-2 and the variant that removes the reasoning-following enhancement module under this setup. The results are presented below:

   Setting	Success Rate (Before Swapped)	Success Rate (Swapped)
   Full Model	12/13	10/13
   w/o Reasoning-Following Module	11/13	2/13

   The results show that our full model maintains strong performance even after the card swap (10/13), indicating robust reasoning-following capability. In contrast, the ablated variant's performance drops sharply (from 11/13 to 2/13), highlighting its reliance on positional biases. This validates the effectiveness of the reasoning-following enhancement module.

W2-1 & Q1 The routing policy is not specified. What prevents the router from simply defaulting to the same expert pairs?

Following the Noisy Top-K Gating mechanism[1] , our router uses linear layer with adaptive noise perturbation before taking the softmax function, which theoretically helps with load balancing.

W2-2 No analysis is provided to examine the diversity or specialization of experts.

We argue that we have made an ablation study on number of experts in the Appendix, Table 1, due to page limitation of the main paper. We also analyzed the possible explanation in the Appendix, Section D.1. We hope this can solve some of your concerns!

W2-3 How Static, shared, and dynamic MoEs are implemented?

We appreciate the reviewer's comment regarding the clarity of our ablation setting. We'd like to clarify the distinctions between static, dynamic, and shared MoE as used in our work.

• Static MoE refers to an architecture where different data modalities are assigned to a specific expert via a fixed routing mechanism. Each data modality exclusively utilizes one predetermined expert.
• Dynamic MoE, in contrast, employs a dynamic routing mechanism, allowing individual token—regardless of their modality—to be directed to various experts based on their representations.
• Shared MoE extends dynamic MoE by incorporating an additional, universal "shared" expert that all data modalities traverse, alongside their dynamically chosen experts.

We will include these explanations in the revised version of our paper.

Q2 Beyond the empirical results, is there any principled reason why mixture-of-experts should be particularly suited for this vision-language-action problem?

The primary rationale for employing MoE stems from the fundamental challenge of catastrophic forgetting when developing Vision-Language-Action (VLA) models from pre-trained Vision-Language Models (VLMs). The significant distribution shift between high-level language and low-level robotic action causes severe cross-modal gradient conflict during fine-tuning, as p by continual learning theory.

MoE directly addresses this interference. By dynamically routing inputs to a small subset of experts (parameter subsets), this architecture intrinsically fosters parameter isolation and sparse activation. These capabilities are fundamental to how MoE mitigates catastrophic forgetting by limiting inter-task interference.

Recent theoretical work [2, 3, 4] rigorously supports this, proving MoE's specialization and sparsity intrinsically reduce interference.

Therefore, MoE is a principled choice grounded in its inherent mechanism for isolating updates and protecting old knowledge, rigorously validated both theoretically and experimentally.

Q3 Why shared representations between VLM tasks and robot control?

While multimodal understanding and robotic control operate in distinct output spaces, they fundamentally rely on processing shared high-level semantic representations (e.g., a door handle affords turning to open) and a unified understanding of environmental context semantics (e.g., the cup near a table edge demands careful action).

In particular, both VLM tasks require grounding in similar semantic concepts—such as spatial relations, object affordances, and scene geometry—to reason or act meaningfully. For example, understanding that a mug is typically grasped by its handle, or that only an empty plate can hold food, is crucial for both describing a scene and interacting with it.

Therefore, leveraging these shared semantic representations is not only feasible but also a key pathway towards building more general and capable embodied agents.

L1 Tasks remain relatively simple.

We respectfully disagree with the reviewer’s assertion that these tasks are “simple.” They are demanding open‑world problems that require simultaneous generalization across visual perception, natural‑language instructions, multi‑step reasoning, and robotic control. In addition, each run is a long‑horizon sequence—our demonstration videos exceed two minutes—so the model must maintain coherent plans over hundreds of low‑level actions. To the best of our knowledge, no existing method—including OpenVLA, π0, π0.5, Gr00tN1, DexVLA, or the original ChatVLA—exhibits the scene understanding and cross‑domain generalization needed to solve such tasks without task‑specific training data.

If the reviewer is aware of VLA systems that successfully tackle comparable open‑world tasks by transferring knowledge from vision–language models to real‑world manipulation, we would greatly appreciate the citations and would welcome further discussion.

L2 Real-world applications may reveal additional safety failures.

We fully agree with the reviewer that real-world applications inherently present more unexpected failures than simulated environments, which is the reason why we conduct all our experiments on real robots (Franka and ARX R5). We did observe a certain failure rate as a result, as is shown in Section 4, Table 1 and 2.

While the primary contribution focuses on advancing the model's reasoning and generalization capabilities in open-world settings, we recognize that further exploration of safety guarantees in diverse open-world scenarios is essential. We are looking forward to advancing this in future work.

L3 MoE models incurs additional overhead

We understand the reviewer's concerns regarding additional overhead. Compared to the dense 3B model, we observe an ~60% increase in inference latency and an ~40% increase in memory usage, primarily due to the increased number of active parameters as we use 8 experts with Top-2 gating. Given the substantial improvement in out-of-distribution task performance (success rate increased from near-zero to 82.7%) in open-world scenarios, we consider this trade-off is justified and the overhead is acceptable.

[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER

[2] Theoretical Analysis of Mixture-of-Experts in Continual Learning, ICLR 2025 Spotlight

[3] Theory of mixture-of-experts for mobile edge computing

[4] A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications

Q1 What does the ‘replace’ in line 186-187 mean?

We appreciate the reviewer's insightful question. In this context, “replace” refers to using the reasoning embedding instead of the observation embedding in the latter half of the transformer layers.

Specifically, we factorize the feature embedding of observation into multiple affine layers and integrate it into the transformer blocks. The Adaptive Layer Norm (AdaLN) block in the original ScaleDP can be formatted as

$AdaLN_i(x) = (γ_i(t, o) + 1) · x + β_i(t, o), 1\leq i\leq N$

where x is the input to the layer normalization, N is the number of layers, and γ(t, o) and β(t, o) are the adaptive scale and shift parameters regressed from the embedding vectors of timestep t and observation o.

In our ChatVLA-2, the AdaLN block is formatted as:

$

AdaLN_i(x) = \begin{cases} (\gamma_i(t, o) + 1) \cdot x + \beta_i(t, o), & \text{if } i < \frac{N}{2}; \ (\gamma_i(t, r) + 1) \cdot x + \beta_i(t, r), & \text{if } i \geq \frac{N}{2} \end{cases}

$

where the original observation o is replaced to reasoning r in the latter half layers.

We will include these explanations in the revised version of our paper. We hope this can addresses some of your concerns.

Q2 Why does the conditioning have to be in terms of adaptive layer norm? Could it simply be concatenated or another way through cross attention?

We agree that alternative conditioning mechanisms merit thorough investigation. Our research, however, primarily focuses on improving the model's reasoning-following capability. We propose the conditioning mechanism and demonstrate that such techniques can effectively enable the model to follow reasoning, especially in open-world scenarios**.** We verified our claim through multiple challenging tasks.

Other implementations (e.g., token concatenation or cross‑attention) are certainly possible, but they lie beyond the present scope. We plan to investigate such variants in future work and hope this addresses your concerns.

Q3 Why ablation on stage 2 seems to achieve very low score on OCR & Math?

Thank you for your valuable feedback. The low score for the "stage-2 only" setting is because it trains only on robotic data without freezing the Vision-Language Model (VLM). The primary objective of this ablation is to analyze the significance of co-training with image-text data (i.e., Stage 1) for preserving the VLM's capabilities under identical training conditions.

We apologize for the lack of clarity regarding the ablation setting. We will incorporate these explanations into the revised version of our paper. We hope this clarification addresses your concern, and we appreciate your thoughtful feedback.

Q4 Why we really need a separate stage II? Is it because stage I is not trained long enough so you observed a lot of training instability with your VLM generated reasoning tokens?

We would like to clarify that the training instability is not due to insufficient training in Stage I. Instead, it arises from the conflict training objective: next token prediction for the language component and noising-denoising for robot action generation.

This scenario mirrors multi-task learning, where diverse training objectives can lead to gradient conflicts, as illustrated in existing researches [1, 2, 3, 4]. Similar phenomena have also been reported in concurrent research [5] in robotics, that naive joint training with diffusion-based action experts significantly slows convergence and impairs language grounding.

Therefore, Stage II training is essential. The objective of introducing Stage II training is to mitigate these gradient conflicts between the two tasks, thereby preserving the pre-trained knowledge within VLM and enabling it to effectively guide robot actions generation.

Q5 The difference between our model vs. ChatVLA

ChatVLA is a prior work that can do both multimodal understanding and robot manipulation in a unified model. However, it fails to transfer its multimodal understanding capability to robot manipulation, i.e., help robot to generalize on different task setup. In other words, even if ChatVLA can easily recognize text and mathematical symbols, while is able to complete math formulas when the user asks questions, it cannot turn their mind into physical action, such as pick the answer card on the board.

The main difference is that our proposed ChatVLA-2 can more effectively leverage the knowledge in real-world scenarios and transfer to a actionable movement, such as complete a math problem on a whiteboard. Our ChatVLA-2 demonstrates stronger generalization under open-world scenarios, representing a significant step toward the development of truly generalizable robotic foundation models.

[1] Gradient Surgery for Multi-Task Learning, NeurIPS 2020

[2] Conflict-Averse Gradient Descent for Multi-task Learning, NeurIPS 2021

[3] RotoGrad: Gradient Homogenization in Multitask Learning, ICLR 2022

[4] Recon: Reducing conflicting gradients from the root for multi-task learning, ICLR 2023

[5] Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better, May 28, 2025