DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

Thanks for your sincere advice. We have addressed your concerns below. and all discussions will be included in the revision.

$\large\textcolor{brown}{Weakness 1.1:}$ The limitation of the potential of end2end optimization.

We employ pre-trained models offline to extract dynamic-region masks, depth maps, and high-level cues before training. Because these models are not executed at inference time, they introduce zero extra runtime cost and do not interfere with the end-to-end optimization of our VLA policy.

$\large\textcolor{brown}{Weakness 1.2:}$ Concern of generalization.

These state-of-the-art flow/depth/segmentation foundation models are trained on massive datasets and exhibit excellent generalization; as shown in Fig. 3 in the manuscript, they extract accurate dynamic regions even in unseen simulated scenes.

$\large\textcolor{brown}{Weakness 2: }$ Boundaries of innovation need to be clearer.

Thanks for your suggestion, we will revise the contribution and innovation of our method in the revision. Our contribution is not a new segmentation or depth-estimation model. Instead, we recast the vision–language–action model as a perception–prediction–action model and make the model explicitly predict a compact set of dynamic, spatial and high-level semantic information, supplying concise yet comprehensive look-ahead cues for planning. Additionally, we introduce a block-wise structured-attention mechanism, coupled with a diffusion-transformer decoder, to suppress representation noise from cross-type knowledge leakage and thus enable coherent multi-step action reasoning.

$\large\textcolor{brown}{Problem 1:}$ Implementation details of dynamic region prediction.

Due to space limitations, we have presented the implementation details of the dynamic region prediction in the Feature Extraction section of the supplementary. Specifically, given a sequence of consecutive RGB frames of resolution $H \times W$, we uniformly sample one keypoint every 8 pixels in both spatial dimensions, resulting in $N = \lfloor H/8 \rfloor \times \lfloor W/8 \rfloor$ sampled locations per frame. For each sampled location, we compute inter-frame displacements $(\Delta x, \Delta y)$ by tracking its position across adjacent frames using CoTracker [50,51]. The magnitude of displacement is converted into a scalar speed value:

$$s_{ij} = \sqrt{(\Delta x_{ij})^2 + (\Delta y_{ij})^2}$$

where $(i, j)$ denotes the spatial coordinates of each sampled patch. We then apply a speed threshold $\tau$ (e.g., $\tau = 1$ pixel/frame) to obtain a binary motion mask. To account for small motions and ensure spatial connectivity, we perform a single-pixel morphological dilation, expanding each positive location to its eight-connected neighbors.

Additionally, the following table shows the ablation experiments to verify the effectiveness of this threshold $\tau$.

Analysis: As $\tau$ increases, performance decreases. A higher threshold makes the dynamic‑region detector more conservative and filters out moderate but task‑relevant motions (e.g., early object/gripper movement). This weakens the guidance provided by predicted world knowledge and harms action reasoning.

$\large\textcolor{brown}{Problem 2:}$ Computation time comparation.

We measure the per‑step closed‑loop latency on an NVIDIA RTX 4090. As shown below, we compare a vanilla VLA (OpenVLA[1]), a VLA that uses a copilot model (Susie[99]) to generate a subgoal image for action reasoning, and our DreamVLA, which integrates comprehensive knowledge forecasting and action reasoning into one model.

Analysis: Generally, VLA uses a copilot model introduces extra time-consuming to generate sub-goal images using a diffusion model, which is the main factor of low latency. DreamVLA achieves better performance while enabling a higher control frequency.

$\large\textcolor{brown}{Problem 3:}$ Potential limitations.

Dynamic obstacles moving faster than ~0.2 m/s occasionally break the frozen flow/depth priors, and large deformable objects are not yet handled.

$\large\textcolor{brown}{Problem 4:}$ How semantic features are used.

We visualize the semantic queries, including sam and dino, whose attention maps concentrate on the target objects, which demonstrate that predicting future semantics teaches the robot which objects or regions will matter for the task, providing a high-level context (for example, object identity and affordances) that guides the selection of goals and grasp choice.

Thanks for your sincere advice. We have addressed your concerns below. and all discussions will be included in the revision.

$\large\textcolor{brown}{Weakness 1 \ and \ Problem 1:}$ Too little description about baseline methods.

Due to the page limitation, we provided a concrete description of them in the supplementary material. Here we revise to provide a brief classification and description of the baseline methods.

(i). Our method surpasses Roboflamingo [26], 3D Diffusor Actor [75], OpenVLA [1], RoboDual [101], UNIVLA [103], Robovlm [32], GR1 [100], which directly projects the RGB/depth image to action signals as shown in Fig. 1(a) in the manuscripts.

(ii). Compared to methods that use a copilot model to generate sub-goal images as input, like Susie [99] and CLOVER [102] as shown in Fig. 1(b) in manuscripts, our model significantly achieves more accurate control.

(iii). DreamVLA outperforms approaches like UP-VLA [42], Seer [41], and VPP [40] as shown in Fig. 1(c) in manuscripts, which merge whole sub-goal image foresight into one VLA to take benefits from a more integrated design and joint optimization.

$\large\textcolor{brown}{Weakness 2 \ and \ Problem 2:}$ The analysis of experiments.

1 Failure-Mode Shift

Analysis: DreamVLA eliminates most perception- and planning-related mistakes; the few remaining errors are low-level motor slips, confirming that the predicted dynamic, depth and semantic cues effectively close the perception–reasoning gap.

---

2 Task Length Matters

Analysis: The advantage grows with horizon length—tasks 4-5 involve long-term spatial reasoning, where comprehensive knowledge offers more gain on long-horizon tasks. We analyze that there are three reasons:

(i). Predicts dynamic-region masks for future frames, so the Transformer keeps explicit attention on pixels that will move, rather than re-detecting them each step.

(ii). Predicts a coarse future depth map, giving the policy a look-ahead of where free space/obstacles will be.

(iii). The comprehensive knowledge fuses spatial and semantic cues into a shared latent, letting gradients flow from the final action back to the earliest relevant pixels.

$\large\textcolor{brown}{Weakness 3 \ and \ Problem 4:}$ Latency Description.

The following table lists the concrete latency of each component in DreamVLA on an NVIDIA GeForce RTX 4090:

Analysis:

(i). Auxiliary modalities add a small overhead. Our “dream queries” are token‑level predictions (no explicit image decoding and no external models). The incremental cost is 3 ms (3.4%) compared to w/o dream query (91 ms vs. 88 ms).

(ii). Main latency comes from the action head, not the auxiliary cues. The 10‑step action head contributes ~60 ms. This part scales roughly with the sampling steps and model size; in practice, we can reduce steps or use a smaller DiT variant, or adopt a faster sampler strategy or use flow policy [1]/mean flow [2,3] to further cut this cost.

(iii). Knobs to trade accuracy for speed (when needed). If an application is extremely latency‑sensitive, we can trim dream queries (e.g., keep only dynamic regions, or dynamic+depth), and/or increase action chunking/run the action head asynchronously to amortize computation without changing the observation path.

[1]. Zhang Q, Liu Z, Fan H, et al. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(14): 14754-14762.

[2]. Geng Z, Deng M, Bai X, et al. Mean flows for one-step generative modeling[J]. arXiv preprint arXiv:2505.13447, 2025.

[3]. Sheng J, Wang Z, Li P, et al. MP1: Mean Flow Tames Policy Learning in 1-step for Robotic Manipulation[J]. arXiv preprint arXiv:2507.10543, 2025.

$\large\textcolor{brown}{Weakness 4:}$ Code Access.

We will release the code.

$\large\textcolor{brown}{Problem 3:}$ Comparasion with Pi0-Fast.

Analysis:

(i).Pi-0-Fast leverages a powerful language backbone (Paligemma-3B) and pre-trains on a much broader dataset, whereas our current model is pre-trained only on LIBERO-90 due to time constraints. In this case, DreamVLA still surpasses Pi-0-Fast on most tasks, especially complex and long-horizon tasks (LIBERO-LONG +29.3). Predicting compact future knowledge(dynamic region+depth+semantics) lets the policy maintain spatial-temporal awareness, whereas Pi-0-Fast must re-solve perception at every frame—small errors compound and success degrades.

(ii). DreamVLA’s design is especially effective when a task requires temporal reasoning and scene updates, while Pi-0-Fast excels at short, isolated grasps thanks to its larger backbone. These complementary strengths suggest future work: combining DreamVLA’s future-knowledge head with a stronger LLM may yield the best of both worlds.

Thanks for your sincere advice. We have addressed your concerns below, and all discussions will be included in the revision.

$\large\textcolor{brown}{Weakness1.1:}$ The extended LIBERO experiments.

Analysis: Due to time limitations, we only pretrain DreamVLA on LIBERO-90, and other models [1][9][31] pretrained on larger datasets, such as Open x-embodiment [8]. In this case, our DreamVLA still surpasses other methods on most tasks, especially complex (LIBERO-GOAL +4.9), long-horizon (LIBERO-LONG +34.0) tasks and average performance (+14.5), demonstrating that future world knowledge prediction could benefit robot manipulation in various tasks.

$\large\textcolor{brown}{Weakness1.2:}$ Whole Image vs. Dynamic Masks in real-world?

To test this, we ran a real-robot ablation on real experiments and two additional, contact-rich tasks (wipe board, stack bowls) and compare full-image reconstruction vs. dynamic-region prediction:

Analysis: Simple tasks (pick and place) show no meaningful gap, but complex spatial-temporal tasks such as drawer manipulating, wiping, and stacking, favour dynamic masks by 5–20 percentage points. We attribute two factors:

(i). Higher signal-to-noise. Gradients focus on pixels that actually move, not static background.
(ii). Data efficiency. Mask supervision is ≈80 % sparser, so each sample delivers more task-relevant signal.

These real-world results, together with our CALVIN gains over Seer [41], indicate that “predict-everything” is not automatically more robust; in complex manipulation, it can dilute learning, whereas dynamic-focused prediction transfers better to physical robots.

$\large\textcolor{brown}{Weakness1.3:}$ The ablation study of each prediction target.

We conduct an ablation study to investigate the effectiveness of each prediction target in CALVIN ABC->D benchmarks, where Dynamic, Depth, SAM and DINO represent only one knowledge prediction. Next, we perform an ablation study (All-X), where we remove one knowledge signal at a time to evaluate its contribution:

---

Analysis:

(i). Dynamic regions prediction is most important (3.64->4.32), because these masks explicitly flag the pixels that are about to change and therefore align almost perfectly with the policy’s action semantics.

(ii) Predicting single depth, DINO, or SAM features alone actually hurts performance. Dynamic masks align with the control objective, providing clear, task-relevant gradients. In contrast, depth and high-dimensional feature losses are noisy and dominate training, diluting useful signals and dragging the backbone below the baseline.

(iii). Why “DINO-only” hurt ( 3.64->3.13)? Although several works [1][2] claim that future DINO features forecasting would benefit intermediate action reasoning, they all use two separate models. DreamVLA, in contrast, learns both visual reasoning and action generation in one backbone; a pure DINO loss drives the backbone to match appearance embeddings instead of action cues, so performance drops (–0.51). DINO is helpful as an extra signal, not as the sole objective in our unified pipeline.

[1]. Zhou G, Pan H, LeCun Y, et al. Dino-wm: World models on pre-trained visual features enable zero-shot planning[J]. arXiv preprint arXiv:2411.04983, 2024.

[2]. Huang Y, Zhang J I, Zou S, et al. LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation[J]. arXiv preprint arXiv:2505.11528, 2025.

$\large\textcolor{brown}{Weakness2:}$ The expanded related work.

Following the reviewer's suggestion, we reorganize the related works by classifying these methods according to the predicted target:

More advanced solutions couple forecasting with control by requiring the policy to produce, in addition to actions, explicit predictions about the future. Concretely, these works ask the policy to output (i) high-level subtask/option sequences or language plans that decompose long-horizon goals [1,2,3], (ii) latent future embeddings / latent actions that compactly encode forthcoming motor intentions[4], (iii) whole sub-goal images or short visual rollouts that anticipate how the scene should evolve[5,6], and (iv) object-centric signals (e.g., bounding boxes) that capture manipulation-relevant dynamics[7,8].

[1]. Lin F, Nai R, Hu Y, et al. OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning[J]. arXiv preprint arXiv:2505.11917, 2025.

[2]. Zhou Z, Zhu Y, Zhu M, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-action model[J]. arXiv preprint arXiv:2502.14420, 2025.

[3]. Shi L X, Ichter B, Equi M, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models[J]. arXiv preprint arXiv:2502.19417, 2025.

[4]. Bu Q, Cai J, Chen L, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems[J]. arXiv preprint arXiv:2503.06669, 2025.

[5]. Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. Int. Conf. Learn. Represent. (ICLR), 2024.

[6]. Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025.

[7]. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, DannyDriess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054,2025.

[8]. Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang,Taoyu Yang, Xuheng Zhang, Heming Cui, et al. Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.03233, 2025.

$\large\textcolor{brown}{Weakness \ 3:}$ The typo of "transformer".

We will revise this typo in the manuscript.

$\large\textcolor{brown}{Problem1:}$ Why do not use SigLIP or DINOv2?

Employing a more powerful visual backbone is orthogonal to our contribution. Therefore, we kept the visual encoder identical to Seer’s MAE [41]. Using the same MAE backbone eliminates confounding factors when we compare against Seer; gains can be attributed purely to our comprehensive knowledge prediction mechanism. We could integrate the SigLIP or DINOv2 encoder in the future.

$\large\textcolor{brown}{Problem2:}$ The concern of batch size.

We apologize for the confusion: 8 is the per-GPU batch size; with 8 A100-80 GB GPUs, the global batch size is 64.

Dear Reviewer Pixv

The author-reviewer discussion period will end in August 6. Please read the rebuttal, post any remaining questions or comments, and confirm the Mandatory Acknowledgement.

Best regards, Lin

Thanks for your sincere advice. We have addressed your concerns below. and all discussions will be included in the revision.

$\large\textcolor{brown}{Weakness 1 \ and \ Problem 1:}$ The concern of success of the reconstruction.

The dream query is internally decomposed into four sub-queries—dynamic region, depth, and semantic query, whose embeddings respectively predict future knowledge for each modality. We have presented visualization results as shown in Fig. 1 and 2 in the supplementary material. Although supervision is applied only to dynamic regions, DreamVLA can reconstruct semantically meaningful representations of the entire scene. This surprising generalization can be attributed to two factors.

(i). The robot arm is in constant motion and frequently interacts with various objects, causing most task-relevant regions to become dynamic at some point in time. This ensures that a large portion of the scene is eventually observed under dynamic supervision.

(ii). Although static regions are not explicitly supervised, the input frames inherently contain global visual context—including background structures, object appearances, and spatial layout, which the model can leverage to hallucinate and complete missing details.

As a result, DreamVLA implicitly learns to integrate temporal dynamics with static priors, leading to coherent and accurate predictions beyond the explicitly labeled regions.

$\large\textcolor{brown}{Weakness 2 \ and \ Problem 2:}$ The prediction step size N.

The number of future actions predicted (N) is treated as a hyperparameter. In our default configuration, we keep N = 3 following the setting adopted by Seer[41]. We add the ablation study as shown in the following table:

Analysis: Varying the prediction horizon from 1 → 5 steps changes the average chain length by ≤0.03, i.e., well within noise. We feed only the current predicted action to the simulator rather than an ensemble of future actions, so the model is largely insensitive to N. We therefore keep N = 3, following Seer [41], for all main experiments.

Dear Reviewer VAuG

The author-reviewer discussion period will end in August 6. Please read the rebuttal, post any remaining questions or comments, and confirm the Mandatory Acknowledgement.

Best regards, Lin