We greatly appreciate the reviewer’s recognition of our work and the constructive feedback provided. We have thoroughly reviewed your comments and address each point with detailed responses below.

Q1: While EmerVerse demonstrates impressive engineering efforts and strong empirical results, its technical novelty is limited as it primarily combines existing techniques—video diffusion, multi-view rendering, and 4D Gaussian Splatting.

We thank the reviewer for acknowledging our engineering efforts and strong empirical results. However, our work goes beyond a simple combination of existing components. Our central contribution is the introduction of a video-generative robotic world model that leverages video prediction to model the robot’s future visual latent space. This model is pretrained on large-scale robotic data, enabling effective domain adaptation from general video generation to robotic world.

The learned latent space forms the basis for policy learning. By attaching a lightweight, plug-and-play action head, we map the visual latent representation to actionable motor commands. This results in a unified framework that unifies robot sense and policy learning within a coherent pipeline, purposefully designed for real-world robotic manipulation. Each module is carefully introduced to support robotic world modeling, rather than assembled through ad-hoc integration.

Q2: While EmerVerse demonstrates strong empirical performance, its complexity and computational inefficiency raise concerns about practicality and novelty.

We thank the reviewer for acknowledging that our framework could achieve strong empirical performance. However, we have to point out the 4DGS is not used for our data flywheel components which is generally an offline procedure. And during policy inference, the video generation process is not necessary. Instead, as stated in Line 144, we only use the feature vector after the first video denoising step and this feature vector is then cachaed and used as the condition feature vector for multiple action denosing steps. The action is predicted with the action chunks (chunk size is 8 in our setting, means we predict 8 step actions from one inference). We achieve 8-step action prediction within 280 ms on a real-world robotic platform equipped with an NVIDIA RTX 4090, which is comparable to the current mainstream VLA model, which is also generally about 200ms - 350ms （for example, OpenVLA[1] is about 180ms, DexVLA[2] is about 310ms）. Furthermore, we demostrate its effectivess with our real-world robotic manipulation tasks.

Q3: Lack of Baseline Comparisons: The paper notably omits comparisons with recent state-of-the-art 3D-aware video generation methods like Gen3C...

We thank the reviewer for this suggestions. The concurrent work Gen3C [3] is open-sourced in GitHub on about Middle June, which is already after the submission deadline of the conference. Moreover, the goals of embodied video generation differ substantially from those of general video generation. Embodied settings place greater emphasis on scene consistency and task-relevant motion, which are not well captured by conventional video generation metrics. Due to the lack of multi-view baselines, we instead compared against SOTA single-view video generators: Kling-1.6, Hailuo I2V-01live, and Open-Sora-2.0. These models were conditioned on the same textual and visual prompts as our method to generate robot-centric videos. We then conducted a human preference study involving 20 annotators, who rated the outputs across dimensions such as scene consistency and object motion. Our method received 75 scores overall, outperforming Kling (64), Hailuo (30), and Open-Sora (1). Besides, in Gen3C, the camera moves around the target object, and the target object is generally with few movements, which differs greatly from our robotic manipulation tasks, where we use static cameras and robotic arm moves. We will add further discussions and comparsions in our revised versions.

Q4: How much does each component (sparse memory, multi-view, 4DGS) contribute to performance?

Thank you for this question. We’ve carried out ablations to isolate the impact of each component:

• Sparse Memory: As shown in Figure 7 (video rollouts) and Table 4 (policy learning), adding sparse memory substantially improves long-horizon generation quality and yields a significant boost in task success.
• Multi-view: Table 2 compares single-view versus multi-view rollouts on LIBERO: incorporating multi views reduces raises success rates, especially on the LIBERO-10 splits.
• 4DGS: We provided some inintal visualization in the Appendix, Figure 14. We also conducted some qualitative analysis， and finding that 4DGS would enhance the cross-view geometric consistency, and dramatically reduces diffusion-induced hallucinations (especially under robotics arm - object occlusions). We will enrich the revised manuscript with additional visualizations of 4DGS’s effects and deeper analysis.

Q5: How sensitive is performance to different camera poses?

We appreciate the reviewer's insightful question regarding the sensitivity of our method's performance to varying camera poses. To address this, we would like to highlight two key points that demonstrate the robustness of our approach against camera pose variations: Firstly, in the LIBERO benchmark, the front view camera mounted positions differ across different task scenarios. Furthermore, as noted in Lines 207 and 225, our policy evaluation leverages both the LIBERO and CALVIN benchmarks. Notably, the camera poses in these two benchmarks are fundamentally different. The consistent performance of our method across these distinct setups directly validates its generalization capability to camera pose variations. Secondly, in our real-world experiments, we intentionally avoided labor-intensive camera calibration throughout the multi-day evaluation process. Instead, we relied solely on the default parameters provided by the robot's hardware client, which inherently contain non-negligible noise. The stable performance observed under such uncalibrated conditions further confirms the robustness of our method to practical camera pose inaccuracies.

References

[1] OpenVLA: An Open-Source Vision-Language-Action Model

[2] DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control

[3] Gen3c: 3d-informed world-consistent video generation with precise camera control

We sincerely thank the reviewer for recognizing the contributions of our work and for providing constructive comments. We have carefully considered your suggestions and provide detailed, point-by-point responses below.

Q1: further real-world robotic experiments are necessary.

We thank the reviewer for this valuable suggestions. In the current submission version, we conducted real-world experiments on the tasks Block Placing, Plastic Objects Sorting, and Fruit Sorting as shown in Section A in the Appendix. And we are trying to add new real-world experiments including Bin Picking， Package Sorting, etc. The visual results will be updated in our revised version paper.

Q2: Clarify why they claim that “wrist-mounted cameras complicate policy learning”?

We thank the reviewer for this questions. This statement is coming mainly from our setting, where the input for the model is the text prompt c, observation images o and the related inaccurate camera parameters. Those camera paramters would serve as the condition information for the multi-view generation diffusion process, then the denoised feature vector will guide the final cation planning, e.g. the pose of the EEF. If we feed the images from the wrist camera, we also need to input its camera parameters. However, its extrinsic parts has directly replected the EEF's pose, which is slightly violating our settings. Besides, in our practical training experiments, we found that when we use both third-view images and wrist images for the training process, we would result in degraded generation performance, and the final policy performance is also suboptimal. As a result, our framework is more suitable to receive multi static view image input, similar to [1]

Q3: Does multi-view generation induce severe hallucinations in long-sequence tasks, and if so, does this directly degrade action generation quality？

We appreciate the reviewer’s concern. In fact, our multi‐view generator is built with cross‐view attention and trained on synchronized multi‐view video data, which strongly enforces geometric constraints and thus reduces hallucinations consistent with prior work on multi‐view consistency[1,2]. As shown in Figure 6, our model achieves high consistency across views, confirming its cross‐view coherence even over long sequences. Crucially, this consistency translates into better action generation: on the LIBERO-Long splits, a policy using only single-view S-RGB rollouts achieves 73 % success, whereas augmenting with two additional rendered views boosts success to 80 %. The improvement is largest compared with other splits, demonstrating that our multi-view conditioning enhances (not degrades) downstream control performance by mitigating hallucinated dynamics. Furthermore, in practice, unlike purely video generation setup,

References

[1] RVT: Robotic View Transformer for 3D Object Manipulation

[2] Zero-1-to-3: Zero-shot One Image to 3D Object

[3]GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image

We appreciate the reviewer's thoughtful feedback. We have thoroughly addressed the concerns regarding methodological clarity and experimental coverage in the responses below, and note that some supporting details are already included in the appendix. We will revise the main paper to improve clarity and ensure all aspects are clearly presented.

Q1: More Training details

We thank the reviewer for this valuable question. And we provide the training details and hyperparameters in the following table, and this table will be added to the main paper in the revised version.

Q2: Details on the dataset

We sincerely thank the reviewer for this question. And we provide the pre-training dataset details as follows. And we will add these information in our main text in the revised version.

Q3: context variable c

In Line 89, we descibe the c as the textual prompt. In our implementation, this textual prompt is encoded by a frozen T5-base and an MLP is used to project the output to the token space.

Q4: Details on EnerVerse-D implementation.

After pretrain the EnerVerse-G, it is capable to receive m-view images (with the camera paramters), task prompt and then produce the predicted m-view videos. Following the standard diffusion process, we add noise to the latent from all m-view images. However, in the data flywheel setting, we have at-least one (N) view camera mounted on the robot has complete and clean visual observation images, and we did not need to predict the future frames for this view and thus NO noise is appied on the correponding latents. Instead, those visual observation will serve as a guidance for other views video generation. We finetune our base model to let it capable to receive such input format. In view of 4D latent space with dimension of CVTHW, for tensor coming from the camera images input, e.g. C1THW, we did not apply noise.

During inference, we use clean N-view camera images and their camera extrinsics to obtail M-view rendered images with the depth wraping, as shown in Figure 4. These M+N view videos with text prompt c then go through the diffusion model to obtain the cleaner denoised M-view images. After that, these M+N view videos and their cooresponing camera parameters are used to construct the 4DGS representation. We use the officaial implementation of 4DGaussian [1], and followed most default settings, the only change is that we use the depth information to do the initialization process and the depth of deformation model is set to 1. The 4DGS model produces higher-precision, geometrically consistent 4D representations. Then, we again render M-view images from the 4DGS to obtain higher-quality observations. These M+N video videos then agian go through diffusion process and 4DGS process to iteratively improve the quality.

Q4: The submission does not contain further elaboration as to the size of the simulated dataset or the diversity of the generated scenes.

We thank the reviewer for this valuable suggestions. The reason why constructed that simulated dataset is that, At the time of this work, all available embodied datasets provided only single third-person camera views, which are insufficient for multi-view generation tasks. We provided visual examples for the simulator collected data in Appendix Figure 13, and this dataset is with about 40K episodes. The collected episodes is with 8 tasks, and crossing industrial and home scenarios. The task descriptions are as following, and we plan to release some splits in the near future to enable other researchers could independently reproduce this training setup.

• Task Descriptions:
  1. place trash into dustbin
  2. pick fruit into basket
  3. pick toy into box
  4. insert pen
  5. place bag
  6. open drawer
  7. place fruits
  8. arrange workpieces

Q5: Symbol defination in Line 86 and Line 93.

The J in Line86 describes the frame number of rendered images. We used the J for rendered views $r_t^{1:J}$ instead of K used for $o_t^{1:K}$ is that the rendered views and the obseravtion camera images could have different frames. For an example, in the EnerVerse-D, the camera obervations have complete offline videos, in such case $K=T$, while the rendered views could only have one frame.

We follow [2] to implement the v-prediction. Instead of predicting the noise $\epsilon_t$, the model predicts $v_t$, defined as: $v_t = \alpha_t \epsilon_t - \sigma_t x_0$.

Q6: Is it possible to condition the world model on actions?

We thank the reviewer for this insightful question. While our current framework focuses on generating videos and subsequently predicting action policies from text instructions and image inputs, we believe it is easy to adapt to support action-conditioned video generation. Although this is not the primary setting explored in our work, the extension is conceptually straightforward: if physical action sequences can be represented in text or visual modalities, they can be directly integrated into our framework, enabling an action-to-video generation process.

Moreover, we view our framework's ability to ground text prompts and visual observations as a foundational capability for building action-conditioned video models. We consider this a promising direction [3] and plan to explore it in future work.

Q7: Is it possible to use the world model to generate synthetic training data for the policy?

Thank the reviewer for this valuable questions. It is a promising and active research domain to use world model to generate synthetic training data for the policy learning as done ine [4], where the model generates both video frames and action lables. For this work, we chose to focus more on the action output, e.g. the robot policy learning.

References

[1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

[2] Progressive distillation for fast sampling of diffusion models

[3] 1X World Model: Evaluating Bits, not Atoms

[4] DreamGen: Unlocking Generalization in Robot Learning through Video World Models

We thank the reviewer for their valuable comments. We have addressed the concerns on methodological clarity and experimental completeness in detail below, with some additional information already included in the appendix. We will revise and polish the main paper to ensure clarity.

Q1: velocity term

As mentioned in L93, we follow [1] to implement the v-prediction. Instead of predicting the noise $\epsilon_t$, the model predicts $v_t$, defined as:

$v_t = \alpha_t \epsilon_t - \sigma_t x_0$

Here, $\alpha_t = \sqrt{\bar{\alpha}_t}$ (signal scale) and $\sigma_t = \sqrt{1 - \alpha_t^2}$ (noise scale), consistent with the forward process equation $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$.

Q2: 4D Model Details

Due to space limitations, we moved the model architecture details to Appendix L591–L604. Additional details are as follows:

• Input and Output: The model takes as input a task prompt c, $m$-view observation images($m \ge 1$)$O_t$, and their camera parameters, and outputs the predicted $m$-view videos.
• Loss: We use the v-prediction diffusion loss to supervise $m$-view image generation, as noted in Q1.
• Dataset: As stated in L155–L158, we use both single-view academic datasets and self-constructed multi-view datasets from the simulator. For single-view videos, camera extrinsics (relative to the robot base) are manually and approximately estimated; for simulator videos (examples in L606), GT parameters are available. During inference, we use the extrinsics provided by the robot hardware. We plan to release the simulator rendering scripts and estimated extrinsics.

Q3: Mechanism on data flywheels.

After pretraining the 4D base model (e.g., EnerVerse-G) on the dataset mentioned in Q2, the model takes as input multi-view images (with camera parameters) and a task prompt, and outputs predicted multi-view videos. Following the standard diffusion process, we add noise to the latent features from all views. However, in the data flywheel, at least one camera (n ≪ m) mounted on the robot provides full visual observations. For these views, we skip noise injection and instead use their latents as guidance for generating other views. We refer to these as sparse observations.

To support this input format, we fine-tune the base model. In the 4D latent space of shape CVTHW, no noise is applied to latents of the n observed views (e.g., C₁THW). During inference (see L153–Fig.4), for examlpe, we use one full video, two rendered initial frames, three camera parameters, and a task prompt. The model then generates denoised videos for the two target views. These, along with the original observed video and parameters, are fed into 4DGS to construct the 4D Gaussian representation. Then rendered frames are obtained from GS, then iteratively.

Q4: Policy Model Trianing.

After obtaining EV-G (L139), we attach a policy head $h\_\theta$, composed of stacked DiT blocks, to enable action generation, resulting in EnerVerse-A. The goal of EnerVerse-A ($f\_\theta$) is to estimate the clean action from a noisy one:

$a_t^0 \leftarrow f_\theta(c, O_t, a_t^k, k) = h_\theta(E, a_t^k, k),$

where the latent vector $E$ is extracted fromthe U-Net backbone in EnerVerse-G. We train $f_\theta$ by minimizing the denoising mean-squared error:

$$\mathcal{L}(\theta) := \text{MSE} \left( a_t, f_\theta(c, O_t, \sqrt{\bar{\alpha}^k} a_t + \sqrt{1 - \bar{\alpha}^k} \epsilon, k) \right), \tag{2}$$

where $k \sim \text{Uniform}({1, \ldots, K})$. As described in L146, we predict a chunk of actions in one shot to improve temporal consistency and reduce error accumulation by decreasing decision frequency.

During inference, $f_\theta$ takes O\_t\ and $c$, which pass through the video diffusion backbone once to compute the latent vector $E$. This $E$ is cached and reused across multiple denoising steps for action generation, and no past actions are used.

Q4: DC for long-video generation.

In L183, we primarily adopt FreeNoise [3], a tuning-free paradigm for extending video length via noise rescheduling. Its core mechanism involves local noise shuffling, constructing long-range correlated noise sequences, and applying window-based attention fusion. In our implementation, the second inference stage takes the results from the first stage as input, and FreeNoise is applied at each inference stage.

Q5: Does DynamicCrafter get the advantage of the same training data as EnerVerse?

For a fair comparison, the EnerVerse reported in Tab 1 and Fig 5 is trained solely on RT-1. Based on the generation results, we attribute the performance gains primarily to our model’s architectural design.

Q6: DC the only video baseline

Our work targets world models for embodied agents on single-GPU or edge-GPU platforms, where video generation is not the core focus. While recent video diffusion models (e.g., I2VGen-XL, Stable Video Diffusion, LivePhoto, PhysGen) achieve high visual quality, they are often unsuitable for our setting—being too large (I2VGen-XL), reliant on extra components like simulators (PhysGen), or closed-source (LivePhoto). In contrast, DynamiCrafter is a well-maintained open-source model with public weights and serves as the basis for AVID. Considering our compute and latency constraints, we selected DynamiCrafter for reproducibility.

Similar to prior works such as VPP[4] and VidMan [5], which adopt a single video model due to high training cost, we additionally selected LTX [6] for further analysis because of its efficiency and favorable inference speed for robotic tasks. We report PSNR results showing EnerVerse-LTX (27.8) > EnerVerse-DC (26), likely benefiting from LTX’s advanced architecture and higher-quality pretraining, and demonstrating the flexibility of our method to different video models. These results and the discussion of additional models will be included in the revised paper.

Q7: No other baselines...(Multi-view)

The availability of open-source multi-view video generation models with publicly released weights is limited. Most SOTA methods are either closed-source, require substantial computational resources for fine-tuning, or depend on bespoke rendering pipelines, making them impractical as baselines under realistic constraints.

Furthermore, the objectives of embodied video generation differ significantly from general video synthesis. Embodied scenarios demand high scene consistency and task-relevant motion—factors not well captured by conventional evaluation metrics. Given the lack of suitable multi-view baselines, we compare against SOTA single-view generators: Kling-1.6, Hailuo, and Open-Sora, under identical textual and visual conditions. In a human preference study with 20 annotators, EnerVerse received 75 votes, outperforming Kling (64), Hailuo (30), and Open-Sora (1).

While EnerVerse may be less competitive in general-purpose video generation, it is more aligned with the requirements of embodied video synthesis. Our primary objective is to validate the utility of multi-view prior synthesis for robotic policy learning. Accordingly, we present qualitative results to establish feasibility and leave more comprehensive quantitative evaluations—including those involving real-robot policy improvements—for future work as more suitable baselines become available.

Q8: assumption better video ... better robotic task performance (L23)

To clarify, in L23 we do not assume a naïve one-to-one mapping whereby any uplift in general video-generation metrics (e.g. FVD, SSIM) will automatically yield better robotic control. Rather, our claim is that representational alignment—i.e., adapting the video model’s latent space so that it encodes 3D, action-conditioned dynamics brings gains.

Q9: not report standard deviations

The selected baselines, such as GR-1 and SUSIE, did not report standard deviations. To ensure a consistent comparison, we also omitted them in the main paper. Here, we provide results comparable to OpenVLA.

Q10: use different visual inputs in Tab 2/3

Our method with a single S-RGB input outperforms all other approaches, including those using S-RGB or S-RGB + G-RGB, demonstrating its effectiveness. Current most visual-policy or VLA models do not support depth input, and our current pipeline requires a static camera for G-RGB.

Q11: baselines across Tab 2 and 3 are different.

In policy learning, CALVIN and LIBERO are two representative large-scale benchmarks, and most prior works evaluate on only one of them. To better demonstrate the effectiveness of our method, we conduct experiments on both.

Q12: quantitative results and comparisons to a subset of baselines in the real-world

In Tab 6(L531), we provided the quantitative results for ours and OpenVLA.

Q13: The impact of GS

We provide the visualization results in the Appendix. The main goal of GS is to enhance the geometric consistency and reduce video diffusion hallucinations (especially under robotics arm - object occlusions). We will provide more visual examples and deeper analysis in the revised version.

Q14: No baselines that leverage multi-view.

In Tab 2, we provide the MAIL with two S-RGB results, and many baselines with S-RGB+G-RGB. MVWM focuses more on the masked autoencoder, while RoboHorizon emphasizes more on the LLM's usage.

[1]Progressive distillation for fast sampling of diffusion models

[2]Learning fine-grained bimanual manipulation with low-cost hardware

[3]Freenoise: Tuning-free longer video diffusion via noise rescheduling

[4] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

[5] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

[6] LTX-Video: Realtime Video Latent Diffusion

I thank the authors for the follow-up experiments and responses. I have three remaining concerns:

• paper requires drastic presentation changes
• no comparisons between 3D video vs. 4D space generation for robotics
• lack of quantitative ablations for 4DGS

I've detailed the points below. As of this moment, I'm willing to raise my rating to borderline reject. However, I remain firm in my opinion that this paper needs significant presentation changes (and some further experimental changes) for acceptance.

Addressed concerns

What is EnerVerse-LTX? Why is there no comparison to pure LTX without any of the contributions from this paper?

The new updated LTX results are helpful. This concern is addressed. Thank you.

---

CALVIN vs. LIBERO mismatch in baselines

I appreciate the new results of OpenVLA on CALVIN. Thanks. This concern is addressed.

---

Remaining concerns

Paper requires drastic presentation changes

TL; DR --- What would the paper look like after incorporating feedback? Not sure

I appreciate that the authors have agreed to fix the clarity concerns. While the technical details of the approach may remain the same, the presentation will have to be significantly different compared the original submission. For example, for some papers, authors could address this by saying "subsection XYZ will be rewritten as follows ....". However, here the entire Section 2 needs to be rewritten, potentially expanding the space requirement in the paper by 1-2 pages. That would require altering the remaining sections of the paper to accommodate this increased spacing requirement. I'm not comfortable accepting any paper without having a clear picture of what exactly the contents of that paper would look like.

---

No comparisons between 3D video vs. 4D space generation for robotics

TL; DR --- What is the value of the core contribution of 4D space generation? How much does it help robotics?

I understand and agree with the authors arguments about better video not implying better robotic performance. However, that was not my follow up question. My question was "why not compare DC-FN and LTX on the robotic control tasks directly?". I'm looking for evidence substantiating the motivation for 4D embodied space generation (L114 - 116) - "Single-view approaches face ... extend the diffusion generator ... to multi-view video generation pipeline".

EnerVerse improves over the base generative models. - Great. This is well demonstrated.

EnerVerse outperforms existing robotic baselines. - Great. This is well demonstrated too.

But how do the base generative models (without multi-view extensions from EnerVerse) fare on robotics? It's possible I'm mistaken, but is it not technically possible to add a policy head on top of the base video generative models (without multiple views) and train a policy?

---

Lack of quantitative ablations for 4DGS

TL; DR --- What is the quantitative benefit of 4DGS?

The authors have again pointed to qualitative examples, which I do appreciate. However, my question was again not addressed. Specifically, "Not understanding the impact of this through experimental analyses is concerning.". This is a nice technical aspect of the generation pipeline, and my motivation for these questions is to better highlight the value through quantitative analyses, not simply through a few qualitative figures.