Learning 3D Persistent Embodied World Models

Dear Reviewer,

We thank the reviewer for the detailed comments and questions. We are glad to hear that you find our paper is well-written and our problem is significant. Here is the response to your question.

"the underlying state of the world is represented as a single image or chunk of images." ... The paper that invented the term "world model" represents the state using an RNN.

We would like to clarify that in this context, our use of "embodied world models" specifically refers to recent large video diffusion models, as discussed in [1, 2]. We acknowledge that the original "world model" framework employed RNNs for state representation, and we will make this distinction clearer in our revision.

[1] Bar, Amir, et al. "Navigation world models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Yang, Mengjiao, et al. "Learning interactive real-world simulators." arXiv preprint arXiv:2310.06114 1.2 (2023): 6.

do not provide any generated video.

In the main paper and Appendix, we included detailed frame-by-frame visualizations to highlight both the strengths and potential failure cases that could be missed in video playback. Unfortunately, due to rebuttal submission policies, we are unable to upload or share additional video material during this stage.

do not provide any code

As stated in our submission, we will open-source both our code and dataset upon acceptance of the paper. We would send the anonymous link related to the code if available at this stage.

What are DiT blocks?

DiT blocks [1] refer to using Transformers as the backbone of diffusion models. In our work, these are implemented in the backbone CogVideoX [2]. We will clarify this in the revision to make it explicit for readers.

[1] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

[2] Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.

Equation 2: What is $\epsilon$? Is this noise? If so, how do you define it? Should it be time-dependent?

$\epsilon$ is noise and is sampled from a Gaussian distribution. This formulation is standard in diffusion models, as also used in CogVideoX [1] and Stable Video Diffusion [2]. We will clarify this notation and provide additional details in the revision.

[1] Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.

[2] Yao C H, Xie Y, Voleti V, et al. Sv4d 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4d generation[J]. arXiv preprint arXiv:2503.16396, 2025.

Equation 2, 3: What is $c$?

$c$ refers to the camera pose transform, which is defined in Section 2.1.

Equation 4: What is $R(v)$? You define the reward function as $r$.

Reward function $r$ is the reward of a single step. Return $R(V)$ is the sum of the rewards in the videos. In our paper, $R(V)$ is defined as the reward of the last frame, following Navigation World Model [1].

We will clarify these definitions in the revision. Additionally, we note that it is common in reinforcement learning literature (e.g., Policy Gradient [2]) to denote the return or expected cumulative reward as $R$.

[1] Bar A, Zhou G, Tran D, et al. Navigation world models[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15791-15801.

[2] Sutton R S, McAllester D, Singh S, et al. Policy gradient methods for reinforcement learning with function approximation[J]. Advances in neural information processing systems, 1999, 12.

"The above optimization objective searches for sequences of actions a", but in the preceding paragraph, you find $a^*$ by "optimiz[ing] an action"

Model predictive control (MPC) will first initialize the solution space with a Gaussian distribution, then search candidate solutions in the solution space, which will be used to optimize the solution space based on the expected rewards $R(V)$.

Line 133: You say you "train the memory blocks only and freeze other parameters". The memory blocks themselves are just tensors, not parameters. Which weights do you actually train here? It is just the cross attention weights? Do you train the feedforward network as well?

To be precise, the memory features are tensors, whereas the memory blocks are neural networks that include both cross-attention and feedforward networks, as shown in Figure 2. When we state that we "train the memory blocks only and freeze other parameters," we mean that we update the weights of the cross-attention and feedforward networks within the memory blocks, while keeping the rest of the model parameters fixed. We will clarify this in the revision.

How is a depth-free variant of your model expected to work?

We used pre-trained depth estimation models (Depth Anything v2 [1]) to estimate the depth of the generated videos, and then updated the 3D memory.

[1] Yang, Lihe, et al. "Depth anything v2." Advances in Neural Information Processing Systems 37 (2024): 21875-21911.

Dear Reviewer,

We sincerely appreciate your thoughtful feedback. Your insights and suggestions have been invaluable in guiding our revisions. We will carefully incorporate the points discussed into the final version to enhance its quality.

Thank you once again for your time and consideration.

Best,

The Authors

Dear Reviewer,

We appreciate the reviewer for the detailed comments and insightful suggestions. We are glad to hear that you find our paper is well motivated and our experimental validation is extensive. Here is the response to your question.

"Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video", but I understand that memory is part, not input, to the model; it is built implicitly.

We would like to clarify that, in our method, 3D memory is explicitly constructed and integrated into the model by learning memory blocks, as illustrated in Figure 2 and Figure 8 (Appendix). Our memory blocks are composed of expert adaptive layernorms, cross-attention blocks, and feedforward blocks. We will revise the paper to better clarify it.

The paper often uses references with absolutely no context, making the paper not self-contained. For example: "Drawing from TesserAct [40], we separately encode RGB and depth with 3D VAE."; a better statement would be "we do XYZ, following [40]." Another example of weird sentence without context is "We also downsample the Plücker camera embeddings with the same compression rate using UnPixelShuffle layers [26]." There are many such cases.

We appreciate your suggestions and will thoroughly review all references to ensure they are introduced with appropriate context, making the paper easier to follow.

The proposed work can only be trained on synthetic data. The paper argues that this issue could be mitigated with depth generation models, but I am skeptical.

I would have hoped to see this method trained and evaluated on at least more challenging synthetic datasets (For example, Fabri et al., MOTSynth, ICCV'21, Virtual KITTI; these could be assessed for video generation without control).

Thank you for your insightful feedback. We respectfully disagree with the notion that our method can only be trained on synthetic data. Recent advances in 3D feedforward models [1, 2] have demonstrated the ability to generate high-quality depth maps from multiple views in the real scenario.

Despite the added challenges posed by dynamic scenes and noisy depth maps, our memory approach still demonstrates improvements in visual fidelity. These results suggest that our method is robust to moderate noise in the depth input, and we believe that further advancements in depth estimation will enhance the generality and applicability of our approach.

[1] Wang, Shuzhe, et al. "Dust3r: Geometric 3d vision made easy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Jianyuan, et al. "Vggt: Visual geometry grounded transformer." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Papers states in 2.2: "For efficiency, we extract the meaningful grids from the 3D map.". Would you please explain?

When constructing the 3D memory, some grids may be empty or contain very few points. For efficiency, we filter out these empty or sparsely populated grids and only retain grids with meaningful information. This reduces computation and memory usage without compromising the quality of the 3D memory representation.

Also, could authors elaborate on the statement: "A key challenge in combining video models with the 3D spatial feature map is that most video models are limited in the abilities of 3D spatial understanding." To the best of my knowledge, this statement is inconsistent with findings in the literature.

We acknowledge that models such as Cosmos demonstrate strong 3D spatial reasoning; however, these models are trained with extremely large-scale datasets and computational resources, typically accessible only to major companies. In contrast, our approach aims to enhance 3D spatial understanding without requiring massive data or GPU resources. By leveraging 3D-structured memory, depth-aware generation, and precise camera control, our method enables effective 3D reasoning in a more accessible and resource-efficient framework.

Dear Reviewer,

Either show that this method can be trained on real data, or acknowledge that this is a limitation. The table referenced above still applies for synthetic data only

We would like to clarify that the results on Virtual Kitti, while still based on synthetic data, hold meaningful insights. They demonstrate that our model maintains robustness even with noisy depth inputs, which is a step toward addressing the challenges of real-world data where depth estimation is often imperfect. This robustness, we believe, provides valuable evidence for the potential of our approach as depth estimation techniques continue to advance.

Our current experiments are indeed conducted on synthetic data and we have explicitly flagged this limitation in the paper "we need the data with depth". We will strengthen this acknowledgment in the revision to leave no room for ambiguity.

most video models are limited in the abilities of 3D spatial understanding

We apologize for the imprecision in our earlier statement. While Cosmos is a promising model, it has significant limitations when it comes to 3D spatial understanding. For instance, even Cosmos itself acknowledges it’s still in the early stages and lacks object permanence.

Our key point, supported by our empirical results, is that explicitly incorporating depth information and 3D memory enhances 3D persistence. As shown in Table 1, our model with depth (Ours) outperforms the ablative version without depth (Ours (w/o depth)) across metrics like PSNR, SSIM, and SRC, directly demonstrating that depth-aware generation enhances the correlation between the hidden states of the video and 3D memory.

Almost every equation in the paper confused me

Because of length limitations, we didn’t include preliminary background in the paper. We recognize that we assumed that readers already have foundational knowledge in this area. However, our equations adhere to standard formulations widely used in diffusion model literature [1, 2]—this is not arbitrary, but grounded in established conventions.

In the revised version, we will significantly expand the preliminary section to include detailed explanations of the diffusion model fundamentals that underpin our equations.

[1] Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.

[2] Yao C H, Xie Y, Voleti V, et al. Sv4d 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4d generation[J]. arXiv preprint arXiv:2503.16396, 2025.

Dear Reviewer,

We appreciate the reviewer for the detailed comments and insightful suggestions. We are glad to hear you find that our idea is appealing and our model has significant improvements. Here is the response to your question.

some evaluations of the physical collisions

Thanks for the suggestions. To evaluate how well our model handles physical collisions, we measure frame-wise similarity using LPIPS when collisions occur. We compare the performance of our model to a baseline that renders videos in the simulation without physical collision handling. As shown in the table, our model is able to accurately simulate future frames after a collision.

What type of information is stored in this model volume? Can you linearly decode a semantic map of the room?

Our model constructs a volumetric memory representation by populating 3D grids with DINO features. These features capture detailed structural and texture information, which aids in reconstructing the geometry and color of objects in 3D space [1].

Yes. The information stored in our memory can be linearly decoded into a semantic map of the room. This is supported by prior work [2, 3] showing that DINO features are highly effective for dense recognition tasks such as detection and semantic segmentation. Additionally, some works [4] have explored using a similar memory volume as a semantic map for vision-language navigation tasks.

[1] Hong, Yicong, et al. "Lrm: Large reconstruction model for single image to 3d." arXiv preprint arXiv:2311.04400 (2023).

[2] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

[3] Ren T, Chen Y, Jiang Q, et al. Dino-x: A unified vision model for open-world object detection and understanding[J]. arXiv preprint arXiv:2411.14347, 2024.

[4] Wang, Zihan, et al. "Gridmm: Grid memory map for vision-and-language navigation." Proceedings of the IEEE/CVF International conference on computer vision. 2023.

What is the sensitivity to voxel size ? How bit of a maximum scene can this approach handle?

Thanks for the suggestion. We evaluated the model with different voxel sizes, as shown in the table. Our results indicate that when the voxel size is 256 or larger, model performance remains stable. However, when the voxel size is set below 256, we observe a slight decrease in performance.

In principle, our approach is capable of handling very large—and even unbounded—scenes. This is achieved by maintaining a large, potentially infinite, grid map and dynamically extracting the relevant local region based on the camera to be fed into the video diffusion model. This design allows our method to scale effectively with scene size.

How does this compare to existing novel view synthesis and world models out there such as stable video camera?

We compare our model with the stable video camera as shown in the table. We find that stable video camera is good at novel view synthesis, while it also generates new content quite easily without memory.

Does the explicit memory approach only show benefits when the trajectories are very long?

Not only. Our memory approach can improve the 3D spatial consistency of the generated videos. We evaluate the performance of the model for short-horizon generation. The results are shown in the table.

And in that case how does it compare to simply using an every updating NERF to store a representation of the environment?

Our model not only predicts future observations under action control but also faithfully reconstructs past scenes with a high degree of spatial coherence. NeRF-based approaches are limited in their generative capabilities and cannot synthesize novel elements beyond their original observations. However, constructing and updating NeRF representations is computationally expensive—typically requiring several minutes for a single scene, which makes it infeasible to use NeRF during training or for scalable, interactive applications. In comparison, our method enables more efficient and flexible scene representation and generation, especially in action-conditioned and open-ended settings.

Dear Reviewer,

Thank you for your detailed feedback—we greatly appreciate your insights. We would like to clarify that our core contribution lies not in novel view synthesis, but in addressing a critical limitation of most existing world models: their lack of memory, which hinders an agent’s ability to generate consistent long-horizon plans.

Our proposed memory mechanism is designed to be general, meaning it can be integrated into other world models such as Stable Virtual Camera (SVC). We believe that equipping world models with such persistent memory capabilities will significantly enhance their utility for planning and policy learning.

We would like to clarify that our comparison with Stable Video Camera is just to illustrate and ablate the effect of memory in our rendering setting. While Stable Video Camera is a stronger rendering model, it lacks memory and thus performs worse than our approach on our dataset. We performed a zero-shot evaluation of Stable Virtual Camera, as its training code has not been made open-source. We are willing to explore the integration of our memory framework with SVC in future work.

Best,

The Authors

Dear Reviewer,

We thank the reviewer for the detailed reviews and insightful suggestions. We are glad to hear that you find our paper is well motivated and well-written. Here is the response to your question.

The volumetric representation could be intractable in large and complex scenes, limiting the usage of this representation in real scenarios.

We acknowledge the potential scalability challenges of volumetric representations in large and complex environments. However, our video dataset is based on HM3D [1], which contains large-scale 3D reconstructions of diverse real-world locations. Each HM3D scene comprises multiple rooms and a variety of objects, with the largest scene covering an area of 2,172 m². Our approach has demonstrated effective scaling to these large and complex environments. We believe this indicates the practical applicability of our method to real-world scenarios, though we agree that further work on memory efficiency and scalability could further enhance its usability in even larger-scale settings.

[1] Ramakrishnan, Santhosh K., et al. "Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai." arXiv preprint arXiv:2109.08238 (2021).

What are the rationales in choosing the proposed 3D representation method, as opposed to alternatives like point cloud, nerf, InfiniCube? Are there ablation studies run on alternative 3D representations, or ablation studies on not using dino features?

Thanks for the suggestions. We evaluate the model with different 3D representation methods as shown in the table. We experimented with integrating NeRF into the video generation pipeline; however, NeRF requires several minutes to reconstruct a scene, making it impractical for use during training. InfiniCube, on the other hand, relies on structured high-definition maps that are primarily tailored for autonomous driving scenarios, which limits its applicability to our more general settings. Compared to point cloud representations and the use of CLIP features, our proposed 3D memory approach using DINO features consistently achieves the best performance.

What’s the long horizon boundary of the given method?

The long-horizon capability of our method depends on the complexity and scale of the scene. There is an increasing probability that the memory module may struggle to maintain temporal coherence, leading to degradation in generation quality when $t$ is larger than 112.

What is the end-to-end inference latency for future frame generation and dynamic memory updates, and how does the latency compare with other methods in the ablation? Is the inference time real-time?

We show the inference latency in the table. At present, our video generation model does not achieve real-time performance. However, we believe that real-time inference is feasible by incorporating recent acceleration techniques such as diffusion-forcing [1, 2]. We plan to explore these optimizations in future work to further reduce latency and enable real-time applications.

[1] Yin, Tianwei, et al. "From slow bidirectional to fast autoregressive video diffusion models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Chen, Boyuan, et al. "Diffusion forcing: Next-token prediction meets full-sequence diffusion." Advances in Neural Information Processing Systems 37 (2024): 24081-24125.

Dear Reviewer,

Thank you sincerely for your thoughtful feedback and for recognizing the significance of our work. We are eager to implement these revisions that address your concerns comprehensively in the final version. Thank you again for your invaluable input.

Best,

The Authors