Learning 4D Embodied World Models

1. For EMBODIED ACTION PLANNING, only RGB information is used as the input of control, while most of the content of this paper is to explain the importance of 3D content. Many current works have proved the feasibility of RGBimage as the input of IDM [1][2][3][4]. The article needs more experiments to prove the advantage of additional information.

2. It is a long pipeline to predict RGB-DN and then build dynamic 4D representations and control the embodied agent through IDM. The real-time performance of the method is worrying.

3. In section 4.4 ai=ID(si,si+1), But immediately after, "These features, combined with the instruction text embeddings, are further processed by an MLP to generate the final action. "The description is inconsistent with the formula

4. In the experimental section, image-based IDM is used by Opensora, which does not accept text input, while the proposed method in question 3 does accept text input. I think it is unfair to compare the effects of the two.

[1]Du, Yilun, et al. "Video language planning." arXiv preprint arXiv:2310.10625 (2023). [2]Du, Yilun, et al. "Learning universal policies via text-guided video generation." Advances in Neural Information Processing Systems 36 (2024). [3]Ajay, Anurag, et al. "Compositional foundation models for hierarchical planning." Advances in Neural Information Processing Systems 36 (2024). [4]Black, Kevin, et al. "Zero-shot robotic manipulation with pretrained image-editing diffusion models." arXiv preprint arXiv:2310.10639 (2023).

I have two major complaints about the paper:

1. the presentation of the paper is poor. The figures are not well-made and sometimes contain obvious errors.
2. the contribution to robot learning is very limited.

• the authors claim that they "collected" the data (e.g. in Fig. 2), but also mentioned that the data is from RT-1. Did the authors collect new data for this work or just reuse data from RT-1? Will the data be publicly released?
• the presentation of this paper is pretty poor. The figures are not very well-made or clear and sometimes contain obvious errors. For example, in figure 2, "guided depth diffusion" and "guided normal diffusion" are flipped. The texts are also not centered or aligned. It is clear that the authors didn't spend enough time polishing the figures which poses some difficulties for readers.
• how generalizable is the model? All of the evaluation seems to be done on in-distribution datasets. What about in-the-wild results? What about other robot form factors like UR5 and Franka?
• one of the main claimed benefits of the proposed model is the benefits to robot policy learning. Is there any actual policy rollouts or robot planning results? If the paper claims contribution to manipulation, evaluation only on prediction quality such as 2d/3d losses is not enough. There's a big gap between a good prediction and a successful robot execution.
• in table 4, why only compare to an invented baseline based on OpenSora? What about typical BC or RL policies? The baseline used doesn't really provide much information since the implementation is not clear.
• is there any deformable objects being tested? From the paper, all figures show only rigid objects and grasping tasks. These tasks don't really require an advanced world model.
• is there any evaluation on the inverse dynamics model?

1. Lack of credit to 3D-VLA. 3D-VLA has proposed the annotation pipeline to estimate optical flow that guide aligned depth estimation for existing videos. However, the paper neither acknowledges 3D-VLA's annotation methodology nor provides a comparative analysis between the proposed method and 3D-VLA.

2. As shown in the website, the consistency of generated RGB-DN flows seem not good enough. This raises concerns regarding the suboptimal quality of dynamic meshes constructed from potentially inaccurate RGB-DN and their subsequent impact on downstream applications.

3. The experimental evaluation would benefit from extended investigation in robotic policy learning. Specifically, the authors could assess the robustness and generalizability of their method across a broader range of tasks in RLBench or alternative benchmarks such as CALVIN.

Approach seems tailor-made for simplistic scenarios

All experiments are performed on simplistic videos in tabletop robot manipulation setting without much camera movements or distractors / clutter. Different aspects of the method rely on the specifics of these settings. For example, using optical flow to infer static parts of the images will work only when the camera is still. This means that both the data annotation generation (i.e., temporally consistent depth and surface normals) and the 4D mesh reconstruction steps are likely to work only in restricted table-top settings. I do not see how such a method can be generalized to robot navigation settings or the even harder egocentric video setting.

Novelty concerns

The idea of estimating temporally consistent depth is not new. How does the proposed method fare against prior work cited below?

Li, Siyuan, et al. "Enforcing temporal consistency in video depth estimation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021. Shao, Jiahao, et al. "Learning Temporally Consistent Video Depth from Video Diffusion Priors." arXiv preprint arXiv:2406.01493 (2024).

Lack of writing clarity

The writing clarity is significantly lacking. This is partly due to the fact that the method builds upon several existing and fairly complex systems. Nevertheless, the approach is hard to completely understand, given the description in the paper. The paper should be self-sufficient to a large degree.

• L174 - 175: Why only one gradient step in Equation 3?
• L216 - 221: "pair of samples" - Not clear what pairs of samples are in this context.
• Equation 7: The noise variables $\epsilon_v, \epsilon_d, \epsilon_n$ are not defined.
• L230 - 236: What is the temporal VAE architecture from Open-Sora? How exactly is the OpenSora architecture modified to accommodate RGB-DN images? Architecture descriptions and diagrams are missing in the paper. Which parameters from the model are finetuned?
• Table 4 - what is the accuracy metric for action planning?
• Section 5.1 - what are the inputs for the 4D mesh prediction task? Is it the entire video or only the first frame?
• Section 5.3: How is the inverse dynamics model trained?

Missing experiments for "why is 4D embodied world model needed"

A key motivation for the proposed method is that directly modeling the 3D world dynamics can more accurately capture the physical interactions and assist with robotic planning (L043 - 048). However, this core hypothesis has not been sufficiently tested in the paper, i.e., is an embodied 4D world model needed for robotic planning? The only experiment for robotic planning is in Section 5.3 and Table 4 on the RLBench dataset. However, the only baseline selected is a video diffusion model trained on OpenSora. While the 4D world model improves over video diffusion, it has poor accuracies on the majority of the tasks in these simplistic settings. Moreover, relevant methods like Pandora, UniSim, RoboDreamer, 3D-VLA, and This&That (some cited in the related work section) have not been compared against, even though they're useful for action planning. How do these non-4D-world-models fare against the proposed method?

Reference for This&That --- This&That: Language-Gesture Controlled Video Generation for Robot Planning

Quality-checking depth and surface normals

The annotation approach from Section 3 is interesting, but the quality of annotations has not been sufficiently validated. Table 2 shows preliminary consistency estimates for static regions. However, a more thorough evaluation of the depth and surface normal accuracy is needed on settings like RLBench where ground-truth temporal depth and surface normal estimates can be obtained from the simulator. This is particularly critical since the RT-1 annotations are used for evaluating model predictions in Table 1.

Other experimental concerns

• Novel view synthesis results look promising in Table 3, but qualitative results are missing to better understand how the difference in metrics is reflected in qualitative visualizations.
• L430 - Has the 4D world model been trained on RLBench data, but the baseline video diffusion model been trained on out-of-domain OpenSora data?