We thank the reviewer for their thorough and insightful reviews, especially in highlighting that the world model is able to “plan in altered environments,” and can handle “simulation of difficult transition dynamics.”

We now address the issues raised by the reviewer.

W1: Evaluations on Additional Benchmarks

“ I would recommend including more standard reinforcement learning and robotics benchmarks, such as DMControl, RoboMimic, MimicGen, RLBench, MetaWorld, FrankaKitchen, ManiSkill, etc.”“I would recommend adding at least one benchmark from TD-MPC2 to strengthen the comparison.”

We thank the reviewer for this suggestion. Please refer to our global comments on why the standard RL benchmarks may not be suitable for evaluating DINO-WM, as it is not an task-specific online RL algorithm. Additionally, we would like to clarify that our evaluations on baselines like Dreamerv3 and TDMPC2 focuses on evaluating the world model component of each method, which is different from the evaluation proposed in their original papers where they have online interactions with the environment, have access to rewards, and focuses on single task settings.

However, we agree that including evaluations on these benchmarks would further strengthen the comparison and demonstrate DINO-WM’s applicability, thus we conducted an additional experiment on DMControl’s Reacher-Hard task, and further enhanced its difficulty by extending it to a multi-goal version. We compared planning performance between DINO-WM and DreamerV3, our strongest baseline. We observe a 43% performance improvement for DINO-WM compared to DreamerV3. Complete results and additional visualizations are provided in Appendix A.4. Additionally, we include planned trajectory visualizations on our website linked in the manuscript for further reference.

Additionally, we performed further evaluations of DINO-WM on LIBERO[1], a benchmark designed for lifelong robot learning and imitation learning. Here, we demonstrated that DINO-WM, trained on multi-task expert trajectories, effectively distinguishes between expert and non-expert trajectories. This capability positions DINO-WM as a promising approach for integration with multi-task or goal-conditioned policies, enabling the world model to facilitate predictions with high-level goal specifications.

We thank the reviewer for their thorough and constructive feedback, especially for highlighting that our work DINO-WM is “quite simple, scalable and effective,” and that it “generates solutions for tasks at test-time zero-shot without any expert demonstration, reward modeling or inverse dynamics models.”

We now address the issues raised in the review.

Clarification on “Passive Data”

“The authors argue in the intro and abstract of the paper that a world model should enable learning from only passive data but the experiments demonstrated in the paper always require action metadata during training of Dino-WM which contradicts the motivation.”

We appreciate the reviewer's feedback. To clarify, by "passive data," we refer to pre-collected offline datasets that do not require additional interactions with the environment during training, instead of datasets with no action labels. This distinction from online interaction-based data collection has been explicitly clarified in the revised manuscript’s introduction and abstract to better align with the paper's scope and methodology.

Planning Results for Diffusion-Based Baselines

“ I’d recommend authors to add comparison of AVDC or another diffusion based methods for whichever task possible in table.1”

We note the original AVDC is not compatible with planning evaluations, as it is a goal-conditioned generation model. This model generates entire sequences of observations conditioned on a target goal state, making it incompatible with our tasks, which involve iteratively planning and evaluating actions. To adapt AVDC for planning, one would require an additional inverse dynamics model to convert generated observation sequences into actionable plans. Furthermore, as we visualized in Section 4.6 of the original manuscript, the plans generated by AVDC often fail to reach the specified goal positions, exhibit large discontinuities between states, and sometimes include unrealistic environment configurations. These limitations would result in a task-solving success rate of 0, even if a trained inverse dynamics model were used.

To address the reviewer’s feedback, we have further implemented an action-conditioned variant of AVDC. Instead of generating the entire sequence conditioned on a goal, this variant predicts the next frame given the current frame and action, enabling it to be evaluated in the same planning framework as DINO-WM. However, this approach exhibited severe compounding errors over long horizons, ultimately resulting in a planning success rate of 0% on the PushT task. The full results, including open-loop rollouts and planning evaluations, have been added to Appendix A.7 in the updated manuscript.

“Currently, the section 4.2 implies there’ll be quantitative comparison with diffusion based methods.”

Thank you for raising this point. We have updated the manuscript to clarify this.

Clarification for “New Environments”

“experiments in section 4.5 tests generalization to unseen configurations of environments seen during training but the description in section 4.5 (L 450-451) implies that experiments test generalization to new environments on which the model was never trained which is not true. I’d recommend authors to please fix the text description in this section.”

We apologize for the confusion in the problem setting in Section 4.5. The experiments indeed focus on testing generalization to unseen configurations of environments encountered during training. For example, in the Wall environment, this involves walls and doors at previously unseen locations; in PushObj, objects of novel shapes and colors; and in the Granular environment, an unseen quantity of granular materials. These experiments do not evaluate generalization to entirely novel environments on which the model was never trained.

We have revised the text in Section 4.5 of the manuscript to clarify the scope of the experiments and ensure the description accurately conveys the intended problem setting. Thank you for bringing this to our attention.

Training Details for DINO-WM

“It is not clear to me what datasets were used for training Dino-WM, how were the trajectories collected, what was the scale of those datasets,etc.”

The dataset size and details about data collection are provided in Appendix A.1 of the original manuscript, including the scale of the datasets and the procedures used to collect trajectories. To address the concern more explicitly, we have updated the manuscript to also include this information in Appendix A.9, where we consolidated training-related details for better accessibility. We have further added the model’s architecture details in Appendix A.9 as well.

If additional specifics regarding training processes, model hyperparameters, or further dataset details would be helpful, we are more than happy to provide them and update the manuscript.

We appreciate the reviewer for their insightful feedback, especially for highlighting the potential for DINO-WM “for training robot foundation models with large quantities of offline data.”

We address the issues raised in the review below. We have also updated the manuscript based on the reviewer’s feedback.

Overall Novelty

"While the proposed framework might be unique in terms of a reconstruction-free reward-free latent-space world model, I do feel the overall novelty is a little bit limited here."

We appreciate the reviewer for acknowledging the novelty of the “reconstruction-free reward-free latent-space world model.” However, as outlined in our global response, we wish to emphasize that the innovation of our work extends beyond these design elements. Specifically, the core novelty and focus lie in addressing the feasibility of constructing general-purpose world models that solely capture environment dynamics, enabling reasoning and planning for arbitrary states. This approach targets the long-term goal of creating task-agnostic world models capable of generalizing to unseen dynamics and goals. This remains a largely uncharted area in existing research, underscoring the unique contributions of our work.

Complexity of Existing benchmarks and Results for Additional Benchmarks

“ IMO they are more or less 2D tasks ”

We thank the reviewer for raising this point. We understand that the deformable tasks may appear as a 2D task as particles mostly move in a 2D plane. However, we want to clarify that these tasks are inherently 3D. The observations capture the full 3D space, as illustrated in the visualizations in Figure 4. Furthermore, for robot execution, the system must perform low-level control actions within the simulator, even though a high-level action wrapper is utilized. To provide additional clarity, we have included unskipped rollout visualizations of the dataset on our anonymous website linked in the manuscript.

“I am curious how will the approach perform for more complicated manipulation or control tasks (e.g., Humanoid Run from DMControl and Move Chair from ManiSkill2)”

While we acknowledge that existing benchmarks feature tasks with lower-level action spaces and greater complexity, tasks like DMControl’s Humanoid Run or ManiSkill2’s Move Chair demand more than achieving a single target state. These tasks require maintaining continuous motions, such as “walking” or “moving without erratic behavior,” making them unsuitable for offline planning and reasoning approaches. Nevertheless, we conducted an additional experiment on DMControl’s Reacher-Hard task, and further enhanced its difficulty by extending it to a multi-goal version. We compared planning performance between DINO-WM and DreamerV3, our strongest baseline, with complete results and visualizations now included in Appendix A.4.

Additionally, we performed further evaluations of DINO-WM on LIBERO [1], a benchmark designed for lifelong robot learning and imitation learning. Here, we demonstrated that DINO-WM, trained on multi-task expert trajectories, effectively distinguishes between expert and non-expert trajectories. This capability positions DINO-WM as a promising approach for integration with multi-task or goal-conditioned policies, enabling the world model to facilitate predictions with high-level goal specifications.

“These tasks might require a much more accurate level of physical reasoning capabilities (objects have momentum and move in a more complex way) which can help us understand better the limit of DINOv2.”

As the reviewer’s concern is based on the model’s ability to perform accurate physical reasoning for objects with momentum and complex movements, we further trained an unconditional world model on the CLEVRER[2] dataset. This dataset features scenarios with objects of various shapes, sizes, and colors interacting in physically realistic ways, including collisions, rolling, and sliding dynamics. We demonstrated the model’s capability by showcasing predicted future video rollouts, illustrating its understanding of intricate object interactions and dynamics. Videos of these predicted rollouts have been included on our website.

We now address the reviewer’s questions below.

The author response has addressed my concern. I will leave it up to the AC to decide whether the overall novelty is adequate for the acceptance, though.

Thank you for your constructive feedback. We are delighted that you found our work to have effectively leveraged a pre-trained vision model to enhance world model training, and that our experiments across multiple environments and tasks are comprehensive and showing the model’s effectiveness. We appreciate your insights and will address your questions and concerns below.

Writing Improvement

“The paper's writing needs to be improved. The citations should be clearly separated from the main text using brackets when needed.”

Thank you for your suggestion. We have carefully revised the manuscript to improve the overall writing quality and ensure that citations are clearly separated from the main text using brackets where appropriate.

Novelty of DINO-WM

“Although DINO-WM uses DINOv2's latent features, the approach of learning a visual dynamics model in the latent space is not groundbreaking. Pretraining latent features for world model learning or learning latent world models through rewards are both exploited in previous literature. Leveraging existing vision models is not novel to me.”

Thank you for your insight. We agree that learning world models in the latent space itself is not novel. However, while there has been a large body of research using latent world models, they always assume access to downstream tasks and reward information. Therefore, it remains unknown whether it’s possible to build a general-purpose action-conditioned world model, and if they would enable generalization to arbitrary goals and novel environment layouts for control tasks. Our aim is to provide insights into this open question, using a pre-trained vision model as a means to facilitate this exploration, rather than proposing the use of pre-trained vision models as a novelty on its own.

Evaluations on Established Image-Based RL Benchmarks

“The work does not include results on established image-based RL benchmarks, such as DM-Control or Minecraft, … Including these benchmarks would strengthen claims of general applicability​.”

We thank the reviewer for pointing this out. While we agree that these benchmarks are widely used for evaluating RL methods, we would like to clarify that our primary objective differs from theirs. Specifically, DINO-WM is designed to operate in an offline, reward-free setting where the task is not known during training, and the goal is to generalize to arbitrary environment dynamics and goals at test time.

However, we agree that including some of these benchmarks would provide more insights of general applicability. Therefore, we ran additional experiments on the Reacher-Hard task of DMControl, as well as the libero_goal task suite on LIBERO[1], a manipulation benchmark for robotics. We have included these results in Appendix A.4.

Lack of Q-Learning

“The model’s lack of Q-learning hinders its capacity to handle tasks requiring long-term planning. In RL, Q-learning or similar approaches are often necessary for efficiently solving complex, high-horizon tasks. This limitation may affect DINO-WM’s scalability to challenging environments.​”

Thank you for the constructive feedback on long-term planning. We agree that handling long-horizon tasks is a significant challenge, particularly in the context of embodied AI. While Q-learning plays a crucial role in model-based RL methods where access to rewards are assumed, it cannot be directly applied in our setting where there is no reward information, and goals are only revealed at test time. We appreciate your suggestion and believe that incorporating techniques such as hierarchical planning could be a promising direction to enhance DINO-WM's ability to address long-term planning challenges in such reward-free and task-agnostic settings.

We now answer the reviewer’s questions.