DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

We thank the reviewer for their insightful review, especially in highlighting that DINO-WM demonstrates “robust generalization” and “is a significant step toward more general-purpose models.” We now address the issues raised by the reviewer.

“I am a little worried about the real-world planning and control performance since the robot is always moving when the proposed model is planning the actions”

Our planning framework follows the principles of Model Predictive Control (MPC)—a well-established approach in real-world robotics. The planner optimizes future actions but executes only a subset before re-planning. The robot’s continuous movement is not an issue, as each planning step integrates updated proprioceptive and visual observations, ensuring robust closed-loop control.

For planning efficiency, we improved our inference code for DINO-WM since submission and now it takes 15.89s for CEM compared to 53s reported in the manuscript. We provide further ways to speed up planning and we refer to our response to Reviewer 57WD due to space limits.

“More experiments on tasks with complex textured backgrounds need to be evaluated.”

We evaluated ClutteredPushT, which adds a complex textured background to PushT. Please see our response to Reviewer 57WD for open-loop rollouts and final performance due to space limits.

“DINO-WM requires a comprehensive offline dataset with sufficient state-action coverage” “validation on real-world robotic platforms”

We agree with the reviewer that DINO-WM requires a diverse dataset for effective planning. While it avoids online interactions, reward signals, and expert demonstrations, it still follows the general trade-off that broader coverage improves generalization. However, this limitation can be mitigated by continuously updating the world model with new experiences, enabling progressive improvement without requiring extensive real-world interactions upfront.

For real-world validation, independent researchers have already successfully deployed DINO-WM on real robots. We are consulting with the AC on how to share this while maintaining anonymity.

“incorporating hierarchical planning or multi-level control strategies could enhance performance on more complex or fine-grained tasks.”

We totally agree that hierarchical planning can unlock further capabilities of DINO-WM. Our current work serves as a proof of concept that even single-level planning can be effective with pre-trained WMs. This establishes a strong foundation on which more advanced planning strategies can be built. We see this as an exciting and promising direction for future work.

“hyperparameter sensitivity and computational trade-offs” “method’s robustness and scalability”

We conducted a hyperparameter sweep for CEM on PushT. We refer to our response to Reviewer 57WD for the computation time and performance tradeoff due to space limits.

For WM training, we ablated image sizes on PointMaze (see response to Reviewer 4KWu). The training hyperparameters for DINO-WM is quite robust, as the same hyperparameters work across all six reported environments.

“Appendix A.4.2 is total a waste.”

Appendix A.4.2 demonstrates the necessity of the frame-level attention mask in DINO-WM’s predictor. The experiments (L749-753) supports our claim that the causal mask effectively ensures attention to past frames only, enabling the model to capture essential temporal dynamics such as velocity and acceleration. Would the reviewer prefer that we remove this section?

“More foundation models could be evaluated”

We agree that evaluating more foundation models can provide further insights on the pros and cons of DINO-WM compared to existing models. To this end, we evaluated Genie, a foundation model for generating interactive environments. We train the Genie model on our PushT dataset (open-loop rollouts in Figure 5). It is evident that Genie performs worse in prediction quality and future state estimation compared to DINO-WM, even with ground truth action conditioning.

For OpenVLA with SigLIP, we note that it is a language-conditioned feed-forward policy trained with expert trajectories, fundamentally differing from DINO-WM, which learns environment dynamics from any interaction dataset and performs goal-conditioned planning. Since our setting doesn’t assume expert data or language conditioning, OpenVLA is unsuitable for fine-tuning. Additionally, its released model assumes a fixed action space, making it infeasible to finetune the model on our datasets.

We also evaluate other foundational vision models such as pre-trained MAE as the encoder for DINO-WM. Please refer to our response to Reviewer 4KWu for the experiment details and final performance of this experiment due to the 5000 char response limit.

We thank the reviewer for their insightful and constructive feedback, and for acknowledging that DINO-WM “expands the current MBRL setting by incorporating the learned visual features from large vision models”, and the idea of using pretrained features to be “neat and simple”. We now address the questions raised in the review below.

DINO-WM results on more robotic tasks

In addition to the robotic deformable manipulation environments and the robotic arm reaching task presented in the manuscript, we further provide results of DINO-WM on LIBERO [1], a tabletop environment manipulating diverse objects, with third-person image observations and a 7DoF action space. We note that CEM in this raw action space would be inefficient starting from completely random action samples (which would result in the robot mostly jittering in place). Therefore, instead of sampling randomly, we sample from a pre-trained diffusion policy [2]. DINO-WM is capable of long-horizon predictions and assists in selecting the best action trajectories. We provide visualizations for open-loop rollouts of DINO-WM in Figure 1 and videos for task planning here. Without planning with DINO-WM, the diffusion policy model achieves a 35% success rate (across 20 trajectories). With planning using DINO-WM, the task success rate is improved to 55%. This demonstrates the effectiveness of DINO-WM, even with a high-dimensional continuous action space and a long task horizon.

TD-MPC2 as a baseline

We completely agree with the reviewer’s insight on why TD-MPC2 underperforms compared to other reconstruction-based baselines, as we have also discussed in the original manuscript (L290-293). Our motivation for including TD-MPC2 as a baseline is that it represents state-of-the-art work in world modeling while also incorporating planning to sample actions during training. We believe that comparing against TD-MPC2 provides valuable insight into the extent to which the performance of world models in this line of work depends on informative reward functions.

Architecture and size for decoders

We thank the reviewer for raising this constructive question. For the (optional) DINO-WM decoder, the architecture is based on VQ-VAE and consists of two stacked Decoder modules. Each Decoder is a transposed CNN with residual blocks. For DreamerV3 and IRIS, we use the decoders provided within the respective algorithms, both of which are CNN-based decoders.

We have included the number of parameters for the decoder in each baseline in the Table below. For DreamerV3 and IRIS, we follow the default parameters from the original implementations. In response to the reviewer’s suggestion, we have also matched the decoder sizes for DreamerV3 and IRIS (denoted as DreamerV3 Large and IRIS Large). The decoded open-loop rollout images are shown in Figure 4. We observe that DreamerV3 Large shows improved prediction quality compared to the original DreamerV3. However, IRIS Large demonstrates slightly worse performance than IRIS, which could be due to the fact that IRIS requires identical parameters for both its encoder and decoder, making the model more difficult to optimize.

[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

[2] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

We thank the reviewer for their constructive feedback, and for acknowledging that DINO-WM “demonstrates superior performance and could serve as a solid baseline for future research” and “could inspire future research a lot.” We address the issues raised in the review below.

“why DINOv2 performs better are not clearly analyzed”“they do not provide evidence that the frozen DINOv2 representations are better in task representation”

While our manuscript presents results on feature reconstruction and task planning, demonstrating that DINOv2 patch features enable more accurate world modeling, we now provide further analysis of the features themselves. A well-established method for evaluating feature quality in downstream control tasks is linear probing, which assesses how well the features encode task-relevant state information. We conducted linear probe experiments on PointMaze, PushT, and Wall, comparing DINO-S Patch, DINO-S CLS, DINO-B Patch, IR3M, and Pre-trained MAE (DINO-S and DINO-B denotes DINO model with ViT Small and Base architecture, respectively). The validation loss for these linear probes is reported in Table 2, where DINO-S Patch and DINO-B Patch achieve the lowest validation loss, indicating their superior task representation capabilities.

To further analyze the DINOv2 features, we visualize the principal components after performing PCA on the patch embeddings from DINO-S on our deformable environments which has the most complex state space, following a procedure similar to that in the original DINOv2 paper (Figure 2). The visualizations show that DINOv2 features effectively identify objects of interest, distinguish the agent, and separate the foreground from the background, reinforcing that frozen DINOv2 representations encode task-relevant information effectively.

“the authors only compare backbones with global representations in Tables 2-4, which are inherently unsuitable for capturing spatial relationships”“It would be more convincing if other semantic-rich alternatives, such as MAE [1], are included in the study”

We totally agree that having more baselines with semantic-rich baselines would provide further insights. We first note that for the IRIS baseline which we compare with in Table 1 and Table 3 of the manuscript, it encodes an image with 16 tokens instead of a global representation, which is also capable of capturing spatial relationships explicitly.

We ran a new experiment on PushT using a pre-trained MAE suggested by the reviewer. The performance is presented in the Table below. We observe that the WM trained with MAE features has lower final MPC performance than the original DINO-WM. We hypothesize this is because MAE prioritizes reconstruction over task relevance, as indicated by linear probe results. Additionally, even the smallest MAE encoder, which we used in the experiments, is still significantly larger than the DINO-S model, with feature dimensionality twice than that of DINO-S. This highlights that a larger and more expensive feature extractor does not necessarily translate to better task performance, and the higher computational cost of MAE makes it a less efficient choice.

“Q1 DINOv2 input resolution.”

Although DINOv2 is pre-trained on 224×224 images, its ViT-based architecture processes images as fixed-size patches, allowing it to handle arbitrary image sizes as long as the height and width are divisible by the patch size (e.g., 14 for DINO-S). In fact, the original DINOv2 paper [1] explicitly demonstrates its ability to work with non-224×224 images, including rectangular and high-resolution images (Section 7.5).

In DINO-WM, our choice of 196×196 input resolution is purely an engineering decision. Our decoder assumes a 16x spatial upscaling, and we aimed to match the environment’s 224×224 observation shape after decoding. Since DINOv2’s fixed patch size is 14, an input of 196×196 results in a 14×14 patch grid, which decodes to 224×224 after the 16× upscaling. However, using 224×224 directly with an additional interpolation layer at the decoder output would be equally valid.

To verify this, we conducted an ablation on PointMaze with input sizes ranging from 28×28 to 280×280. The results in Table 3 show that image size 224×224 performs on par with 196×196. This shows that our choice does not limit DINOv2’s capabilities. Moreover, the ability of DINO-WM to process images of various sizes enhances its flexibility in modeling environments of varying complexity, allowing for adaptive trade-offs between granularity and computational efficiency.

[1] DINOv2: Learning Robust Visual Features without Supervision

We thank the reviewer for their constructive feedback, for identifying that DINO-WM “significantly improves world modeling quality” and shows “a good embedding space makes world modeling + MPC a winning recipe”. We address the issues raised in the review below.

“it is uncertain whether frozen DINOv2 feature patches provide sufficient resolution for contact-rich tasks”

We show that frozen DINOv2 patch features capture state accurately, even in contact-rich tasks like PushT and deformable manipulation. Figure 4 in the manuscript illustrates how they precisely represent particle positions—challenging for global features like ImageNet-pretrained ResNet or R3M.

We further trained a DINO-WM on LIBERO [1], a tabletop environment manipulating diverse objects, with third-person image observations and a 7DoF action space. Figure 1 compares open-loop rollouts of WMs trained with DINO patch features vs. DINO CLS features. Reconstruction scores for the predicted frames can be seen in Table 7. This shows that DINO patch features can accurately represent the object within the gripper, whereas global representations struggle with dynamic elements—both the object in the gripper and the gripper itself. This further reinforces the suitability of patch features for contact-rich interactions.

In our response to Reviewer 4KWu, we provide linear probe results on the environment state using DINO patch features, along with PCA visualizations, demonstrating its ability to capture task-relevant information.

“it is unclear whether planning will be fast enough”

We address planning efficiency in 3 ways.

1. We improved our inference code for DINO-WM since submission. For the same hyperparameters, planning now takes 15.89s, compared to 53s reported in the manuscript.
2. While we report results for CEM with 100 samples per iteration, this can be adjusted based on task complexity. Table 1 presents the tradeoff between sample size, planning time, and performance on PushT. This flexibility allows us to balance computational efficiency with performance depending on task requirements.
3. Training and planning with a DINO-WM using a larger frameskip can speed up planning. This effectively increases the planning horizon as each prediction covers a longer time span. We train a DINO-WM with frameskip 25 on PushT and report the performance in Table 6. While modeling long-term dependencies is more challenging as frameskip increases, it presents an opportunity to improve efficiency, making hierarchical planning promising directions for future research.

“In cluttered environments with complex backgrounds, the model's performance remains uncertain”

We ran experiments on ClutteredPushT, where the background of the PushT environment is a cluttered real-world tabletop. Open-loop rollouts of a trained DINO-WM is in Figure 3, and we report the task performance in Table 5 compared to the original PushT.

In ClutteredPushT, DINO-WM can still identify the effect of agent actions and predict the motion of relevant objects. The final planning performance is only marginally degraded. This shows DINO-WM’s capability of modeling environments with complex backgrounds.

Additionally, we have trained an unconditional world model on the CLEVRER [2] dataset where multiple objects may collide. Videos of open-loop rollouts are provided here.

Q1. Environment with SE(3) action space.

We further provide results of DINO-WM on LIBERO which has a 7DoF action space. Due to space limit, we refer to our response to Reviewer dmt6 for planning videos and performance.

Q2. Addressed in the ClutteredPushT experiment.

Q3. Does increasing the latent-space planning horizon degrade performance?

Not necessarily. Longer planning horizon enables discovering states that are further away, with the tradeoff that the WM’s long-term prediction would be less accurate. We balance this via planning with receding horizons as in our deformable environments.

Q4. We conduct ablations with different image sizes (yielding different patch sizes after DINO encoder) on PointMaze. We report the planning success rate (SR) for CEM, MPC, and the predicted frame’s image scores in Table 3. With MPC, all models eventually obtain a decent SR, but models with larger patch size are able to achieve better SR with CEM. This shows that bigger patch sizes can contain more precise state information, making the world model more accurate for zero-shot open-loop planning.

Q5. Yes, our decoder is trained on the same data with all baselines including DreamerV3 and IRIS.

[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

[2] CLEVRER: CoLlision Events for Video REpresentation and Reasoning