DINO-Foresight: Looking into the Future with DINO

Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:

Claims about VAEs

We appreciate the point regarding VAE features and acknowledge the success of VAE-based approaches like 4M [2,3]. However, we highlight several important distinctions:

First, our claim is that standard VAEs trained with image reconstruction objectives achieve inferior semantic alignment in their latent spaces compared to powerful encoders like DINOv2. To substantiate this claim, we conducted experiments using a publicly available VQ-VAE from the LDM framework [1]. As shown in Table 2 of our main paper, DPT heads trained on this VAE's latent embeddings significantly underperform compared to those using DINO and other VFMs in the oracle scenario. We further validated this by training a model to predict future discrete tokens within this VAE's vocabulary using cross-entropy loss, where predicted codes were converted to continuous features for downstream tasks. We will incorporate these findings in the relevant section (lines 322-326) of the main paper.

Regarding 4M, we clarify that it employs separate VQ-VAE tokenizers for each modality, including one specifically trained for DINOv2 features (trained to reconstruct these features). This DINOv2 feature tokenizer differs fundamentally from standard image tokenizers. Our claim specifically concerns the limitations of image-based VAE tokenizers for scene understanding tasks. (As an aside, we note that our PCA reduction of DINOv2 features can be viewed as a simple continuous tokenizer alternative.)

Question 1: Discrete vs. Continuous features

In continuation to the previous discussion, and in response to the reviewer’s request, we compared forecasting performance using 4M's publicly available DINOv2 tokenizer (with cross-entropy loss on the predictor) against our proposed approach. For fair comparison, we retrained a simplified DINO-Foresight (ViT-B14 without registers, using only last-layer features). While the tokenized variant shows slightly worse oracle performance, DINO-Foresight without the VQ-VAE tokenization demonstrates significantly superior results for semantic future prediction.

Comparison with V-JEPA

Thank you for highlighting this. We have cited V-JEPA in supplementary material and will be happy to add it to the main manuscript. However, there are key algorithmic differences: V-JEPA is a self-supervised pretraining method for learning general video representations (i.e., video encoders) via masked feature prediction, without any explicit focus on future feature forecasting (e.g., its evaluation is focused on video understanding tasks such as action classification). In contrast, our work specifically targets the future prediction of semantically rich features from frozen Vision Foundation Models (VFMs) to enable downstream future scene understanding. This reflects a fundamental difference in objectives: V-JEPA aims to learn transferable video representations, while our approach emphasizes multi-task semantic forecasting.

Additionally, since V-JEPA's goal is to learn the video encoder, it jointly trains both encoder and predictor in an end-to-end manner, ultimately discarding the predictor after training. In contrast, Dino-Foresight keeps the VFM encoder frozen, training only the temporal predictor for forecasting. This design choice simplifies optimization and leverages the strong semantic consistency of DINOv2 features, making Dino-Foresight particularly well-suited for future prediction tasks, while V-JEPA remains more suitable for video understanding

Futurist discussion

You're right that we should better discuss Futurist. While Table 1 shows our competitive results, Futurist requires training separate models for each task (semantic segmentation and depth), whereas our single model handles all tasks simultaneously. This scalability advantage represents a key practical contribution (see provided discussion about advantages over futurist in the response to Reviewer Vuqi). We will expand this discussion in Section 4.2.

Technical novelty

While we build on established components, our core contribution is demonstrating that forecasting VFM features creates a unified semantic representation space for multi-task future prediction. This shifts the paradigm from task-specific or pixel-level forecasting to semantic feature forecasting. The 2D-3D attention decomposition and two-phase training are practical implementation choices rather than our primary novelties. Although these components are not novel themselves, we believe our work makes an important contribution by providing an accurate, open-source framework (code to be released) that will significantly advance research in feature-based future prediction.

Question 2: Alternative masking strategies and loss functions

We conducted extensive ablations on different loss functions and masking strategies, as shown in the tables below.

Results demonstrate that different loss functions do not provide significant differences in performance:

Short Term Results:

Mid Term Results

For masking strategies, we implemented random masking for the future frame and compared it with our full masking approach. Our experiments demonstrate the superiority of the full masking strategy on future frame prediction over random masking.

Short Term Results

Mid Term Results

[1]. High-Resolution Image Synthesis with Latent Diffusion Models

[2]. 4M: Massively Multimodal Masked Modeling

[3]. 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:

Difference with DINOv2 and temporal MAE

While our method leverages DINOv2 features and employs masked transformer architectures, our core contribution centers on modeling the temporal dynamics of VFM features to achieve accurate, multi-task, dense semantic future prediction. In contrast, methods like Temporal MAE (or VideoMAE) focus on self-supervised pretraining of video representations (i.e., learning generic video encoders) through masked video modeling objectives. This represents a fundamental difference in goals: while their aim is to learn transferable video representations, ours is to enable precise future semantic forecasting.

Paragraph structure

We appreciate your feedback on the manuscript structure. Indeed, in both cases mentioned, we would like to emphasize our main contribution and highlight differences with latent generative approaches respectively. We will restructure the paragraphs to provide smoother transitions and clear flow.

Advantages over prior works

While Table 1 shows competitive numerical results with FUTURIST, there are important practical advantages that our method provides:

1. Unified framework: Our single model handles multiple tasks (semantic segmentation, depth, surface normals, instance segmentation) simultaneously using expressive and general frozen VFM representations, unlike prior works that would require task-specific training for each modality and dataset. For instance, FUTURIST would need to precompute pseudo-labels separately for each modality and dataset, significantly increasing implementation complexity.
2. Scalability: New tasks can be added by simply plugging in new heads that can be trained with limited labels, without fine-tuning the feature extractor or the prediction architecture. Furthermore, since Dino-Foresight operates directly on RGB inputs rather than precomputed modality-specific features, it can readily leverage additional raw video data for further improvements. This avoids potential misalignment issues that can arise when mixing precomputed features or pseudo-labels across datasets with differing label spaces or class definitions. Also, once FUTURIST is trained on specific modalities, it must be retrained from scratch to accommodate additional modalities.
3. DINO-Foresight is the only method to deliver future predictions for all four fundamental scene understanding tasks simultaneously, while being readily extensible to many more new tasks through its modular design. In contrast, prior works provide future predictions for to two tasks simultaneously at most.

Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:

Discussion with some prior works

Thank you for raising this important point. We acknowledge our manuscript could benefit from a more thorough discussion of additional related methods such as V-JEPA and DINO-WM.

• V-JEPA is a self-supervised pretraining method for learning general video representations (i.e., video encoders) via masked feature prediction, without explicit focus on future feature forecasting. In contrast, our work specifically targets the prediction of semantically rich features from frozen Vision Foundation Models (VFMs) to enable downstream future scene understanding. This reflects a fundamental difference in objectives: V-JEPA aims to learn transferable video representations, while our approach emphasizes multi-task semantic forecasting.
• DINO-WM utilizes DINO features for world modeling, primarily applied to action-conditioned planning in simulated environments with synthetic data. Our work, instead, focuses on real-world autonomous driving scenarios and multi-task future prediction, with an emphasis on accurate dense (pixel-wise) semantic forecasting rather than planning. We would like also to mention that these two works were developed concurrently.

We will revise the manuscript to incorporate these discussions.

Model size and Data Scale scalability

You raise an excellent point about scalability analysis. While our initial experiments focus on demonstrating the core concept with a fixed architecture, investigating how performance scales with model size and data would provide valuable insights into our method's potential. To this end, we provide preliminary results of model size and data scale experiments conducted to assess the impact on downstream task performance:

The following two tables demonstrate how varying the model size—while keeping dataset size and training duration fixed—affects downstream task performance for both short-term and mid-term future prediction. We evaluate a small and a large variant alongside our default (base) architecture by modifying the hidden dimension and number of attention heads. Results indicate that our framework benefits from increased model capacity, with improvements particularly notable for mid-term predictions. These findings motivate future work to explore even larger variants and leverage more data to fully realize the potential of forecasting VFM features for multi-task scene understanding.

Short Term Results:

Mid Term Results:

The following two tables illustrate the impact of varying training data size—while keeping model size fixed—on downstream task performance for both short-term and mid-term future prediction. Specifically, we combine the Cityscapes (CS) and nuScenes (NU) datasets, ensuring equal-probability sampling from each during training. Preliminary results on the Cityscapes test set show that the model trained on the combined datasets achieves consistent improvements in semantic future prediction across all tasks. These findings highlight the benefit of increased and diverse training data and motivate further exploration of scalability with larger datasets and extended training to better realize the potential of forecasting VFM features for multi-task scene understanding.

Short Term Results:

Mid Term Results:

Thank you for your comprehensive review and insightful feedback. We appreciate the assessment of our work and the constructive suggestions for improvement. Below, we provide our responses to each of your points:

Clarifying the technical contribution of our work

We appreciate your assessment of the method's simplicity as a strength while noting concerns about novelty. Our core contribution lies in demonstrating that modeling the temporal evolution of VFM features enables effective multi-task semantic future prediction. While the individual technical components are not novel, our design reveals that VFM features maintain semantic richness when predicted temporally, enabling a unified framework that operates across multiple scene understanding tasks without task-specific training.

Predicting future RGB frames from DINO

While out of scope for this work, we explored the ability of our method to predict future RGB from predicted DINO features. To this end, we trained offline a DPT head with simple L2 loss to "reconstruct" RGB from DINOv2 features using just ~3000 RGB images and then applied it to our prediction transformer following our pipeline. Preliminary results show that in the oracle setting, DINOv2+DPT can sufficiently reconstruct RGB frames with very low MSE, and in the prediction setting provides sufficient RGB predictions with clear structure but blurry regions in high motion/ambiguous areas. This preliminary experiment demonstrates that while our focus remains on semantic prediction for autonomous driving applications, our framework shows promising potential for extension to RGB prediction - potentially through a generative decoder (e.g., diffusion models) that could convert predicted DINO features to RGB frames.

Zero-shot Evaluation

You raise a valuable point regarding the importance of cross-dataset evaluation. While our primary analysis has focused on Cityscapes and nuScenes, the model is built on top of DINOv2 features, which are inherently designed for strong generalizability across diverse visual domains. Due to time constraints and the lack of standardized benchmarks and data loading modules for Waymo (Panoramic Video Panoptic Segmentation Subset), we provide preliminary zero-shot results on nuScenes validation set using a model trained solely on Cityscapes. Notably, the zero-shot performance on nuScenes is only slightly worse than that of a model trained directly on nuScenes and surpasses all baselines. These initial findings demonstrate encouraging potential for zero-shot transfer beyond the original training dataset, supporting the view that our approach can generalize effectively to new domains. We see comprehensive cross-dataset and zero-shot evaluations as an important direction for future work.

Short Term Results on Nuscenes

Mid Term Results on Nuscenes

Motion-aware evaluation

We appreciate the suggestion to incorporate motion-aware evaluation metrics. In this work, we have followed standard benchmarks and evaluation protocols established by prior works, including MO-mIoU, which measures segmentation performance specifically on movable objects and thus partially addresses the dynamic aspects of street scenes. Incorporating motion-weighted evaluation techniques would be an interesting addition to assess the model’s ability to predict dynamic scene elements. To the best of our knowledge, such a comparison or implementation is not available in the standard benchmarks we adopted. However, due to the limited available time for the rebuttal, we were unable to implement such metric at this point. We are open to adopting such metrics in the final version of the paper or as part of future work.

Multi-modality

You correctly identify that our current approach produces deterministic predictions rather than capturing the multi-modal nature of future possibilities. We acknowledge this limitation in our paper's limitations section, where we discuss that our deterministic architecture does not account for inherent uncertainty in future predictions and propose incorporating stochastic elements such as diffusion losses on tokens to better capture this ambiguity. Despite predicting the most plausible future based on observed conditions, it does not capture the full distribution of possible outcomes. This represents an important future direction for handling scenarios where multiple futures are equally plausible (e.g., vehicles at intersections).

Domain diversity

We acknowledge that our evaluation is currently limited to street scenes, which constrains broader claims about general applicability. However, our approach operates on frozen DINOv2 VFM features, which are inherently generalizable and not domain-specific, having been trained on diverse visual data from multiple domains. By conducting temporal modeling directly in the VFM feature space, we further reduce the domain gap that would typically exist in raw RGB space, increasing the likelihood that our method can transfer to varied input domains beyond autonomous driving. Thus, extending our evaluation to more diverse video datasets is an important direction for future work. .

Overall the rebuttal has addressed all my concerns. In particular, It is great to see the zero shot generalization results! I would strongly vote for acceptance.

For the final version, I would encourage the authors to include a discussion on V-JEPA.