Object Concepts Emerge from Motion

Dear Reviewer Rb8Z,
We greatly appreciate your valuable comments and questions. Our responses are as follows.

Abstract: "Our key insight is that motion boundary serves as a strong signal for object-level grouping". This is not at all a new insight. You are just using this very old insight in a novel way.
Our idea builds upon numerous prior works on object discovery, and we did not intend to claim this as a novel insight. We will revise the sentence to: “We were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos.”

Introduction: "we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual representations." This statement also seems rather bold. There is a long history of works (pre-dating the deep learning era) of using motion to segment objects or infer the 3-D structure of the world. I think you should replace "emergence of object-level semantics in visual representations" with something more specific.
We agree that the current statement in the introduction may appear too strong. We fully acknowledge the rich body of prior work leveraging motion cues for object segmentation and 3D structure inference. We will revise it to “we are first to demonstrate the effectiveness and scalability of using motion as an unsupervised supervisory signal on a large-scale dataset and modern model architectures”

You frequently use "Developmental Neuroscience" when "Developmental Psychology" is more correct.
Since our work mainly addresses cognitive aspects, we agree that 'Developmental Psychology' would be a more appropriate term. We will revise the text accordingly.

Occasional grammar problems
We are grateful to the reviewer for pointing out these grammar issues. We will thoroughly review the manuscript and make the necessary corrections.

For future work I'd invite the authors to think about replacing the off-the-shelf optic flow algorithm with a learning-based framework.
We sincerely appreciate your valuable suggestions regarding future work directions. In our current method, optical flow estimation is separated from model training. This was a deliberate choice to first test the basic effectiveness of our framework using a simple and reliable setup. However, we agree that an end-to-end learning framework is a promising direction. We are exploring ways to train the optical flow estimator together with our model, so that the main task can help improve flow estimation, and better flow can in turn improve the model’s performance and generalization.

Dear Reviewer dAst,
We greatly appreciate your valuable comments and questions. Our responses are as follows.

Questions about SimMIM
For all methods compared in our paper, we utilize the official pretrained weights. Unfortunately, SimMIM does not provide models smaller than the Base version. Here we report results for the Base version of SimMIM on depth estimation.

SimMIM adopts a dense pretraining strategy, which shows advantages on tasks like depth estimation and occupancy prediction. However, on 3D object detection, our model significantly outperforms SimMIM.

Our model leverages all available autonomous driving data. We believe that access to more diverse datasets could further improve its performance and generalization ability.

How does the method perform in other 2D downstream tasks, such as 2D semantic/instance segmentation?
The following table presents the results of semantic segmentation using UperNet, 2D detection and instance segmentation using Mask R-CNN on the nuImages dataset.

As shown in the table, our method achieves comparable performance on 2D detection and instance segmentation tasks, but also shows noticeable shortcomings in semantic segmentation. Consequently, features learned by our model are instance-discriminative rather than category-discriminative. However, as demonstrated in paper's Table 3, our features are complementary to class-level features.

Thanks for the thoughtful response. The experiments have resolved my concerns about downstream tasks and performance. I will raise my score.

Dear Reviewer 9fwV,
We greatly appreciate your valuable comments and questions. Our responses are as follows.

Extension beyond just driving scenes: How does the method behave when trained on other natural scenes?

We use driving data because current optical flow models perform better in such environments. In real-world datasets like Ego4D, where high-quality optical flow data is lacking, flow estimation remains unreliable. However, as discussed in paper's Sections 5.1 and 5.2, our method shows reasonable generalization to natural scenes.

How does the approach behave under different parameter combinations & optical flow models?

The following table shows the depth estimation results of the Swin-T model after 200 epochs of pretraining with different values of $\\theta\_f$​.

Here, we present the results of some downstream tasks with the margin set to 0.5, 1.0, and 1.5, trained for 100 epochs on one-quarter of the dataset using Swin-T as backbone.

According to the ablation study, our model demonstrates robustness to hyperparameter variations when trained on large-scale datasets, except in extreme cases—such as setting the margin to 0, which indicates infinite clusters.

We chose VideoFlow specifically because it performs joint multi-frame estimation, which provides better temporal consistency compared to two-frame-based models such as RAFT. This is particularly beneficial for long-term motion tracking in videos.

However, our method is not limited to a specific optical flow model — it can benefit from improvements in flow estimation. For example, we are actively exploring more robust approaches like CoTracker $4$ , which offer enhanced pixel-level tracking across time.

How does it compare to other video SSL approaches and other DINOv2 sizes?
Additional results on DINOv2 are provided in the supplementary materials.

We tested the TimeT $3$ model on depth estimation and BevFormer v2(704 × 256 resolution). The official TimeT GitHub repository only offers the ViT-S model. The following table shows the comparison results for the Small model.

As shown in the table, since the TimeT model was not designed or trained for large-scale pretraining, it exhibits a significant gap compared to existing methods on both tasks. Nevertheless, these works remain valuable, and we will include a discussion of them in the paper.

$4$ Karaev, Nikita, et al. Cotracker: It is better to track together. ECCV 2024.

Dear Reviewer 6akB,
We greatly appreciate your valuable comments and questions. Our responses are as follows.

Novelty
Our idea builds upon numerous prior works on object discovery, and we did not intend to claim this as a novel insight. Our main contribution lies in demonstrating the effectiveness and scalability of this idea on a large-scale dataset and achieving comparable performance to foundation models such as DINO on downstream tasks.
We will revise some sentence in our paper, for example:

1. "Our key insight is that motion boundary serves as a strong signal for object-level grouping." will be revised to “we were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos.”
2. "we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual representations." We will revise it to “We are first to demonstrate the effectiveness and scalability of using motion as an unsupervised supervisory signal on a large-scale dataset and modern model architectures”

Is there a curation process that the data has to go through to be able to generate reasonable clusters?

Since the nuPlan dataset provides camera distortion parameters, we applied undistortion to all images when processing the nuPlan data. Apart from this, all procedures are exactly as described in the paper(Lines 173–191) and Section A of the supplementary material, with no additional processing involved. We retained all samples with at least two pseudo-label(i.e. at least one foreground cluster) since samples that cluster all pixels into a single group have a loss of 0 and are not useful for training.

The clustering is done based on flow information. What if the data contains video with low motion/ or videos of scenes with no specific objects, how do instances like these affect the clustering?

The video contains low motion or no objects which failed to cluster instances will be filtered out during preprocessing, as described in Lines 189–191. Given the large scale of the dataset, the model still learns generalizable representations, even with these less informative frames removed.

Qualitative examples
Additional Qualitative results are provided in the supplementary material. Unfortunately, due to system limitations, we are unable to include more during the rebuttal phase.

Writing
We appreciate your suggestions. We will continue to improve our paper and address these writing errors.

Dear Reviewer h9dF,
We greatly appreciate your valuable comments and questions. Our responses are as follows.

Flow models

We chose VideoFlow specifically because it performs joint multi-frame estimation, which provides better temporal consistency compared to two-frame-based models such as RAFT. This is particularly beneficial for long-term motion tracking in videos.

However, our method is not limited to a specific optical flow model — it can benefit from improvements in flow estimation. For example, we are actively exploring more robust approaches like CoTracker $3$ , which offer enhanced pixel-level tracking across time.

Evaluations on other tasks
The following table presents the results of semantic segmentation using UperNet, 2D detection and instance segmentation using Mask R-CNN on the nuImages dataset.

We conducted a linear probe evaluation on Cityscapes dataset with ResNet-50 as backbone.

As shown in the table, our method achieves comparable performance on 2D detection and instance segmentation tasks, but also shows noticeable shortcomings in semantic segmentation. This is because the features learned by our model are instance-discriminative rather than category-discriminative. However, as demonstrated in paper’s Table 3 and the table above, our features are complementary to class-level features.

The architectural difference
As described in lines 233–234, although our method is pretrained with an FPN, during fine-tuning on downstream tasks, only the pretrained weights of the backbone are used. This is because our pretraining task involves pixel-level supervision, which benefits from multi-scale features. We also conducted additional experiments at lower resolutions, as shown in Table 2 of the supplementary material. The results demonstrate that our method remains effective even at lower resolutions.

Comparisons with relevant works
We found it quite difficult to make fair comparisons with other relevant works under our experimental setup. Many works do not open-source their code $1$ or fail to provide downloadable pretrained weights $2$ . Nevertheless, these works remain valuable. We will include them in the discussion section of the paper and conduct comparisons once the official code is available.
We tested a video self-supervised learning model TimeT $4$ on depth estimation and BevFormer v2(704 × 256 resolution). The official TimeT GitHub repository only offers the ViT-S model. The following table shows the comparison results for the Small model.

As shown in the table, since the TimeT model was not designed or trained for large-scale pretraining, it exhibits a significant gap compared to existing methods on both tasks.

$1$ Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping. NeurIPS 2020

$2$ Liu et al., The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos. NeurIPS 2021

$3$ Karaev, Nikita, et al. Cotracker: It is better to track together. ECCV 2024.

$4$ Salehi, Mohammadreza, et al. Time does tell: Self-supervised time-tuning of dense image representations. ICCV. 2023.