Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision

Thank you for your positive review and insightful feedback. We are glad that you find our approach novel and conceptually interesting, which also achieves state-of-the-art tracking performance with strong generalization capabilities. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.

1. Discussions on the appearance and motion features

Thank you for your comment. When tracking similar-looking objects, motion features are most useful for determining object identity, while appearance features provide fine-grained details for precise segmentation. Once identity is determined, the remaining task is to leverage those appearance cues to refine the object mask. Therefore, appearance features also bring some benefit after combining with motion features, when tracking similar-looking objects. We will include these discussions in the revised paper.

2. Discussions on computation cost

Thank you for your comment. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. The significant improvement in tracking accuracy is worth this additional computational cost. We also discuss optimizing computational efficiency in Appendix B.4 and Appendix C.

3. Clarifying the visual elements in Figure 1

Thank you for your suggestions. In Figure 1(c), the green and red masks represent segmentation maps of different objects. The boxes are used to draw the reader’s attention to specific regions. The blue, green, and red boxes highlight the ground truth regions, correctly predicted regions, and incorrectly predicted regions, respectively. We will follow your suggestions to revise the caption of Figure 1 to be more clear.

Thank you for the valuable feedback. We are glad that the reviewer finds our analysis thorough and insightful, while the proposed method achieves good results. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.

1. Discussions on related work

Thank you for suggesting these papers[a,b,c]. We will cite and discuss them in the revised paper. We highlight that our work significantly differs from these studies[a,b,c] as follows.

• Track4Gen [a] tackles point tracking by training video diffusion models on labeled point trajectories, whereas our work explores pretrained video diffusion models for object segmentation without any tracking‑specific training.

• Diff‑Tracker [b] is built on image diffusion models with additional motion encoders to learn temporal cues. By contrast, our work directly explores the built‑in motion of pretrained video diffusion models without extra modules.

• VidSeg [c] performs instance‑agnostic video semantic segmentation that cannot distinguish different objects in the same category. It also requires maintaining and updating an additional KNN classifier to learn temporal changes during tracking. By contrast, our approach can distinguish even similar‑looking objects without extra components.

Apart from the above differences from [a,b,c], we highlight our contributions go beyond performance improvements, with unique insights and findings which are not discussed in [a,b,c].

• Our work is the first to reveal that features from video diffusion models at early denoising stages can capture temporal motions.
• Our work is the first to reveal that video diffusion models excel at tracking similar-looking objects without supervision.

Notably, the studies [a,c] are published in June 2025 (CVPR 2025), after the NeurIPS 2025 submission due in May. According to NeurIPS 2025 policy quoted as follows, "For the purpose of the reviewing process, papers that appeared online after March 1st, 2025 will generally be considered "contemporaneous" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work." these studies[a,c] are contemporaneous work to our submission.

[a] Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. CVPR 2025. [b] Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers. ECCV 2024. [c] VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models. CVPR 2025.

2. Comparison with recent self‑supervised methods

Thank you for your comment. To the best of our knowledge, we have compared with the latest self-supervised baselines available at the time of submission. If the reviewer is aware of a relevant self-supervised method published recently, please let us know, and we would like to compare it.

3. Clarification on computation cost

Thank you for your comment. We report the computation cost of our method in Section 5.3 (lines 290–293) and provide more details in Appendix B.4. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. We also discuss optimizing the computational efficiency of our method in Appendix B.4 and Appendix C.

4. Implementation details and evaluation results on three more standard datasets and tasks

Thank you for your comment. We describe the selection process for the YouTube-Similar and Kubric-Similar datasets in Section 5.1 (lines 215–220). YouTube-Similar is manually created from the YouTube-VOS dataset by selecting videos that contain multiple objects with similar appearances. Kubric-Similar is a new dataset generated using the Kubric simulator, with randomized object colors, sizes, and motions. We include all generated videos as Kubric-Similar. We will revise the paper for clarity.

Following your suggestions, we conducted experiments for human pose tracking on the standard JHMDB dataset. We also evaluated on two additional standard datasets for video object segmentation: YouTube-VOS [1] and MOSE [2]. Experimental results in the following table demonstrate that our method consistently improves tracking accuracy across diverse datasets and tracking tasks. We will include these results in the revised paper.

[1] Ning Xu, Linjie Yang, et. al. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.

[2] Henghui Ding, Chang Liu, et.al. MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV2023.

5. Discussions on the interpolation of segmentation maps

Thank you for your comment. Following prior work, we first generate segmentation maps at the feature resolution and then interpolate these segmentation maps to the original image size. While this lower-resolution process may lose details for objects with irregular or complex shapes, it is a common challenge in existing methods. Our experiments on multiple datasets (e.g., Figure 5) empirically show that our approach is effective for tracking objects in real-world videos.

Thank you for your valuable suggestions for strengthening this paper! We answer your questions below.

1. How does the proposed method make particular designs for the matching task?

Thank you for your question. The key contribution of our work is demonstrating that simply using representations from video diffusion models already enables the tracking of similar-looking objects, significantly outperforming all existing self-supervised methods. Our method does not rely on tracking-specific training or complex designs.

As noted by Reviewer aXR3, "The proposed approach is novel and conceptually interesting. It leverages video diffusion models in a previously unexplored way for tracking." While editing supervised training data or introducing tracking-specific modules can improve tracking performance, our work highlights that video diffusion models have already learned an inherent ability for tracking, and is worth publishing.

2. Contributions of Kubric-Similar dataset and Youtube-Similar dataset

To study the principle that video diffusion models excel at tracking similar-looking objects, it is necessary to conduct a study on a controlled dataset. We build Kubric-Similar, not to introduce complexity, but to disentangle the factors that are important for tracking. Kubric-Similar allows us to understand the weaknesses of existing self-supervised methods, which over-rely on appearance features. We then find that once video diffusion models are pretrained, their features can track identical-looking objects without tracking-specific supervision. Kubric-Similar also enables further analysis, such as our finding that video diffusion models capture motion cues in the early stages of the denoising process.

We also evaluate the problem on real-world videos. Since there is no existing video object segmentation dataset focused on similar-object tracking, we annotate videos from the YouTube-VOS dataset. We find that all existing self-supervised methods often fail in these scenarios, while our method significantly narrows the gap between self-supervised and supervised approaches.

3. Implementation details on Youtube-Similar

Thank you for pointing this out. We clarify the selection process and creation rubric of the YouTube-Similar dataset as follows. To ensure quality, three graduate students were hired to manually select videos according to this rubric. Only the videos that all annotators find qualified are included in the final YouTube-Similar dataset.

• Object similarity. To create a dataset with similar-looking objects, we first select videos that contain at least two objects belonging to the same category from YouTube-VOS.

• Video filtering. We exclude static videos from the pool obtained in the first stage. In such videos, spatial appearance features can serve as a shortcut for tracking, which may influence our evaluation of the motion features learned in model representations.

We will include these details in the revised paper.

Thank you for your positive review and insightful suggestions. We are glad that the reviewer recognizes our work as an important study for the community, for highlighting the challenging and understudied task of tracking similar-looking objects. We also appreciate the positive comments on the contribution of new datasets, the well written paper with clear motivation, and the impressive qualitative visualizations that demonstrate significant improvements across diverse video scenarios. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.

1. Clarification on video diffusion models

1.1 Discussions on video diffusion models

Thank you for your comment. Video diffusion models are often obtained by adding a temporal dimension to image diffusion models, using 3D architectures to capture temporal context. In this work, we study the widely used 3D UNet, which is built by inserting temporal layers into a 2D UNet. Our tracking method achieves significant improvement in tracking similar-looking objects based on these models. However, our tracking method is model‑agnostic: it relies on intermediate features from any pretrained video diffusion model, which may not be a 3D UNet. We will include these discussions in the revised paper.

1.2 Clarification on temporal layers in video diffusion models

Thank you for pointing this out. The key to effectively tracking similar-looking objects is learning temporal motions among frames. We adopt the widely-used 3D UNet from video diffusion models, which is typically created by inserting the temporal layers into a 2D UNet. Temporal attention and 3D convolution are two successful and widely used temporal layers in video diffusion models. We will follow your suggestions to clarify this part in Section 4.1 of the revised paper.

2. Discussions on model layers for feature extraction

Thank you for pointing this out. In Section 4.2 (lines 136-137), we define $n$ as the layer index for feature extraction. Following your suggestions, we will revise the paper to explicitly define $n_v$ and $n_i$ as the block indices for video and image diffusion models, respectively.

Our framework is agnostic to specific layers. Motion and appearance features can be taken from different layers, and the optimal layers for the two backbones do not need to be the same. We empirically extract motion features from the 3rd block of I2VGen‑XL and appearance features from the 8th block of ADM. These choices are noted in Section 4.5 (lines 189 and 194). For your reference, we report the effect of layers from video diffusion models on DAVIS dataset in the following table, which shows that features from 3rd block achieve the best tracking results. We will include these experiments and discussions in the revised paper.

3. Additional evaluations

3.1 Evaluation on additional video object segmentation datasets

Thank you for your suggestions. We conducted experiments on Youtube VOS and MOSE datasets as you suggested. Experimental results in the following table show that our method consistently outperforms the state-of-the-art DIFT method in both datasets. These results demonstrate the general effectiveness of our proposed methods across various video scenarios. We will include these results in the revised paper.

3.2 Evaluation on additional tracking tasks

Thank you for your comment. Yes, our method can be extended to other video label propagation tasks. Specifically, we conducted experiments on the human pose keypoints tracking task on JHMDB dataset, and report the experimental results in the following table. Experimental results show that our approach significantly outperforms baseline methods, demonstrating the effectiveness on tasks beyond video object segmentation. We will include these results in the revised paper.

4. Comparison with supervised methods

Thank you for your suggestions. We conducted experiments to evaluate state-of-the-art supervised models, including SAM2 [35], Cutiie [8], XMem[9], and STCN[10]. As the reviewer anticipated, self-supervised methods underperform the supervised ones. We will include these results in the revised paper.

5. Visualization of video examples

Thank you for your suggestions. We have prepared the video examples you suggested, showing that our approach captures consistent motion features and consistently outperforms baseline methods across different frames. However, due to this year's NeurIPS policy that prohibits external links or appendix materials during the rebuttal stage, we are afraid that we are unable to present video examples at this time. We will include them in the revised version of the paper to better demonstrate the effectiveness of our approach.

6. Implementation details about appearance features

Thank you for your suggestions. We will revise the paper to describe the appearance features in the main method section. Specifically, we will incorporate the explanation currently in lines 191–193 into the method section (Section 4.3) to be more clear.

3.2 Youtube-Similar

Our method significantly outperforms prior self-supervised methods in tracking similar-looking objects in real-world videos, as shown in Figures 1 and 5 of the main paper. We further compare it with supervised approaches below.

• Object sizes. While our method effectively distinguishes different objects, it underperforms supervised methods on small objects. This is partly because small objects occupy fewer pixels in the reduced-resolution feature space, making it harder to preserve their information.

• Boundary predictions. When our method underperforms the supervised counterpart, one reason is that although we predict the correct identity, the object boundary is not clear. We believe this is partly due to the resolution reduction in the feature used for propagation, which may cause the loss of fine-grained details near object edges.

• Occlusion. Our method outperforms self-supervised baselines in handling occlusion. As shown in Figure 1 of the main paper, where one deer is occluded by another during movement, our method correctly distinguishes object identities after occlusion, whereas prior self-supervised methods fail. We find that supervised methods outperform ours in some cases because they capture fine-grained details more effectively, such as the legs of the deer in Figure 1.

We will include these discussions and visualizations in the revised paper.

4. Discussions on Youtube-VOS and MOSE

Thank you for your suggestions. We report the results on YouTube-VOS videos that do not contain similar-looking objects in the following table. The results show that our method outperforms the state-of-the-art DIFT method in non-similar object scenarios.

Experimental results show that our method outperforms state-of-the-art self-supervised DIFT method on MOSE, as shown in the following table for your reference.

Thank you for your valuable suggestions for strengthening this paper! We answer your questions below.

1. Evaluation on VIP dataset

Thank you for your suggestions. Due to the limited time, we did not provide the results on VIP dataset during the first rebuttal period. Following your suggestions, we conducted experiments to evaluate our method on VIP. Experimental results in the following table show that our method outperforms baseline models on VIP dataset. We will include these results in the revised paper.

2. Contributions of Kubric-Similar dataset and Youtube-Similar dataset

To study the principle that video diffusion models excel at tracking similar-looking objects, it is necessary to conduct a study on a controlled dataset. We build Kubric-Similar, not to introduce complexity, but to disentangle the factors that are important for tracking. Kubric-Similar allows us to understand the weaknesses of existing self-supervised methods, which over-rely on appearance features. We then find that once video diffusion models are pretrained, their features can track identical-looking objects without tracking-specific supervision. Kubric-Similar also enables further analysis, such as our finding that video diffusion models capture motion cues in the early stages of the denoising process.

We also evaluate the problem on real-world videos. Since there is no existing video object segmentation dataset focused on similar-object tracking, we annotate videos from the YouTube-VOS dataset. We find that all existing self-supervised methods often fail in these scenarios, while our method significantly narrows the gap between self-supervised and supervised approaches.

3. Comparison with supervised tracking methods

Thank you for your suggestions. We compare the tracking results of our self-supervised method with supervised approaches below.

3.1 Kubric-Similar

Our work significantly improves the tracking accuracy of state-of-the-art self-supervised methods on Kubric-Similar by 35.1% (see Table 1 in the paper). Visualization results also show that our method successfully distinguishes identical-looking objects with varying colors, sizes, and motions, while all prior self-supervised methods fail. One example is visualized in Figure 2 of the paper. We further analyze key factors and compare with supervised methods below.

• Object sizes. Kubric-Similar includes moving balls of random sizes generated by the Kubric simulator. Despite its best performance among self-supervised approaches, visualizations show that our method can still be improved when compared to supervised methods. For example, when tracking small objects, we observe some pixels belonging to small objects may be misclassified as background.

To evaluate the impact of object size, we sort videos in Kubric-Similar by object size and split them evenly into three subsets: Large, Medium, and Small. As shown in the table below, our method performs best on large objects and worse on smaller ones. Supervised methods also show a similar trend, highlighting the general challenge of small objects in our Kubric-Similar. Notably, the performance gap between our method and SAM2 increases from 10.6% (Large) to 14.3% (Small), suggesting that object size is an important factor influencing tracking accuracy.

• Boundary predictions. Another performance gap with supervised methods lies in inaccurate boundaries in some cases. For example, with spatially adjacent objects, a small number of pixels near the object boundary may be mislabeled as belonging to the wrong object.

• Occlusion. We find that our method outperforms self-supervised approaches in handling occluded objects, achieving a $J_m$ of 86.2% compared to 59.5% by DIFT. However, it still underperforms the supervised method SAM2, which reaches 97.5%.

Thank you for the valuable feedback. We are glad that the reviewer finds our proposed tracking method interesting and effective, supported by experiments. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.

1. Paper scope and contribution of our empirical findings

Thank you for your comment. As you pointed out, our work empirically demonstrates that features learned by pretrained video diffusion models at early denoising stages capture motion dynamics. We highlight that one of our contributions is to be the first to report this property and validate its effectiveness for tracking through empirical experiments. Theoretical derivation is beyond the scope of this work, and rigorous theory remains an open challenge for the deep learning field.

2. Our method is fully self-supervised

Thank you for your comment. Our work uses diffusion models trained without any tracking-specific labeled data, and is self-supervised in terms of tracking.

As you suggested, we ran experiments comparing our approach with state‑of‑the‑art supervised methods, including SAM2[1], Cutie[2], XMem[3], and STCN[4]. Self‑supervised methods often underperform their supervised counterparts. Nevertheless, our method achieves the best tracking accuracy among self‑supervised approaches and significantly narrows the gap between self‑supervised and supervised methods. We will include these results in the revised paper.

[1] Nikhila Ravi, Valentin Gabeur, et al. Sam 2: Segment anything in images and videos. ICLR 2025.

[2] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. CVPR 2024.

[3] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson327 shiffrin memory model. ECCV 2022.

[4] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. NeurIPS 2021.

3. Consistent improvement across three additional standard datasets and tasks

Thank you for your suggestions. In the main paper, we have already shown significant performance improvements on three datasets, including standard DAVIS, YouTube-Similar, and Kubric-Similar.

Following your suggestions, we conducted experiments on three additional, widely-used typical datasets and tasks, including:

• Standard datasets for video object segmentation: YouTube-VOS [5] and MOSE [6].
• Standard dataset for human pose tracking: JHMDB [7].

Experimental results in the following table demonstrate that our method consistently improves tracking accuracy across diverse datasets and tracking tasks. We will include these results in the revised paper.

[5] Ning Xu, Linjie Yang, et. al. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.

[6] Henghui Ding, Chang Liu, et.al. MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV2023.

[7] Hueihan Jhuang, Juergen Gall, et.al. Towards understanding action recognition. ICCV2013.

4. Clarification on computation cost

Thank you for your comment. We report the computation cost of our method in Section 5.3 (lines 290–293) and provide more details in Appendix B.4. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. We also discuss optimizing the computational efficiency of our method in Appendix B.4 and Appendix C.

5. Discussions on fusion weight factor

Thank you for your comment. We conduct experiments to investigate the effect of the weighting factor $\lambda$ in Section 5.3 (lines 270–281). Experimental results show that, across a range of fusion weight $\lambda$, our fused representation consistently outperforms using either motion or appearance features alone in real‑world videos. These results demonstrate the effectiveness of integrating features from video and image diffusion models through a simple weighted fusion scheme.