Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

R1 Q1: Learning motion clues from unlabed video is similar to the previous idea [1], I recommend that the authors cite and discuss this literature in the related work section.

R1 A1: Thank you for the suggestion. We will include a discussion of [1] in the related work section. That work focuses on learning video representations for video action recognition by reconstructing motion trajectories and object shapes from sparse video patches, using ground-truth motion trajectories extracted via traditional optical flow methods.

By contrast, our work serves a different purpose: we are not aiming to learn a video representation for video action recognition. Instead, we propose a method to extract optical flow signals from a world model that has already been trained to predict future RGB frames. Our approach is self-supervised in the sense that it does not require external flow annotations or supervision. In fact, the flows extracted via our method could be used as training data for models like [1], providing a viable alternative to using other available off-the-shelf flow algorithms.

---

R1 Q2: The proposed parameterized counterfactual probe generator is trained in an unsupervised manner without direct supervision, which means the generalization ability of the probe generator is still restricted by the heuristic used to generate the training data. Since the probe generation process is similar to the image editing task, the authors can try to use off-the-shelf image editing methods like [2] to generate probes.

R1 A2: Thank you for the suggestion regarding the use of off-the-shelf image editing methods such as [2]. However, we would like to clarify that our probe generation process is fundamentally different from image editing, particularly from text-driven editing pipelines like [2]. Our method does not aim to generate semantically meaningful image edits. Instead, we apply a small, image-conditioned, localized perturbation (a learnable patch) to the input frame and analyze how this perturbation propagates across time through the model's predicted next frame. This allows us to estimate the local optical flow.

---

R1 Q3: Firstly, the quality of the generated next frame depends on the pre-trained world model. Once its capacity can not match the scale of the training data, the flow model performance may be hindered by the sub-optimal synthesized frames.

R1 A3: Thank you for your comment. We agree that the quality of the generated next frame depends on the capacity of the underlying world model. As noted in prior work on CWM [2, 23] as well as results in our paper, increasing model capacity consistently improves performance on flow estimation. While it is possible that such gains may eventually saturate, current results suggest we have not yet reached that point.

We therefore believe it is premature to conclude that motion estimation will be fundamentally limited by the fact that the model's performance may saturate at larger scales. Rather, we view this as an open question for future work. For now, our approach is justified by the empirical improvements observed at the scales at which the current CWM model is trained.

---

R1 Q4: Secondly, to calculate the position of the probs in the next frame, the authors apply a heuristic method that calculates the difference image and finds its peak. This means the inserted probs should be sparse and may be affected by occlusion of different probs; thus, the number of reference points can not be scaled up in an example. Although the proposed training pipeline neglects the number of available probs, it may still face the challenge of training efficiency.

R1 A4: We agree with the reviewer that inserting multiple probes into the same image can lead to interference effects such as occlusion and overlapping influence regions, which makes simultaneous probing challenging. However, our current implementation addresses this by applying a single perturbation per image and executing all probes in a batched and parallelized manner. This allows for efficient extraction of flow signals without compromising the integrity of individual probes.

Furthermore, once we extract sparse flow supervision from the model using these probes, we can train a separate supervised optical flow estimator (e.g., SEA-RAFT) on this data. As shown in our paper (Table 2, right), these distilled models are not only significantly faster at inference time, but also achieve good performance, demonstrating that our approach is practical despite the sparsity of individual probe. We summarize these results below, and present an additional experiment where we distill Opt-CWM into the newer DPFlow [A] model, which achieves better performance.

[A] DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework - Henrique Morimitsu, Xiaobin Zhu, Roberto M. Cesar Jr., Xiangyang Ji, Xu-Cheng Yin

R2 Q1: The proposed method uses two large base predictors, one with 175M and the other 1B learnable parameters, which makes the comparison not very fair over the existing methods.

R2 A1: We acknowledge that the capacity of our model differs from some baselines. Our base CWM is a general‑purpose video prediction foundation model (used here to extract motion, but can also be used to extract depth/segments), which justifies the larger parameter count. Regarding fairness at inference time, we follow a standard approach: distill Opt‑CWM into a small, task‑specific optical‑flow model. Concretely, Table 2 (right) shows that SEA‑RAFT distilled from Opt‑CWM (trained only on pseudo‑labels, no ground‑truth flow) outperforms SMURF (self‑supervised) and is competitive with supervised RAFT, while running at the small‑model speed. We summarize these results below, and present an additional experiment where we distill Opt-CWM into the newer DPFlow [A] model, which achieves better performance. It is also worth noting that this distillation procedure can also be further tuned and result in higher final performance. We will communicate this more directly in the paper and discuss compute cost details.

[A] DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework - Henrique Morimitsu, Xiaobin Zhu, Roberto M. Cesar Jr., Xiangyang Ji, Xu-Cheng Yin

---

R2 Q2: Due to the introduce of the large models, the inference time of the proposed method should be much higher than the existing methods. It would be great to see the comparison on this axis.

R2 A2: The size of the base predictor model does in fact lead to slower inference speed. We describe our distillation-based solution to this added inference time cost in R2 A1 which will have the same inference-time speed as standard optical flow methods. We commit to adding inference time details in the final version.

---

R2 Q3: To address the limitation on high inference time, it is stated that "We demonstrate a pathway to resolving this by distilling the trained Opt-CWM into small and efficient architectures". However, I do not see such results.

R2 A3: Thank you for raising this, those results are in Table 2 (right) of the main text, but we realize our text could emphasize them better. Please also refer to our discussion in R2 A1.

---

R2 Q4: What is the impact of the scale of the base predictors? Currently two models of 175M and the other 1B are employed. How about using smaller or even larger models? It would be great to see the trend of the performance with respect to the model size.

R2 A4: We acknowledge that establishing scaling laws for base predictors is an important topic. Within our compute budget, we choose to train and evaluate at two orders of magnitude (175M, 1B) and show improvements of 47.53 $\to$ 51.88 for AJ (higher is better) and 8.73 $\to$ 7.70 for AD (lower is better). A complete scaling study (smaller/larger CWMs) is a promising direction and we plan to include it in future work.

---

R2 Q5: Many sentences are too long and complex, making the paper hard to read and follow.

R2 A5: Thank you for this helpful feedback, we will revise the draft for clarity and improve the sentence length and structure.

R3 Q1: Could the authors provide more detailed analysis or ablation experiments showing how the model distinguishes between perturbations that correspond to moving/visible objects versus those that are static or occluded? Is there a theoretical or empirical criterion to ensure that significant model output changes are reliably associated with true object motion, and minimal changes indicate occlusion or background?

R3 A1: To clarify, we briefly re-state the key “theory” behind our approach. Our counterfactual compares the predictor’s next‑frame output with vs. without a small, image‑conditioned perturbation localized at the query pixel in the first image. The resulting difference image $\Delta$ reveals where and how strongly the model carries information from that pixel into the next frame.

This concept leads to different behavior for different points, depending on the condition:

• If the point is visible and moving... then the perturbation appears at a new location in the next frame; $\Delta$ exhibits a sharp, high‑magnitude peak with some displacement from original coordinates.
• If the point is visible and static... then the perturbation appears in the same position in the next frame; $\Delta$ has sharp, high magnitude peak but with near‑zero displacement.
• If the point becomes occluded/out‑of‑view... then the model cannot “carry” the perturbation forward; $\Delta$ has low magnitude and a diffuse response indicating occlusion.

This behavior is agnostic to foreground/background: background points can move under camera motion (and we detect that as a displacement). We would like to emphasize that we do not claim that minimal change is equivalent to being on the background. Minimal change indicates occlusion/out‑of‑view rather than background. But our motion method works both when the motion is due to camera motion (background) or object motion (foreground): the same calculation is accurate either way.

R3 Q1(b): The reviewer asks about an empirical criterion

R3 A1(b): Regarding an empirical criterion, the evaluation of Occlusion Accuracy (OA) and Occlusion F1 (OF1) already establishes this. We determine whether a point is occluded based on the magnitude of the $\Delta$ response, and observe competitive occlusion estimation performance on TAPVid First (DAVIS: OA 80.44, OF1 68.43; Kinetics: OA 80.74, OF1 76.21; Kubric: OA 90.11, OF1 72.56).

R3 Q1(c): The reviewer also asks about ablation experiments.

R3 A1(c): Our ablations also show that learned counterfactual probes help: we obtain better occlusion separation than fixed patches: OA 80.34 / OF1 50.06 (learned) vs. 75.38 / 27.21 (red square) and 76.89 / 19.10 (green square) on the 512px with MM‑3/MS‑2 setting.

---

R3 Q2 & Q3: The current layout places important figures (e.g., Figures 1 and 2) far from their detailed descriptions in the main text, which hinders readability. I recommend the authors to consider reorganizing the paper so that figures and their explanations appear together. Figure 2 does not fully reflect the described functional form in the main paper. In addition, it omits some input variables (such as, which may cause confusion about the model’s actual input-output structure.

R3 A2 & A3: Thank you for this suggestion. We will revise the organization of the paper and update Figure 2 according to this recommendation to help with readability.

---

R3 Q4: Does Opt-CWM achieve true optimality in generating counterfactual probes, or are there cases where it may still be suboptimal? Is there any analysis or discussion of the theoretical limits of the proposed approach?

R3 A4: This is an important point. We want to emphasize that we do not make any tight mathematical claims in our work. Our method has an intuitively clear idea for how to optimize the parameters of the function that predicts image-conditioned probes, based on the downstream flow-conditioned reconstruction objective of $\Psi^{`flow`}$. This is a principled surrogate for learning counterfactual probes that result in accurate motion estimates. Intuitively it makes sense why it should work, even though we don’t have a math proof.

With our joint optimization approach, probe estimation and sparse flow-conditioned next-frame reconstruction are coupled. While we do not have a formal lower bound of the coupling strength, Figure 3B directly examines the coupling, and finds that it is sufficiently strong to enable optimization of the probes.

And ultimately this concept pays off: while we cannot formally demonstrate global optimality, our learned image-conditioned probes consistently outperform hand-designed alternatives (Table 3 left).

---

R3 Q5: Can the authors provide a detailed explanation of all symbols and notations used in the tables, to enhance readability and reproducibility?

R3 A5: We provide information on the symbols used in our tables:

• S, U$^\dagger$, U: Supervised, semi-supervised, self-supervised, respectively. U$^\dagger$ is reserved for Doduo, which uses segmentation labels and thus is not strictly self-supervised.
• $**bold**$, $\underline{\text{underline}}$: 1st and 2nd in performance, respectively.
• shaded: Best performing supervised models are considered separately.
• $\ddagger$: Trained on Kubric. This is to separate GMRW results on Kubric.
• AJ, AD, $<\delta^x_\textrm{avg}$, OA, OF1: TAP-Vid metrics. Please refer to Section 4 for details on each metric.
• MM, MS: Multi-mask and multi-scale, which are inference-time enhancements. Please refer to Section 3.2 for details.

R1 Q1: The authors state that most recent works in point-tracking are semi-supervised (like BootsTAP, TAPNext etc.) and thus are out of scope for comparison. While I appreciate this, I feel it would be disingenuous not include some of these results in the paper for comparison to show where the SoTA is for these tasks. Methods like BootsTAP or TAPNext are extremely scalable because they use pseudo-labeling. Therefore, one might ask if the goal is to solve point-tracking through occlusions etc. generally then what is the inherent benefit of using something self-supervised if the performance is still drastically worse than those methods (see numbers in TAPNext paper Table 1 for example)? The main driver of self-supervised learning to me is it's generality and scalability to places where finding supervision is hard or not clean, but if semi-supervised learning can bridge this gap effectively should these tasks (e.g. TAP-Vid) be the main demonstration of the benefits of this approach?

R1 A1: We agree that this is important to achieve good results when compared to methods trained with more supervision. We believe we have achieved this, although it may be confusing since multiple protocols exist in the literature, e.g. Table 1 in the TAPNext paper reference uses a different (and somewhat easier) protocol than the one we report. On the standard but more relatively challenging benchmark that we report (please refer to evaluation details below), our method is SoTA, even compared to the semi-supervised methods like CoTracker, and performance numbers for all methods are lower overall.

Why self-supervised? Despite the recent strong results of semi-supervised learning, it is still fundamentally limited because it leads to a limited exploration of the possible label space of point tracks. Specifically, pseudo-labeling will further reinforce what the model can already predict confidently. Parts of the data distribution that are difficult to synthesize with graphics rendering engines (and therefore will not be in the initial training dataset before any pseudo-labeling is done) will not be learned with pseudo-labeling. Examples include non-rigid deformable motions (e.g., fabric, dough), or volumetric/particle motion (e.g., liquids, smoke, dust). Self-supervised learning through predictive modeling, like the Counterfactual World Modeling paradigm that we build upon in this paper does not have this limitation of pseudo-labeling and can learn such challenging scenarios from data.

Evaluation details: We follow the more challenging “first” protocol for TAPVid evaluation, whereas the TAPNext paper follows the easier “strided” protocol, leading to higher reported tracking performance. Briefly, in the “first” protocol, points are tracked from the first time they appear through the end of the video. In the "strided" protocol, a point is queried every 5 frames if it is visible and tracked forward and backward in time. First is more challenging because on average points are tracked over longer temporal frame gaps. For more information refer to Appendix H of the original TAPVid paper. We also run all models on pairs of query and target frames to track a point through a video. This is (1) to remove the effect of additional algorithmic heuristics on how to chain frame-pair predictions into tracks and to reveal the capability of the underlying motion representation, and (2) because this represents a fundamental unit of the problem of motion understanding–tracking directly emerges from estimating the motion/occlusion of a point across two frames

---

R1 Q2: The authors miss some key citations of work that needs to be compared to. For example DINO-Tracker from Tumanyan, Singer et al (ECCV 2024) is also self-supervised and seems to present quantitative results that are far better than what is reported in this paper. Can the authors include this comparison or comment on why it is not necessary?

R1 A2: We thank the reviewer for raising this important prior work, we will make sure to add this paper to the related work. Regarding a comparison, DINO-tracker is not a strictly self-supervised method because it assumes an off-the-shelf RAFT optical flow model to derive pseudo-labels, which has been trained with explicit optical flow supervision. In contrast, our approach is only supervised by a next frame pixel prediction loss. We will include a comparison with DINO tracker in the final version.

---

R1 Q3: There are some discrepancies in the numbers reported in this work vs. the literature. For example for DAVIS AJ and delta_avg, the authors report GMRW as 36.47 and 54.59. However in the GMRW paper they report: 41.8 and 60.9 . These kinds of discrepancies should be clarified in the work otherwise comparison across works becomes quite tricky. Is there something about the evaluation methodology used that differs from prior work for example?

R1 A3: We thank the reviewer for raising this point. As in the R1 A1 response, we would like to clarify that in this paper we follow the more challenging “first” protocol for TAPVid evaluation, whereas the GMRW paper follows the simpler “strided” protocol, leading to higher reported tracking performance. As discussed, our more challenging evaluation protocol evaluates models' capabilities at the fundamental problem of estimating the motion of a point across a pair of frames.