Taming generative video models for zero-shot optical flow extraction

We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method "novel" and "interesting" (uSVM) and our empirical results and analyses to be "strong" and "detailed," and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below:

The baselines from other models (e.g., cosmos) are relatively weak.

Flow improvement is primarily measured by comparing to stronger, task-specific baselines such as SEA-RAFT (Table 2), against which our method achieves better performance. The results from Table 1 that showcase Cosmos and SVD separately demonstrate the challenge of extracting motion cues from large-scale generative models that lack controllability. We will integrate Cosmos and SVD results into the main TAP-Vid table (Table 2) in future iterations for a more comprehensive comparison. Further, we introduce a significantly stronger (though more architecture-specific) baseline, DiffTrack [1], that also utilizes Diffusion models, and note that our method still achieves higher performance.

The overall design only allows 2-view point-tracking in a single inference, while optical flow is usually evaluated on (semi-)dense pixels. I recommend authors to follow the standard optical flow settings and evaluate KL-tracing on Sintel, KITTI, and Spring.

Thank you for this important suggestion. As our main focus is a generalizable flow extraction method for real-world videos, we chose TAP-Vid as our main benchmark, following a recent trend [2, 3], as we believe it better captures a model's ability to generalize to challenging, in-the-wild scenes. To further improve our coverage, we also include results on Kubric (Table 2) from the same benchmark, which is a synthetic dataset similar to Sintel or Spring. We appreciate the suggestion for additional benchmarks and will consider expanding our evaluation in future work.

[1] Nam et al., 2025. "Emergent Temporal Correspondences from Video Diffusion Transformers."

[2] Jiang et al., 2023. "Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow."

[3] Karaev et al., 2024. "CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos."

Wait, something is a bit confusing here for us. KL tracing (and the other methods we investigate in our paper) are all optical flow methods, not tracking methods. That means, by definition, they apply to the 2-frame case. We set out to solve the problem of optical flow, not multi-frame tracking, to begin with. Doesn't that mean the usual 2-frame case is the proper evaluation for our question setting? (Maybe it was confusing when we used the term "tracing" with KL-tracing? It wasn't meant to imply that we were attempting to build a tracking solution, but maybe the words 'tracing' and "tracking" sound similar and that was confusing?)

One good way to think about this is: the 2-frame setting is the "core dynamics understanding" problem. It's harder than the multi-frame case. And thus all methods, whether they are co-tracker or KL-tracing or RAFT or whatever, will naturally post lower absolute numbers in the 2-frame setting as compared to the multi-frame setting, where it's possible to average over information from the multiple frames to boost the numbers. The 2-frame setting isn't "unfair", it's just different, and equally important, to the multi-frame setting.

We've shown, with our 2-frame co-tracker comparison, that the supervised co-tracker method doesn't have some magical ingredient that makes it better at the base, hard optical flow case. (That's what we though you would care about, since our whole paper is in the optical flow domain rather than tracking, to begin with.) In other words, we've shown that whatever higher numbers Cotracker is getting in the multi-frame case comes from the additional information of the multiple frames, not some deeply better dynamics-understanding principle. That's a fair comparison for us to make, and allows us to put that method on the same footing as all the other optical flow methods that we investigate in the paper (not just KL-tracing).

Are you are kind of saying we should have been trying to solve multi-frame tracking the whole time, and that you think the 2-frame case is somehow "less valid" and really shouldn't have been what we were trying to go for in the first place? Like are you saying that the multi-frame tracking case is somehow the "real standard" that we all should be testing against?

Or is your point that we should be evaluating a multi-frame version of the KL-tracing method -- and all the other methods we investigate -- against multi-frame trackers? Would that mean you think we should as part of this current paper have developed a multi-frame version of all the various methods we were testing in our paper? That's actually a pretty good idea -- it probably should be done in the medium term, because typically a good 2-frame optical method can usually be turned into a good multi-frame tracker with some additional work -- but that seems to us to be a somewhat different research project than the one we presented (one focused on tracking rather than optical flow), and not the sort of thing one can do in the time frame of a rebuttal.

One question about All-Tracker -- it looks like a cool method, but isn't it a supervised architecture that is pretty task-specialized for multi-frame tracking? It seems like it would be pretty far afield to compare self-supervised zero-shot optical flow methods to supervised multi-frame tracking methods. I mean, naturally a supervised multi-frame model will be "better" in some absolute sense than a 2-frame zero-shot method, but is that comparison very meaningful?

We thank the reviewer for the thoughtful review and constructive suggestions! We address the main points below:

1. Diffusion Model Comparisons: We appreciate the reviewer bringing up existing diffusion methods. We acknowledge that several important attempts have been made in this direction, and we will include this in our related works and discussions.

After our submission, DiffTrack [1], a zero‑shot flow extraction procedure using a pre‑trained Diffusion Transformer, has been introduced. The authors include metrics from TAP‑Vid, and the method is shown to supersede many existing diffusion baselines, including DIFT [2]. The reviewer also brought up Track4Gen [3], which published comparable metrics that we present below (we note, however, that Track4Gen is not training‑free$^\dagger$, as their "zero-shot" evaluations are performed after they fine-tune the base model with trajectory annotations generated using optical flow).

We see that on the commonly reported metric $<\delta^x_{\text{avg}}$, which measures the percent of points that fall within a set of error thresholds, our method outperforms SOTA diffusion baselines. Comparing with Track4Gen [3], we outperform or are competitive on all metrics, despite being completely training‑free.

The reviewer also brought up additional concurrent work; however, at the moment direct comparison is challenging due to availability of code [4, 5]; difficulty in applying to non-generated videos (as mentioned in Sec. 6 of [6]); or differences in tasks [7], such as semantic correspondence vs. optical flow. Per the reviewer's suggestion, as public artifacts of previous methods become available, we will include more evaluations for comparison.

In sum, many existing diffusion baselines have already been superseded by DiffTrack [1], against which our KL‑LRAS method shows improved results. By including the latest, highest-performing baselines, we hope to better contextualize our results. We will continue to incorporate additional comparisons as methods become publicly available.

2. Speed: Extracting information from any large, generative video model is computationally demanding, as we discuss in Section 5. It just sort of "comes with the territory." However, it is still an important direction to take in order to leverage their deep understanding of scenes and motion. In fact, among the generative models we tested, our method is notably faster:

But as discussed in our section on limitations, the natural next step to take after successful flow extraction—as is common in the literature on using general foundation models to build deployable, task-specific models—is to distill it into a faster architecture that can output dense flow in real time. The main purpose of our method therefore is to be used as a sparse pseudo‑labeler. This will significantly increase domain diversity for flow datasets, which have traditionally been limited to synthetic, or heavily curated scenes, without incurring the intractable cost of human labeling.

Since submission, we have been actively exploring distillation strategies, where we too have been interested in understanding the effect of label sparsity. In particular, we have trained an existing, off‑the‑shelf dense flow model on a synthetic dataset with ground‑truth labels using different levels of sparsity. We find that a model trained on an extremely sparsified (0.03% of points labeled) dataset performs competitively to the same model trained on the original, dense dataset (endpoint error: 2.47 for dense, 2.63 for sparse), strongly encouraging us on the likely effectiveness of sparse distillation. Our goal is to have the distilled model ready for release with the NeurIPS camera-ready version, if accepted.

3. Exploration of different extraction procedures:

Experiments throughout the paper only cover one family of approaches (propagating intervention points).

To our knowledge, there are two main approaches for obtaining motion information from pre-trained models: (1) representation learning / mining approaches, as highlighted by the diffusion papers the reviewer mentioned, and (2) "propagating intervention points," as discussed in our paper. We acknowledge that our focus has primarily been on intervention-based approaches. This is because, if possible to execute, (2) is simpler, more generalizable, and naturally zero-shot. Also many other works have reported baselines on type (1) already — and which our results surpass. But ultimately, we agree with the reviewer's suggestion to expand the discussion of other branches of related work by including results from the latest diffusion methods (DiffTrack, etc.) from the table above.

4. Support for Diffusion models:

The proposed KL-based method ... does not directly apply to the prevalent model paradigm: diffusion models.

This is an important point. Though diffusion models are currently mainstream, and are great at generating photorealistic scenes, they have known difficulties with precise steering. Our results give a reason to continue exploring diffusion alternatives in future generative video models. (Other recent works provide independent reasons for this conclusion from different angles.) While methods that reinforce the use of a particular class of models that are currently popular can be valuable, we hope the same can be said for work that sheds light on complementary approaches.

Our results expose two fairly general properties that, if present, are likely to be sufficient for good tracing: (1) locality, and (2) full distributional predictions. These criteria, while exposing a weakness of current diffusion models, are consistent with other classes of models that are actively being explored in the field. In fact, that diffusion models have this weakness is one of the points we are trying to make — one which we believe is quite valuable for the community.

5. Comparison with tracking methods:

Limitations [do not] apply to optical tracking methods.

To add context to this discussion, we present comparisons with CoTracker‑v3, the latest SOTA point tracker.

Point trackers face similar limitations, requiring synthetic data due to labeling difficulty (as discussed explicitly in [8]). The above results show CoTracker-v3 run under the same two-frame setting on TAP-Vid DAVIS to ensure fair comparison. The lower performance of CoTracker in this more difficult setting shows that the underlying motion estimation still remains challenging for optical trackers. Per the reviewer's advice, we will include this, and BootsTAP results with relevant discussion.

6. Limitations: In the discussion, we do try honestly to mention the main limitation of our approach — the computational demand. We will further highlight this clearly in future versions.

7. Statistical Significance: The TAP‑Vid benchmarks do not include standard procedures for measuring statistical significance, and most papers do not report such results. We interpreted the corresponding checklist entry to have been met by closely following the conventions of our domain, though we appreciate the reviewer bringing this to our attention. Part of the reason why this is not as emphasized in this domain is because the datasets are fairly large, so standard error bars tend to be extremely small and a little bit beside the point. Nevertheless, we computed the standard error for the Average Distance (endpoint error) metric for each model that we ran, as well as a paired t‑test to verify that our method significantly outperforms other models. For example, a one‑sided paired t‑test shows that our method achieves significantly lower endpoint error than Cosmos (p = 6e‑08). We will report these numbers in the supplement if they are believed to be important.

[1] Nam et al., 2024. "Emergent Temporal Correspondences from Video Diffusion Transformers."

[2] Tang et al., 2023. "Emergent Correspondence from Image Diffusion."

[3] Jeong et al., 2025. "Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation."

[4] Wang et al., 2024. "COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing."

[5] Zhang et al., 2024. "Emerging Tracking from Video Diffusion."

[6] Xiao et al., 2024. "Video Diffusion Models are Training-free Motion Interpreter and Controller."

[7] Fundel et al., 2024. "Distillation of Diffusion Features for Semantic Correspondence."

[8] Karaev et al., 2024. "CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos."

The question about speed is important and I will let my colleague address that question separately. Here I wanted to address the issue of the "necessary" vs "sufficient" claims.

You're right that to totally mathematically tightly make the claim that diffusion methods could never under any circumstances do what our two "sufficient conditions" ask for would require a mathematical necessity proof. But that kind of mathematical certainty seems usually unobtainable in empirical machine learning -- the problems are just too complicated for that kind of proof.

Instead, what we can do is empirical studies that try to triangulate what is likely to be the case by gathering a bunch of evidence from different specific cases. Here, what we've done is provide a clear framework (The two conditions) for what seems likely to be important for being able to do this type of zero-shot extraction from a generative model. Our framework provides an understandable inductive reason why diffusion models might have trouble in this case -- because of the well-known issues they have with steerability -- we are far from the first to notice this problem, we just show evidence that this problem seems to affect the ability to do zero-shot optical flow. This doesn't absolutely prove the case (of course), but it does give decent inductive evidence for it.

It other words, we think it is likely that diffusion models have a weakness, and that if one wanted to solve the problem of zero-shot optical flow extraction from diffusion models, one would probably have to solve this weakness. I don't think we think (or claim in our paper) that it's impossible to solve those issues, but just that they probably have to be dealt with if one wanted to use diffusion for this purpose. Are you saying it would be better if we more thoroughly emphasized that this is an statement of "likelihoods" rather than "absolute certainties"? We would be very happy to do that if it's not coming across strongly enough in the current write-up.

Or are you saying a stronger thing -- that like maybe the inductive method of this kind shouldn't be deployed at all here? Like are you saying we should not make any suggestions at all about what seems likely (to us) to be true about diffusion until we have a air-tight proof of it somehow? That type of conclusion would seem like it would rule out a lot of useful investigation in empirical machine learning ...?

(Really sorry if we're not understanding your point yet... definitely not trying to give you a hard time, just trying to understand.)

We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method "novel" and "interesting" (uSVM) and our empirical results and analyses to be "strong" and "detailed," and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below:

The best performing setup relies on a specific model (LRAS), which somewhat limits the broader applicability of the method.

Could the authors discuss more concretely what would be required to make KL-tracing work with architectures beyond LRAS?

The key properties needed to make KL tracing work are (1) locality (local tokenizer enables detailed perturbation control) and (2) distributional prediction (predicting entire distributions instead of a single sample). Please refer to Section 3.5 for more detail. These features are model-agnostic and can be incorporated into a generative model's architecture in various ways. While the mainstream models of today (e.g., Cosmos, SVD) may lack these properties, and among existing models LRAS happens to have them, we believe that our results highlight an alternative direction for building powerful future generative video models, since the requirements that KL tracing suggest are not overly restrictive. For clarity, we will include further discussions outlining the general properties we have identified to work particularly well for KL tracing.

I believe the evaluations these two models should also be included in Table 2 for completeness.

We agree that placing SVD and Cosmos results directly in Table 2 makes the comparison clearer, and will add full results for both models in the revision. At the moment, to help with the discussion and contextualize diffusion-based extraction, we include TAP-Vid results from DiffTrack [1], a recent SOTA paper that uses a model-specific readout on Diffusion Transformers (DiT) for zero-shot optical flow:

This shows that flow extraction from diffusion models remains challenging compared to KL tracing used with LRAS.

How sensitive is KL-tracing to the magnitude and shape of the perturbation? Would a different form (e.g., a low-frequency mask or structured noise) work similarly?

We thank the reviewer for engaging with our work and proposing interesting ablations. We evaluated this using the same setting as in Table 1. For "shape" and "magnitude", we vary different parameters of our Gaussian bump perturbation such as amplitude ($\alpha$) and standard deviation ($\sigma$). For the low-frequency mask perturbation, we use a standard Gaussian blur kernel. We find empirically that our choice of a Gaussian "bump" perturbation is quite important, though the results are less sensitive to the specific parameters. However, we are open to the idea that a more focused investigation on the optimal perturbation may yield better results or a different form. Our results are below:

Finally, we greatly appreciate the concerns brought up about writing clarity. We will take the reviewer's advice and (1) include a more proper introduction of Counterfactual World Models, (2) do a heavy reorganization of the method section and (3) add more discussions directly related to our contributions.

[1] Nam et al., 2025. "Emergent Temporal Correspondences from Video Diffusion Transformers."

We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method "novel" and "interesting" (uSVM) and our empirical results and analyses to be "strong" and "detailed," and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below, highlighting transferability and additional tracking baselines:

The paper does not empirically demonstrate zero-shot transfer to significantly out-of-distribution domains.

The reviewer brings up an interesting point of transferability and generalizability. We believe that this is demonstrated, at least to some extent, in our work both quantitatively and qualitatively. Although the base LRAS model is trained on real-world videos, KL-LRAS, with no additional training, performs strongly on both real (TAP-Vid DAVIS) and synthetic (TAP-Vid Kubric) benchmarks (Table 2), demonstrating strong cross-domain generalization within the larger range of "normal" videos containing people and objects. Further, qualitative results in Figures 1, 6, and the attached video demo also show robustness on challenging, long-tail motions (e.g., motion blur or in-place rotation).

To be honest, however, we don't know if a model trained on YouTube videos will transfer to very different kinds of videos where notions of object and timescale are extremely different (like medical data). Regardless, our approach, KL tracing, works with any base predictor that is effective at its underlying prediction task, whatever the domain may be. Thus—and this is the most important point—the main advantage of our approach is that we can easily extend our method to handle any new domain merely by extending the training data of the base predictor model of which our approach operates on top. We don't need to get new labels for that domain, bypassing the intractable cost of obtaining human labels for anything other than synthetic, carefully curated computer-graphic-type scenes. As foundation models improve across domains, our method's performance scales accordingly without requiring domain-specific flow annotations.

We will make this point clearer, and also include further qualitative examples of out-of-domain "stress tests" such as performance on night-time videos (please note that we are unable to submit additional demos as part of this rebuttal).

The paper failed to compare with other tracking methods like cotracker3

We agree that this reference is informative and therefore include results for CoTracker-v3.

We note that the above results show CoTracker-v3 run under the same two-frame setting on TAP-Vid DAVIS to ensure fair comparisons with the flow and generative model approaches. The lower performance of CoTracker in this more difficult setting shows that the underlying motion estimation still remains challenging for trackers. Following the reviewer's advice, we will also add additional point tracker and flow baselines to our paper, as well as express the above point more clearly.

[1] Karaev et al., 2024. "CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos."

Dear Reviewers,

ACs have been instructed to flag non‑participating reviewers using the Insufficient Review button (Reviewers fTmd and uSVM). Please ensure that you read the rebuttals and actively contribute to the discussion.

Kindly note that clicking Mandatory Acknowledgement before the discussion period ends does not exempt you from participating. This should only be submitted after you have read the rebuttal and engaged in the discussion.

Best regards, AC