DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO

Q. Although many contributions are provided, they are a little lack depth, and feel somewhat scattered, like a collection of bundled findings. (but we have to acknowledge the contributions' value)

Please see the answer 1 of our response to common issues.

Q. About the unclear writting

We re-wrote the Sec. 3.2 in our revised paper to improve clarity. Please see the answer 2 of our response to common issues.

Q. About the definition of M, N'? What is "local random patch of tracks"?

We have clarified the definition of M, N' in Sec 3.2 (L286-291) in our revised paper. In summary, during training, instead of using the entire frame as dense track, we randomly crop a small patch of size $N'=h' \times w'$ and supervise on this patch. Due to the fact that patchwise training limits spatial attention to the cropped region, neglecting the global context of the first frame, we introduce $M$ sparse anchor tracks, with starting positions uniformly sampled across the first frame. Thus, in the global attention, virtual tracks first cross-attend to these sparse anchors, followed by attention to the local patch. This strategy captures global motion while maintaining efficiency through reduced dense track size.

Q. It seems that the differences between the proposed Local-Global attention and the global one in CoTracker are a) use anchor tracks to attend virtual tracks. b) add local attention. Although a) can reduce the calculation consumption, b) needs additional calculation consumption. During evaluation, local-global even requires O(TKM + TLN^2) more calculation.

The computational cost for part (2) in Fig. 3 is actually $TK(N + N) + KT^2 + TK^2$, where $TK(N + N)$ represents the two cross-attention blocks. Initially, we used O notation, simplifying $O(TK(N + N))$ to $O(TKN)$. To avoid confusion, we’ve revised this notation to show the exact computational cost: $TK(N + N) + KT^2 + TK^2$. For our approach in part (3), the cost is $TK(N + 2M) + KT^2 + TK^2 + TLN$.

When the number of anchor tracks $M$ is much smaller than the number of dense tracks $N$, we effectively reduce the global attention cost by $TKN$, while local attention adds only $TLN$. Since $L$ and $K$ are of a similar scale, our Local-Global attention has the same overall computational cost as CoTracker’s global-only attention but achieves significantly better performance in dense tracking.

Q. The idea of using the spatial-temporal transformer is not new.

We would like to argue that we did not claim the spatial-temporal transformer is our contribution. Our keys contributions are the novel combination of global-local spatial attention (Sec. 3.2), the attention-based upsampler with Alibi (Sec. 3.3), and the log of depth change ratio representation (Sec. 3.4).

Q. The authors are suggested to provide the compilation cost of each module to verify the efficiency of TAPE3D. Are all inference experiments testing on a same machine? The information of the machine including GPU and CPU are suggested to provide.

We thank the reviewer for this insightful suggestion. All computational cost experiments were conducted on a machine equipped with a single NVIDIA A100 80GB GPU and an Intel(R) Xeon® Platinum 8275CL CPU @ 3.00GHz. Below, we present a detailed profiling of each component in our approach as well as previous methods when densely tracking 384x512 pixels of a 100-frame video. We also include the performance (EPE) on long-range optical flow prediction on the CVO-extended set.

Table R2: Runtime analysis.

For the above profiling, we measure the runtime of the main components for each method. For CoTracker, SpaTracker, and SceneTracker, due to memory limitations, these methods cannot process the entire 384x512 tracks in a single forward pass. Instead, we set the number of tracks to the maximum that fits within the same amount of GPU memory and perform multiple passes until all pixels are tracked.

DOT, being a two-stage approach, performs sparse tracking on approximately 8,000 points, followed by optical flow computation for each frame pair. We report the runtime of each stage accordingly. Although DOT demonstrates the lowest runtime, we observed a significant "flicker" effect in its optical flow predictions (see Fig. 6 and the videos in the anonymous website), caused by per-frame optical flow predictions.

We would greatly appreciate it if you could let us know if the above answers can address your questions.

Q. Regarding the training procedure, it's not clear why the authors train on such small number of videos (5.6K) if these are synthetic videos that can be generated at a larger scale at ease. It's also not clear adding other data sources, such as PointOddysey would improve performance on the TAPVid-3D benchmark.

While we could generate an unlimited number of synthetic videos, diversity is limited by the variety of scenes and objects in the Kubric simulation. Consistent with prior work [1, 2], we found that around 6K videos, enhanced with on-the-fly augmentations, provides adequate variations for training.

We also experimented with combining Kubric and PointOdyssey synthetic data for training our 3D dense tracker. However, this reduced performance on the real-world test set (TAPVid3D) -- a similar behaviour is also observed in 2D tracking benchmarks (https://github.com/facebookresearch/co-tracker/issues/44#issuecomment-1938502621). Since both datasets are synthetic, combining them has limited impact on real-video performance. We believe that fine-tuning on appropriate real video datasets is essential to improve real-world 2D/3D tracking performance.

We would greatly appreciate it if you could let us know if the above answers can address your questions.

[1] Cotracker: It is better to track together.

[2] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis.

[3] Train short, test long: Attention with linear biases enables input length extrapolation.

Q. While the paper is a valuable contribution with good results, the model design seems to be mainly a combination of ideas from SceneTracker and CoTracker. The paper does not clearly state which ideas are novel, and which are borrowed from these previous methods. Could you please describe which elements in the model design are novel and which are borrowed from prior work?

Please see the answer 1 of our response to common issues.

Q. Have you tried evaluating your method on TAPVid-DAVIS (first)? Does it obtain SOTA results on 2D tracking compared to CoTracker and BootsTAPIR?

We did include the 2D tracking performance on the TAPVid2D benchmark (covering DAVIS, Kinetics, and RGB-Stacking subsets) in Table 7 in the Appendix of the original version. We additionally include the performance of BootsTAPIR in our revised version. Our method achieves competitive results compared to concurrent methods specifically designed for 2D tracking. Notably, BootsTAPIR benefits from a large-scale dataset of 15M real-world videos to enhance sparse 2D tracking, whereas our approach focuses on the more challenging task of dense 3D tracking, trained only on synthetic data. We believe that fine-tuning our model on large-scale, real-world video data could further improve both its 2D and 3D tracking performance.

Q. It is not clear how the dense tracks from Fig. 3, (3), are computed and how they connect to the tokens $G^i_t$ of eq (1).

The dense tracks in Fig. 3 part (3) are initialized similarly to the sparse tracks in CoTracker, with the same initial position, depth value, and visibility. The only difference is the total number of tracks. In the sparse tracking setting, the number of tracks can be any arbitrary number but cannot encompass the entire frame's pixels, as this would significantly exceed GPU memory limits (80GB) for a single forward pass. In the dense tracking setting, however, we track every pixel in the reduced-resolution image, forming a dense grid of size $N=H/r \times W/r$ (See L.252-253 in our revised paper).

Sparse tracks, dense tracks, and anchor tracks all are encoded using the eqn. (1) so that each track will be represented by a set of token $G$. The token $G^i_t$ represents a segment of the $i$-th track at time $t$ and serves as an abstract term that can refer to sparse tracks, dense tracks, or anchor tracks, depending on the context.

Q. The eq. (3) seems mathematically incorrect or at least too vague in notation and lacking explanation. Please clarify how the proposed attention-based upsampling works and provide a corrected eq. (3) if possible. Please explain how the $\tau$ cross attention blocks from Fig. 4 operate. Are these multiple heads in parallel?

Thank you for your constructive feedback. The equation 3 is not mathematically wrong, yet we have re-written it more clearly to avoid confusion.

In the revised version, we have rewritten this section with more details (see Sec. 3.3 in the revised paper). Below, we briefly summary answer your concern:

• Feature map extraction: We first extract a coarse resolution feature map $\mathcal{F}\_{coarse} \in \mathbb{R}^{ \frac{H}{r} \times \frac{W}{r} \times D}$ and a fine resolution feature map $\mathcal{F}\_{fine} \in \mathbb{R}^{H \times W \times D}$ with our convolution encoder and decoder.

• Input to Cross-Attention: For each pixel $(u,v)$ in the fine-resolution map, its feature vector $\mathcal{F}\_{fine}^{(u,v)}$ serves as the query, while the corresponding $\kappa \times \kappa$ neighborhood in the coarse-resolution map $\{\mathcal{F}\_{\text{coarse}}^{(u'\_{j}, v'\_{j})}\}\_{j=1}^{\kappa \times \kappa} \in \mathbb{R}^{(\kappa \times \kappa) \times D}$ provides the key and value. Thus the fine-resolution feature map is refined by: $\mathcal{F}\_{fine}^{(u,v)} = CrossAttn\Big(\mathcal{F}\_{fine}^{(u,v)}, \{\mathcal{F}\_{coarse}^{(u'\_{j},v'\_{j})}\}\_{j=1}^{\kappa \times \kappa} \Big)$

• Operation of Cross-Attention: The cross-attention operation is applied to every pixel in the fine-resolution map (see the revised text for the a more detailed form of the attention scores with Alibi). After this step, the refined fine-resolution feature map is passed through an MLP to predict the weight map $\mathcal{W} = MLP(\mathcal{F}_{fine})$, where $\mathcal{W} \in \mathbb{R}^{H \times W \times (\kappa \times \kappa)}$. This weight map enables us to compute high-resolution tracking by taking a weighted average of the coarse-resolution tracks.

• Multi-Head Cross-Attention Blocks: In practice, we use a series of $\tau$ multi-head cross-attention blocks with Alibi bias [3].

Q. How the key parameters such as M, r, N' and L are chosen?

We empirically set these hyper-parameters to balance between the computational cost and model performance. More details regarding these parameters are included in the Appendix (L837-852).

Below we add the ablation study on $L$ and $r$. These experiments are conducted with our 2D version.

Table R1.1: Ablation on the local patch size $L$.

Empirically we found that using patches of size $L=6\times 6$ in the local spatial attention yields the best performance.

Table R1.2: Ablation on the downsample factor $r$.

A smaller upsample ratio ($r=2$) improves performance but increases runtime by about $6\times$, while a larger ratio ($r=8$) reduces performance. We select $r=4$ as a balance between performance and efficiency.

We set $M=9 \times 12$ and $N'=30 \times 40$ during training to fit the model in the GPU memory.

We would greatly appreciate it if you could let us know if the above answers can address your questions.

[1] Depth anything: Unleashing the power of large-scale unlabeled data.

[2] Depthcrafter: Generating consistent long depth sequences for open-world videos.

[3] UniDepth: Universal Monocular Metric Depth Estimation.

Q. Why do we need certain anchor tracks? What if we sample some certain points in the dense tracks?

Please see the answer 2 of our response to common issues.

Q. About the different between sparse and dense tracks

Please see the answer 2 of our response to common issues.

Q. What is N' and L, the meaning of these two parameters are not well defined and explained.

We have clarified the definition of N', L in Sec 3.2 in our revised paper. In brief,

• During training, instead of using the entire frame as dense track, we randomly crop a small patch of size $N'=h' \times w'$ and supervise on this patch (L287).
• To capture fine-grained representations of local relations among dense tracks, we apply self-attention within very small spatial patches containing L pixels (L300).

Q. If the depth is predicted from monocular video, the scale is not specified. How can we ensure their consistency across different frames?

Baseline methods that combine 2D track + depth estimators (e.g., CoTracker + ZoeDepth/UniDepth) heavily rely on the quality and scale-consistency of the input video depth. However, frame-wise depth estimators such as ZoeDepth or Unidepth lack scale-consistency, leading to inferior 3D tracking performance and the "jitter" effect (as illustrated in the videos on our anonymous website). In contrast, 3D tracking approaches like ours, SpaTracker, and SceneTracker are trained end-to-end for 3D tracking, enabling them to rectify depth across time even when the input depth video is inconsistent. As discussed in Sec. 3.4 of our paper, a key difference with prior work that further addresses this issue is to leverage a log-depth representation for depth, which is scale-invariant and reduces dependency on the depth map accuracy, thereby improving performance on real-world benchmarks where depth estimation is often inaccurate.

Additionally, advancements in consistent video depth estimation methods, such as DepthAnything [1] or DepthCrafter [2], can further mitigate this challenge.

Q. The attention upsampling method is quite related to the well known guided filter work. It is better to discuss the relationship with it.

Our attention upsampler shares a conceptual similarity with guided filters in the use of a neighboring window in the low-resolution image as guidance. However, in our approach, we explicitly define each pixel’s output in the high-resolution map as a weighted combination of a neighboring region in the low-resolution map, with weights predicted directly through attention layers and an MLP. In this sense, the attention block acts as guidance in our case. The attention blocks are fully differentiable and optimized via gradient descent like guided filters.

We also experimented with a baseline combining a convolution-based convex upsampler with a bilateral filter (a variant of the guided filter), but it did not yield significant improvements. Based on these observations, we argue that the attention-based upsampler is a more suitable and effective choice for this task, offering both flexibility and ease of learning.

Q. Why the 2D version is consistently better than 3D version in Table2? Is there any analysis on this unexpected results?

Our dense 2D tracking version performs slightly better than the 3D version in Table 2, though this difference is minor compared to the significant improvements that our method (both 2D and 3D) achieves over previous approaches. The slight drop in 2D performance for the 3D version can be attributed to the additional objectives involved in training, which include predicting 3D locations. To support this, the 3D model’s architecture incorporates depth correlation features and an additional head for predicting depth changes. These enhancements, while critical for 3D tracking, introduce slight trade-offs in optimizing pure 2D predictions.

Below, we address common questions raised by the reviewers.

Q1. Regarding the novelty of the method and each of the components.

Our paper tackles the critical yet underexplored task of dense, long-term, 3D tracking from in-the-wild videos—a capability vital for real-world applications such as robotics, augmented reality, and autonomous systems. Existing approaches struggle to simultaneously achieve reliable, dense, long-range 3D motion tracking in a feed-forward manner. As shown in Table 1, the closest comparable works, such as SceneTracker and SpatialTracker, are not designed for efficient dense tracking and are more than eight times slower than our approach. Our method is the first to overcome these limitations, achieving state-of-the-art performance across multiple tasks, including long-range 2D optical flow, dense 3D tracking, and 3D point tracking.

We introduced three key technical designs that are essential for adapting recent advancements in 2D and 3D point tracking into an efficient, dense 3D tracker.

1. A combination of global & local spatial attention for tracking (Sec. 3.2) that captures both global and local spatial interaction for dense 3D tracking. This intuitive design choice has not been explored in motion tracking before.
2. An attention-based upsampler enhanced with Alibi (Sec. 3.3), to obtain high-resolution dense tracks with sharp motion boundaries. This new architecture demonstrates noticable improvements over commonly used convolution-based upsampler.
3. We found that using the log of depth change ratio representation (Sec. 3.4) achieves the best 3D tracking performance compared to other commonly used depth representations. While various depth representations, including log-of-depth, have been explored in the literature, there has been no comprehensive comparison to date. Our work is the first to provide a definitive conclusion in the field of long-term motion tracking.

These contributions are carefully designed, well-motivated, intuitive, and essential for achieving our strong results, setting a new SoTA for dense, long-term 3D tracking.

Beyond state-of-the-art tracking, we demonstrate broader applicability by extending our method to non-rigid structure-from-motion (see Appendix), enabling recovery of both the camera pose and dynamic 3D structure from monocular dynamic video — capabilities unattainable with existing tools like Colmap [1] or DUSt3R [2].

We believe our contributions will inspire further research and applications in 3D/4D computer vision.

Q2. Regarding the definition of sparse, dense, and anchor tracks and the notation in Section 3.2: Joint Global-Local Attention

We thank the reviewers for highlighting the need for further clarification. In the revised version, we have improved these definitions for clarity. Please first refer to Sec. 3.2 for detailed explanation. Here we outline the key terminologies:

• Sparse track: Used in methods like CoTracker and SpaTracker, sparse tracks refer to tracking a set of $N$ arbitrary points. Typically, $N$ ranges from 1 to 40K points, constrained by memory limitations.

• Dense track: dense tracks refer to tracking every pixel from the first frame (or any other reference frame) across the entire video (see L149-L155: Problem Setup (Sec 3.)). In Fig. 3 (part 3), $N$ denotes the number of dense tracks, calculated as $N = H/r \times W/r$, as we track from a reduced-resolution image, then subsequently upsample to full resolution.

• Anchor track: A set of $M$ tracks ($M \ll N$) uniformly sampled across the entire image. In our method, anchor tracks are auxiluary tracks which are introduced to efficiently compute virtual tracks through cross attention. This novel concept plays a crucial role in enabling efficient dense tracking and facilitating end-to-end training. (see L290-L299).

We also clarify other important notations used in this section:

• $N'$: during training, instead of using the entire reduced resolution ($H/r \times W/r$) as dense track, we randomly crop a small patch of size $N'=h' \times w'$ and supervise on this patch (see L286-288 in our revised paper).
• $L$: refers to the patch size used in our local spatial attention mechanism (which is defined in the Patch-wise dense local attention in Sec 3.2), set to $L=6 \times 6$ in our experiments (see L300-305 in our revised paper).

[1] COLMAP (https://colmap.github.io/)

[2] Dust3r: Geometric 3d vision made easy.

[3] Sea-raft: Simple, efficient, accurate raft for optical flow.

[4] Raft: Recurrent all-pairs field transforms for optical flow.