Track-On: Transformer-based Online Point Tracking with Memory

We extend our sincere appreciation to the reviewer for their valuable insights and constructive feedback. We have responded to their questions and concerns, and we hope our explanations thoughtfully encompass the raised points.

W1. Missing Citations

We appreciate the reviewer highlighting the similarities with previous works. We have added citations in the paper (highlighted in blue), specifically in sections where we discuss the coarse-to-fine strategy (line 144) and the feature drift (line 215).

W2. Analysis of Spatial Memory

We have adjusted our claim in line 78 to clarify that Spatial Memory helps mitigate tracking drift rather than completely “addressing” it. Additionally, we conducted a thorough analysis of the effect of spatial memory on drift, detailed in Appendix D (highlighted in blue). We designed two metrics:

• Similarity Ratio Score: This metric evaluates the similarity of the updated features to the ground truth features in the feature space. Our analysis showed that features updated using spatial memory have a higher similarity ratio score—improving by 20% over the initial query features—indicating better alignment with the target. Please refer to Figure 12 in Appendix D for a visual example illustrating how the similarity ratio score evolves over time across different tracks.

• Matching Score: This evaluates tracking performance when using the updated query features directly for tracking. We found that in several cases, the initial query fails to track the target due to feature drift, whereas the updated features from the spatial memory successfully recover the track (Figure 14a–c in the Appendix D). Overall, the matching score improved by 32% with the use of spatial memory and consistently enhanced tracking performance across all videos in the DAVIS dataset (see Figure 13 in Appendix D).

  We also observed failure cases where spatial memory could not recover from drift, as shown in Figure 14d. This indicates that while spatial memory is effective in many scenarios, it may not fully address tracking drift in all cases.

W3. Continuity of Temporal Positional Embeddings

We present an analysis of the learned temporal positional embeddings in Appendix E. We utilize learnable temporal positional embeddings in our model to help it differentiate between different time steps, similar to the approach used in learnable spatial positional embeddings of ViTs:

• Guaranteeing Continuity: To ensure continuity when extending the sequence length during inference, we apply linear interpolation to the learned temporal positional embeddings. This, similar to the resizing of positional embeddings in ViT, generates new embeddings that smoothly transition between learned positions, preserving temporal order.

  Example: Given learned embeddings of size 3, $[p_0, p_1, p_2]$, we extend them to size 5 as $[p_0, \dfrac{p_0 + p_1}{2}, p1, \dfrac{p_1 + p_2}{2}, p_2]$. This linear interpolation maintains the relative positions and ensures continuity between embeddings.

• Importance of Positional Embedding: We conducted an experiment to assess the impact of removing temporal positional embeddings, provided in Table 10 in Appendix E, and below for your reference. Without positional embeddings, the model’s performance drops by 1.8 points in AJ, 1.1 points in $\delta^{x}_{avg}$, and 2.1 points in OA. These results demonstrate that positional embeddings are crucial for capturing temporal dependencies.

• Visualization and Analysis: We visualize the temporal positional embeddings by projecting them to 3D with PCA (Figure 15, Appendix E; in blue). We first show that ​the model learns temporal order, as seen from the color changing smoothly from the most recent to the oldest. Then, we extend the embeddings to longer time steps and show that the temporal order is preserved after interpolation.

Q1. Explanation of Fig. 4 (Drift Plot)

Yes, the reviewer’s understanding is correct. We sampled from the ground-truth position and calculated the similarity with the initial query feature. Formally, we plot the scaled dot product between the initial query $\mathbf{q}^{init}$ sampled from the query frame, and the features sampled from the ground-truth position $\mathbf{p}_t$ on the feature map ($\mathbf{f}_t$) at time t: $\mathbf{q}^{init} \cdot \text{sample}(\mathbf{f}_t, ~\mathbf{p}_t)$

Q2. Typo

Thank you for identifying this typo. We fixed it in the paper, $\gamma$ denotes the learnable, temporal positional embeddings.
 

Tables

Table: Temporal Positional Embeddings

We are grateful for the reviewer's feedback and perceptive remarks. We have provided responses and hope these will resolve any issues you've raised.

W1./Q1. Ablation on Critical Modules

Based on the reviewer’s suggestion, we conducted extensive ablation studies on critical modules of our model. Detailed results are presented in Appendix C (Fig. 11, Table 8), and below for your reference:

• Number of Layers in Query Decoder, Offset Head, and Memory Write Module: We tested configurations with 1 to 4 layers. The best performance was achieved with 3 layers. For instance, increasing to 4 layers decreased occlusion accuracy (OA) by 0.8, indicating potential overfitting due to increased model complexity.
• Number of Scales: We varied the number of scales from 1 to 4. The model with 4 scales outperformed others, confirming the effectiveness of multi-scale features in capturing spatial information.

Due to time constraints, these models were trained for 50 epochs instead of the full 75, but the trends provide valuable insights. Please refer to Appendix C for more detailed analysis.

W2./Q2. Fixes in the Paper

We have clarified the notation and revised the caption of Figure 5 to introduce the content from left to right (highlighted in blue). Additionally, we clarified the color of the equation 9 (line 310)

W3. Speed of Online Tracking

Run-time speed is critical to online point tracking applications. If we understand correctly, we already reported Track-On's inference speed in Fig. 7 of the main paper by varying the memory size. While large enough memory is critical to accuracy (Fig. 6), our method offers reasonable trade-offs between accuracy (AJ; Fig 6) and speed (FPS; Fig. 7) with real-time options (~20 FPS).

W4. Comparison between Online and Offline

While a performance gap is expected due to the inherent advantages of offline methods utilizing future information, our method nonetheless achieves competitive results—even surpassing offline methods on some datasets—while operating under stricter constraints.

W5. Analysis of Other Online Methods

While only a few existing methods, namely TAPIR, BootsTAPIR, and DynOMo, report results in the online setting, they are not specifically designed for online processing.

• Combining existing offline methods, TAPIR performs iterative updates (PIPs), starting from a rough initial estimate (TAP-Net). The online version of TAPIR was first introduced in RoboTAP for robotics tasks that require online processing. To adapt TAPIR for online use, it is re-trained with a temporally causal mask, processing frames sequentially on a frame-by-frame basis.

• BootsTAPIR is a scaled-up version of TAPIR, trained on 15M real images to enhance performance. However, it still follows the same underlying approach and is not inherently designed for online operation.

• DynOMo introduces an expensive test-time optimization method, which takes approximately 45 seconds to process a single frame. This significant computational cost makes it impractical for real-time applications that require prompt responses.

• We have also evaluated other offline methods, e.g. CoTracker and SpatialTracker, under the online setting, as explained in our paper (lines 360–366). However, these methods are not designed for online processing.

• In contrast, our proposed method Track-On is specifically designed for the online setting. We introduce novel memory modules that effectively capture temporal information, enabling accurate point tracking without relying on future frames. We clarified these points in the related work section, lines 492–496, highlighted in blue.

We hope these clarifications address your concerns. Thank you once again for your valuable feedback, which has been instrumental in improving our work.
 

Tables

Table: Query Decoder Layer Number

Table: Offset Head Layer Number

Table: Query Memory Writer Layer Number

Table: Number of Scales

We thank the reviewer for their constructive feedback, including the suggestion for additional ablations to provide deeper insights and a clearer comparison with other online models. We kindly ask if our rebuttal has sufficiently addressed your concerns and if the score might be reconsidered.

We sincerely thank the reviewer for their thoughtful feedback and have addressed each point below.

W1. Typo

We appreciate the reviewer pointing out the typo in line 289 (now, 291). We have corrected the mistake in the revised version, highlighted in blue.

W2. Backbone Comparison

Following the reviewer’s suggestion, we conducted additional experiments where we replaced the backbone of our model (ViT-Adapter) with the Residual CNN used by CoTracker. The results of these experiments, along with the number of learnable parameters and backbones for each model, are presented in Appendix C (Table 9), and below for your reference. Our findings are summarized as follows:

• Performance with ViT-Adapter Backbone: Our default model, utilizing the ViT-Adapter backbone, achieves the highest performance among the compared methods. Notably, it does so with significantly fewer parameters, approximately half that of CoTracker (23M vs. 45M) and one-third of BootsTAPIR (78M).

• Performance with Residual CNN Backbone: When we switched to CoTracker’s Residual CNN backbone, our model experienced a slight performance decrease (a reduction of 1.3 in AJ). However, even with this backbone, our model still outperforms other methods with more parameters, including CoTracker itself (our model: 61.6 vs. CoTracker: 55.9 in AJ).

These results demonstrate that our method’s superior performance is not solely attributable to the choice of backbone but is also due to our novel approach and effective use of the memory module. The comparison reinforces the robustness and generalizability of our method across different backbone architectures.

Q1. Longer Videos

We appreciate the reviewer’s suggestion to evaluate our method on longer video sequences from recent datasets. To address this, we have started incorporating the PointOdyssey dataset into our evaluation. We have reached out to the authors of CoTracker for guidance on data preprocessing and evaluation, and are currently awaiting their response. We will include these results in the final version of the paper or provide them during the rebuttal phase once we receive the necessary files.
 

Tables

Table: Backbone and Number of Learnable Parameters

We thank the reviewer for their insightful feedback and have addressed each point below. We hope that these clarifications address the reviewer’s concerns.

W1. /Q2. Common Failure Cases

We have identified two common failure cases:

• Points on Thin Surfaces: Tracking accuracy diminishes on thin surfaces due to limited spatial information. Increasing the feature resolution might alleviate this issue, although it would require additional memory overhead.
• Points in Uniform Areas: Uniform regions lacking distinctive visual features pose a significant challenge for point tracking. The absence of descriptive cues makes it difficult for the model to establish accurate correspondences. Training on large-scale real-world datasets can mitigate these challenges, as demonstrated by BootsTAPIR’s improved performance resulting from training on 15M real images.

We have included examples illustrating these failure cases across multiple datasets in Appendix F (highlighted in blue).

Q1. Results on Additional Datasets

Following the reviewer’s suggestion, we have evaluated our model on two additional datasets: RoboTAP and Dynamic Replica. The results are presented in Appendix B (Tables 4 and 5; highlighted in blue), and below for your reference. Our method outperforms all other models on these datasets, including offline methods, except for BootsTAPIR on RoboTAP in certain metrics (in AJ and $\delta^{x}_{avg}$). The superior performance of BootsTAPIR in those metrics is likely due to its extensive training on a non-public dataset of 15 million real images. Notably, our model surpasses all other models that were trained on publicly available dataset TPVid-Kubric. These additional experiments reinforce the robustness and generalizability of our approach across diverse scenarios on 6 datasets in total.

W2. Trade-off Between Memory Size and Inference Speed

As the reviewer correctly noted, there is a trade-off between memory size and inference speed. While this might seem like a weakness, it actually provides flexibility to choose models with varying capacities and speeds, including some real-time options (~20 FPS). Our analysis also shows that memory needs to function as a bottleneck because increasing memory size decreases performance beyond a certain point. One future direction could be to learn how to compress large temporal information before storing it in memory, to obtain faster inference even when storing more information in memory.

Q3. Differences with Other Transformer-Based Trackers

Our method fundamentally differs from other transformer-based tracking approaches like CoTracker and SpatialTracker, especially in how correspondences are established:

• Previous Methods: CoTracker and SpatialTracker process all frames within a temporal window simultaneously. They use transformers to iteratively update point features and correspondence estimations by sharing spatial and temporal information across frames. After several iterations, they regress the point coordinates based on these updated features. This approach requires multiple passes and access to all frames in the window, making it less suitable for real-time applications.
• Our Approach: We treat points of interest as queries in a transformer decoder and directly decode them to establish correspondences on a frame-by-frame basis. During training, we employ a classification loss, which enables our model to predict correspondences without the need for iterative updates. This allows our method to process each frame independently, without relying on future frames or multiple iterations.This key difference enables our method to operate online, processing frames sequentially as they arrive. This is necessary for real-time applications or scenarios where future frames are not available during inference. In contrast, methods that depend on iterative updates over multiple frames are not suitable for such situations because they require access to a window of frames.
   

Tables

Table: Results on Dynamic Replica

Table: Results on RoboTAP

We thank the reviewer for their constructive feedback, including the suggestion to enhance the paper with two additional datasets and a failure analysis. We kindly ask if our rebuttal has addressed your concerns and whether the score could be adjusted accordingly.

Dear reviewer miCL,

We have conducted an additional evaluation of our model using the PointOdyssey dataset, which includes longer videos (~2000 frames). For detailed insights, please refer to our response to Reviewer BkZB. We hope that this extensive evaluation across seven datasets with diverse characteristics sufficiently addresses your concerns.

Dear reviewers,

As the discussion period nears its conclusion, we kindly ask for a response to our rebuttal. We are happy to provide further clarifications.

Thank you.