TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation

Dear reviewer 7a51,

Thank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the competitive results, low computation cost, and the first introduction of the segmentation model in point tracking tasks. We will address your concerns and questions in the response below.

---

[W1] The main contribution of our paper

Thank you for acknowledging our intuitive idea of using segmentation models to enhance point tracking. We list our main contributions, which are commonly acknowledged by the reviewers jQxW and BSTo, as follows:

• Designing a flexible model that can combine different pre-trained tracker and pre-trained instance segmentation models, not tied to a specific model
• A new idea of pruning input video frames for point tracking, which addresses a fundamental drawback in existing point tracking models caused by the video down-sampling structure
• A joint framework for the point tracking and the segmentation tasks, which bolsters the performance of the two tasks in a synergistic manner

As pointed out by the reviewer jQxW, our method addresses the fundamental limitation of losing high-frequency information crucial for point tracking, innovating the way current point tracking models handle high-dimensional video inputs.

As the segmentation improves point tracking in our framework, it can also enhance the segmentation task in a synergistic manner. Specifically, the enhanced fine-grained tracking performance (e.g., 1-pixel errors in Table 1) can bolster the segmentation task by providing more accurate queries for segmentation. While prior art (e.g., SAM-PT) contributed to enhancing the segmentation from the point tracking task, there has been no recursive structure similar to ours, and their performance gain is only a one-off.

Finally, we note that our method does not require fine-tuning, and is pluggable to different existing models (Table 2). It is also worth noting that the efficacy of our pruning method is significant even if the segmentation model is not employed (e.g., 57.5 -> 62.5 AJ in Table 3).

---

[W2] Comparison to SAM-PT

Despite the margin being relatively small in the segmentation task, we note that our primary focus is point tracking where our framework demonstrates significant enhancements.

Nevertheless, we also find that jointly performing the tracking and segmentation in our framework is useful for the segmentation task as well, achieving improved results than powerful baselines, such as SAM-PT. These segmentation baselines, on the other hand, do not provide point tracking functionality.

We believe the impact of our finding is not limited to helping a tracking task, but can also inspire new methods for video segmentation by considering a synergistic effect between the segmentation and different tasks.

---

[Q1] Additional FLOPs comparison in a different baseline

Since our framework is a plug-in for a point tracking model, directly comparing FLOPs of our main TrackIME (incorporated with TAPIR) and other baseline (e.g., CoTracker) would not be a lateral comparison.

Nevertheless, we provide the additional FLOPs results for CoTracker processing 64 frames in three different input dimensions: 384, 768, and 1080 pixels at the shorter side, and our TrackIME incorporated with CoTracker, experimented in DAVIS-F. In a similar trend as our experiment based on TAPIR, the benefit from the larger input dimensions is limited since the baseline tracker is only optimized for low-resolution input frames (384 × 512) to meet the memory constraints, and it is non-trivial to process high-resolution inputs without fine-tuning. When incorporated with TrackIME, on the other hand, the model can better utilize high-frequency details without fine-tuning and in a computationally efficient manner, e.g., TrackIME vs. CoTracker (1080 $\times$ 1440) gives 6217G vs. 14138G (FLOPs), 64.5 vs. 62.2 (AJ), 79.2 vs. 76.7 ($\delta^{x}_{\text{avg}}$), 88.5 vs. 86.6 (OA), respectively.

We hope this answers the question and will include these results in the updated manuscript.

\begin{array}{l c | c c c } \text{Method} ~ (\text{Input Dim.}) & \text{FLOPs} & \text{AJ} & \delta^x_\text{avg} & \text{OA} \newline \hline \text{CoTracker} ~ (384 \times 512) & 2707 \text{G} & 60.8 & 76.1 & 86.0 \newline \text{CoTracker} ~ (768 \times 1024) & 7670 \text{G} & 62.3 & 77.8 & 87.1 \newline \text{CoTracker} ~ (1080 \times 1440) & 14138 \text{G} & 62.2 & 76.7 & 86.6 \newline \hline TrackIME ~ \text{with CoTracker} \newline (384 \times 512) & 6217 \text{G} & \mathbf{64.5} & \mathbf {79.2} & \mathbf{88.5} \newline & \newline \hline \end{array}

Dear reviewer BSTo,

Thank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the enabling high-resolution point tracking, the valid experiments for ablation, and the extended results in zero-shot video object segmentation. We will address your concerns and questions and provide additional results in the response below, which we will also include in the updated manuscript.

---

[W1] Employing more complex motion models for the trajectory aggregation

Thank you for the great suggestion. Although more complex motion models can satisfy cases such as rotations and scale changes, we find only marginal changes compared to our simple design, when applied to the trajectory aggregation step (Equations 3 and 4). For example, we provide the results employing an affine motion model (i.e., $T_t = A_t * T_{t-1} + b_t$) [R1] by fitting the least-square solution for $A_t$ and $b_t$ using the baseline tracking results, experimented on DAVIS-F benchmark.

We would like to emphasize that the trajectory aggregation is not required to predict accurate motion, since its purpose is only to determine the center of pruning windows. Even if the center escapes out of the object due to complex motions, it still suffices as long as the pruning window can contain the query point.

[R1] Li, et al. "An efficient four-parameter affine motion model for video coding." IEEE Transactions on Circuits and Systems for Video Technology 28.8 (2017): 1934-1948.

---

[W2] The use of visibilities as the confidence weights in Equations 3, 8, and 12.

Thank you for acknowledging the importance of confidence weights. We first would like to clarify that our strategy combines both the hard 0-1 visibility predictions as well as the confidence weights (e.g., see $\mathbf{o}_t \geq 0.5$ in Equations 3 and 8). This strategy effectively mitigates potentially erroneous confidences by the false positives, since our method tends to demonstrate a high precision (the portion of true positives) for the visibility classification, e.g., we get 93.7% @ threshold 0.5 in DAVIS-F. Our strategy is valid as far as a sufficient number of visible tracking points are available. For the cases where an object is occluded for a few frames and then reappears, our framework can maintain the number of tracking points via the point re-sampling, even if the visibility classifier fails to predict the reappearance.

To further support the validity of our strategy, we measure the average confidence of the true positives and the false positives and find 0.902 (true positives) and 0.737 (false positives), so the remaining false positives would be penalized through the weighted aggregation. We also provide additional ablation, where we force equal weights in the aggregation, experimented in DAVIS-F. For example, we find 1.3 points improvement in $\text{AJ}$ by using our strategy.

---

[Q1] Explanations for the inconsistent results with the baselines’ original reports

• Inconsistent TAPNet results in Table 1

This inconsistency is mainly caused by the difference in the image backbone provided by the official checkpoint we utilized to reproduce TAPNet and TAPIR; it provides an updated ResNet18 backbone, instead of TSM-ResNet18 used by TAPNet in the original paper. Otherwise, the tracking method is the same as TAPNet (i.e., cost volumes without PIPs iterations).

Nevertheless, we additionally provide the results based on the checkpoint with the original TSM-ResNet18 image backbone (which we find in the recent update of the official repository). When experimented with DAVIS-F, DAVIS-S, and Kinetics benchmarks, we find TrackIME keeps demonstrating significant gains, e.g., 32.8 $\to$ 47.0 AJs (14.2 points) in DAVIS-F. We will include these discussions and results in the updated manuscript.

\begin{array}{l c c c c c | c c c c c | c c c c c} & & & & \llap{\text{DAVIS}-\text{F}} & & & & & \llap{\text{DAVIS}-\text{S}} & & & & & \llap{\text{Kinetics}~~~~} \newline \text{Method} & J_1 & \text{AJ} & \delta^x_1 & \delta^x_\text{avg} & \text{OA} & J_1 & \text{AJ} & \delta^x_1 & \delta^x_\text{avg} & \text{OA} & J_1 & \text{AJ} & \delta^x_1 & \delta^x_\text{avg} & \text{OA} \newline \hline \text{TAPNet (TSM-ResNet)} & 5.8 & 32.8 & 11.1 & 48.4 & 77.6 & 6.7 & 38.4 & 12.6 & 53.4 & 81.4 & 8.1 & 38.3 & 14.4 & 52.5 & 79.3 \newline +~TrackIME & \mathbf{18.7} & \mathbf{47.0} & \mathbf{28.6} & \mathbf{60.6} & \mathbf{80.9} & \mathbf{21.5} & \mathbf{50.8} & \mathbf{32.4} & \mathbf{63.8} & \mathbf{81.4} & \mathbf{17.0} & \mathbf{44.5} & \mathbf{26.3} & \mathbf{57.9} & \mathbf{80.4} \newline \hline \end{array}

• The effect of library/hardware-dependent numerical characteristics

Since TrackIME is a plug-in to all baselines, we reproduced all results in our system for fair comparisons. This induces minor inconsistencies due to library/hardware-dependent numerical characteristics, even if the same model is used. For example, the improved performance of TAPIR (mentioned in L431) is caused by the different characteristics between JAX and PyTorch. We note that such inconsistencies are also observed in other references, e.g., $\delta^x_{\text{avg}}$ for PIPS2 on DAVIS-F: 69.1 (reported by CoTracker paper); 70.6 (reported by PIPS2 git repository); 69.4 (reported by our paper).

Dear reviewer jQxW,

Thank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the significance of the method, exhaustive experimentation, and clear presentation. We will address your concerns and questions in the response below.

---

[Q1] Commentary on why TrackIME may be performing better than SAM-PT on zero-shot video object segmentation

The key advantage of TrackIME is removing erroneous query points for segmentation caused by the tracking failure on intricate object parts, and enabling even finer query points for segmentation, e.g., demonstrating outstanding results in 1-pixel thresholds (e.g., Table 1), which is possible due to the pruning structure in our framework to maintain the high-frequency information. Although SAM-PT contributes to enhancing the segmentation from the point tracking task, their performance is still bounded by low-resolution tracking and suffers from tracking failures.

Suppose SAM could provide perfect segmentation for a given query, and the point tracking model could provide perfect tracking and occlusion predictions up to their resolutions. Nevertheless, the tracking resolution would impose an upper bound for the video object segmentation performance because the query points in the fine-grained object parts suffer from failure modes in lower resolutions.

We would like to attribute the better performance demonstrated by TrackIME to this point.

---

[Q2] Table 7: Are the resolutions displayed here (excluding TrackIME since there's no training involved) the input resolutions during training or during testing?

The resolutions for TAPIR models in Table 7 are the inference resolutions, where we utilized the official checkpoint trained in the 256 x 256 resolution. We will clarify this point in the updated manuscript.

---

[Q3] L274: "23.2" is supposed to be 28.2, right?

Yes. We will correct this typo in the updated manuscript.

---

[Q4] Editorial comments

Thank you for the constructive editorial comments. In the updated manuscript, we will provide hyperparameters more clearly (e.g., $r=0.10$ for the segmentation), revise the presentation to mention our ablation studies in the introduction of Section 4 and that our tracking backbone is TAPIR in Table 3, and revamp the drawings in Figures 2 and 3.