TAPTRv2: Attention-based Position Update Improves Tracking Any Point

Summary of review ratings.

Rebuttal - zD8r

We thank the reviewer for the recognition of the novelty, effectiveness, extensive experiments, and the potential of TAPTRv2 in scaling up the training process and accelerating the development of algorithms.

Q1. Main concerns-1: Speed comparison.

A1 We thank the reviewer for this constructive suggestion. We apologize for overlooking such an important point. Indeed, the comparison should be an important part of our work, and we will add this comparison to our camera-ready version. We follow the main-stream works in the object detection field to utilize the FPS, GFLOPS, and the number of parameters to show the comparison of speed, computational efficiency, and resource requirements. As shown in the following table, TAPTRv2 exhibits a faster speed and lower resource requirements compared to TAPTR. More importantly, it's a common case that in the downstream tasks, we need to track all pixels in a region rather than just a few scattered points. In this case, the number of points to be tracked will reach to tens of thousands. However, since the computation of the cost-volume and also the cost-volume aggregation operation in TAPTR increases sharply with the number of tracking points, with the number of tracking points increased, the advantage of TAPTRv2 will become more and more pronounced. As shown in the following second table, when the number of tracking points reaches 5000 (which is only 1.9% of the pixels in a 512x512 image), the advantage of TAPTRv2 in speed and resource consumption becomes much more significant (about 24% faster and 20% fewer computational resource requirements).

Q2. Main concerns-2: The efficiency of APU operation.

A2.1 In fact, in our implementation of the APU, it is only an additional weighted summation operation (as highlighted by the red flows in Fig. 3). Therefore, its impact on efficiency is neglectable.

A2.2 By contrast, the computation of the cost volume and its aggregation in decoder layers have a significant impact on efficiency, especially when the number of tracking points increases, as we have discussed in A1.

Q3. Assess the generalization of the improvements.

A3.1 We thank the reviewer for the constructive suggestion. Indeed, including more datasets will help us assess the generalization ability of TAPTRv2 over TAPTR. Here we further include another challenging dataset with long videos (up to 1000+ frames, about 1 minute long) developed by Deepmind, called RoboTAP, into consideration. The comparisons between TAPTR and TAPTRv2 on the RoboTAP dataset are shown below. The performance advantage of TAPTRv2 reflects its general improvement. We will further include more datasets in our camera-ready version.

A3.2 As suggested by reviewer-wowB, we conduct a statistical significance analysis to verify whether the improvements of TAPTRv2 over TAPTR are significant. We utilize the Friedman test, and the resulting p-value is 0.042. If we take the results in RoboTAP into consideration, the resulting p-value further decreases to 0.024. These statistical results indicate that the improvement of TAPTRv2 over TAPTR is statistically significant, showing its generalization capability.

Q4. Inconsistent names in Fig. 4.

A4 We thank the reviewer's kind reminder. The inconsistency is actually our typo, we will fix this typo in our camera-ready version.

Dear Reviewer wowB, The anonymized video link for the offsets’ visualization has been permitted by the Area Chair. As required by NeurIPS 2024, the link is provided in a separate comment block titled “Anonymized Video Link.” We apologize for the inconvenience.

Limited by the space of the above rebuttal block, we put some other information here. We apologize for the inconvenience.

Summary of review ratings.

Some statements that need the reviewer's further clarity.

Some of the reviewer’s statements are not very clear, so we have not responded to them currently. If the reviewer wants to discuss these points in the discussion phase, we'd appreciate if the review could further clarify the following questions.

1. What does the 'different alternatives to the attention blocks' mean in the third-to-last paragraph?
2. What does the 'gain' mean in the third-to-last paragraph?

We thank the reviewer's kind suggestions and questions. We also thank the reviewer for recognizing our writing, motivation, deep analysis, and appreciation of our design in APU block.

Q1. The initialization of point queries.

A1 We appreciate the reviewer’s thorough review and pointing out the unclear parts. We will provide a more detailed description of the point-query initialization process from a clearer aspect here. If the reviewer still has questions, we can further discuss this in the discussion stage.

For simplicity, without losing generality, we assume that all points are tracked starting from the first frame. Thus, the positions of these points in the first frame are given; otherwise, we wouldn’t know which targets to track. For the i-th point, to obtain a feature that describes it, we perform bilinear sampling on the feature map of the first frame at the location $l_e^i$ of that point, resulting in $f_e^i$. Subsequently, if we want to track this point over the next N-1 frames, $f_e^i$ and $l_e^i$ will be copied N-1 times and distributed to the following N-1 frames to initialize the content and positional part of the point-queries for detecting this point in the following N-1 frames respectively (as we no longer need to track the target point in the first frame since it is already known).

Meanwhile, let’s continue with the i-th point as an example. The initial content and positional parts of the point-queries that are responsible for detecting the i-th point are not static and will be updated. For instance, once we have completed the detection of the i-th point in the first window (frames 0-7), the content and positional part of the point-query that is responsible for detecting the i-th point in the 7-th frame $f_7^i$ and $l_7^i$ will be used to update the initial content and positional part of the point-queries that are responsible for detecting the i-th point in subsequent frames $[(f_8^i, l_8^i), (f_9^i, l_9^i), \cdots, (f_{N-1}^i, l_{N-1}^i)]$.

Since the tracking of each point is relatively independent, the points interact only within the self-attention block in decoder. Therefore, the number of points does not affect the initialization process.

Q2. Analysis of the improvement.

A2.1 At first, we have to thank the reviewer for this constructive suggestion. We conducte a statistical significance analysis to verify whether our improvements are statistically significant. We applied the Friedman test to the results in Table. 2, and the resulting p-value is 0.042. Since the p-value is less than 0.05, our experimental results are statistically significant. At the same time, if we take RoboTAP dataset, which is proposed by DeepMind and commonly used in many TAP methods, into consideration as suggested by reviewer-zD8r, the p-value decreases to 0.024, further verifying the significance. We put the comparison of the main metric (AJ) between TAPTRv2 and TAPTR on the four datasets in the table below for your convenience in reviewing.

A2.2 As for the marginal improvement on DAVIS-Strided, we believe it may be due to the shorter video lengths compared to other datasets as shown above (for more detail, please refer to Sec.4.1). This makes it relatively simpler than other datasets, so the issue of point-query content contamination in TAPTR is less pronounced, leading to the marginal improvement of TAPTRv2 over TAPTR. However, as shown in the above table, with the length of video increased, the improvement becomes more and more significant, verifying TAPTRv2's superiority.

Q3. Computational complexity.

A3 We thank the reviewer for this constructive suggestion. Since this is a common issue raised by all the reviewers, due to space limitations here, please refer to our responses in zGVx-A3, yaYP-A1, and zD8r-A1 for more detail. We apologize for the inconvenience.

Q4. Visualization of the offsets.

A4 We thank the reviewer for this constructive suggestion, which helps us have a more intuitive understanding of the of TAPTRv2's function. The corresponding anonymized video link is provided in a separate comment for AC as required by NeurIPS'24, and the illustration of the video is provided in Fig. 1 of the attached pdf in our global rebuttal.

Q5. Choice of sampling points.

A5 We conduct a group of comparative experiments to show the effect of different kinds of sampling points in APU shown below, where 'Normal' indicates the choice of sampling points in our main paper, which contains 4 neighboring points in each scale of the feature maps. The 'local grid' yields the worst results, which is expected, because the fixed sampling grid prevents the model from flexibly adjusting its receptive field to accommodate varying target motion amplitudes. After that, it can be observed that increasing the number of sampling points ('Broader Set') also does not lead to an improvement. We suspect this might be due to our model’s capacity being insufficient to handle such a large amount of information.

Q6. Proof-reading.

We thank the reviewer for the thorough review, we will correct these typos in our camera-ready version.

Discussion

Due to space limitation, our description of decoder in the main paper is relatively abbreviated. We apologize for this and will provide a more detailed description in the camera-ready version. As shown in Fig. 3, the decoder incorporates both 'self-attention' as well as 'temporal-attention' mechanisms. The 'self-attention' considers multiple points that belong to the same timestamp, while the 'temporal-attention' considers the N frames of the same point. This decomposition of 'temporal' and 'multiple-point' dimensions not only clarifies feature processing but also helps reduce computational cost and memory usage.

We have to thank the reviewer's thorough review and valuable suggestions again. At the same time, we would like to express our gratitude for the reviewer’s recognition of our efforts.

Summary of review ratings.

Rebuttal - yaYP

We thank the reviewer for recognizing the simplicity and effectiveness of TAPTRv2 and its contribution to broadening the point tracking field's horizon for advancements. We would also like to thank the reviewer for recognizing the rationalization of our research methodology.

Q1. Elucidate the tangible benefits of the elimination of cost-volume.

We thank the reviewer for this constructive suggestion. We follow the main-stream works in the object detection field to utilize the FPS, GFLOPS, and the number of parameters to show the comparison of speed, computational efficiency, and resource requirements between TAPTR and TAPTRv2 to verify the benefits of removing the cost-volume. As shown in the following table, without the additional overhead of computing the cost volume and aggregating the cost volume in the decoder layers, TAPTRv2 exhibits a faster speed and lower resource requirements compared to TAPTR. More importantly, it's a common case that in the downstream tasks, we need to track all pixels in a region rather than just a few scattered points. In this case, the number of points to be tracked will reach to tens of thousands. However, since the computation of the cost-volume and also the cost-volume aggregation operation in TAPTR increases sharply with the number of tracking points, with the number of tracking points increased, the advantage of TAPTRv2 will become more and more pronounced. As shown in the following second table, when the number of tracking points reaches 5000 (which is only 1.9% of the pixels in a 512x512 image), the advantage of TAPTRv2 in speed and resource consumption becomes much more significant (about 24% faster and 20% fewer computational resource requirements).

Q2. Analysis of the extra MLP's effect on inference speed.

A2 We thank the reviewer for raising this good question. In fact, the extra MLP is small, so its effect on inference is almost negligible theoretically. We conduct comparative experiments to further verify this, the results are shown in the following table.

Q3. Disentangler operation may disrupt the inherent normalization of the attention mechanism.

A3 We appreciate the reviewer’s thorough review. Yes, indeed, in our implementation we divide the attention weights by the square root of the dimensionality before sending them to the $`Disentangler`$ to keep the inherent normalization. The Eq. 5 should be:

$\Delta l_t^i = `SoftMax`\left(`Disentangler`\left(A_t^i / \sqrt{d}\right)\right) \cdot S_t^i$.

We will correct this typo in our camera-ready version.

Dear reviewer-yaYP,

Thanks for your recognition and constructive suggestions again. Considering that the deadline is approaching, we would like to kindly remind you to check out our responses and raise the questions that you may still have. This will help us to provide more detailed explanations.

Thanks for your time and effort.

Summary of review ratings.

Rebuttal - zGVx

We thank the reviewer for recognizing the clear description of the issue to solve, the strong validation of our claim, and the novelty and effectiveness of our proposed APU. We will respond to your questions in the following, hoping to address your concerns. If our response still does not address your concern, please bring it up in the discussion section, and we will reply as soon as possible.

Q1.Theoretical justification to support the detrimental effect brought by different distribution.

We thank the reviewer for this constructive suggestion, we will add the following analysis to our camera-ready version.

A1.1 We measured the distribution of the attention weights for content and position update, as visualized in Fig. 2 of the attached pdf file in our global rebuttal, the distributions of these two groups of attention weights show a significant difference, indicating that the attention weights required by content and position update are different. We will add this visualization to our supplementary material and refer to this analysis in the main paper in our camera-ready version.

A1.2 At the same time, as shown in our ablation studies in Table 3, if we do not disentangle the weights, the performance will suffer a significant drop, indicating the detrimental effect of not disentabling the weights.

Q2. More comparative experiments with alternative methods for weight separation.

A2.1 Inspired by our analysis in Q1, instead of using an MLP to obtain the separated attention weights, by reducing the "temperature" in the softmax calculation, we obtain a group of attention weights with smoother distribution. We conduct experiments with temperature = 0.2 / 0.5. The results are shown in the following table. The results show that disentangling attention weights using different temperatures indeed helps improve the performance. At the same time, consistent with the conclusion drawn in Q1, a smoother distribution (temperature=0.2) leads to better results. However, it still lags significantly behind disentangling the weights through an MLP.

$\Delta l_t^i = `SoftMax`\left(A_t^i / \sqrt{d} * temperature\right) \cdot S_t^i$

A2.2 As suggested by the reviewer, we further disentangle attention weight through an attention mechanism. The results are better than the last method, but still poorer than the MLP one proposed in our paper.

Q3. Computational efficiency and resource requirements.

A3 We thank the reviewer for this constructive suggestion. Indeed, this comparison should be a necessary part of our work, and we will add this comparison to our camera-ready version. We follow the main-stream works in the object detection field to utilize the FPS, GFLOPS, and the number of parameters to show the comparison of speed, computational efficiency, and resource requirements. As shown in the following table, TAPTRv2 exhibits a faster speed and lower resource requirements compared to TAPTR. More importantly, it's a common case that in the downstream tasks, we need to track all pixels in a region (e.g. tracking a text written on the back of a horse) rather than just a few scattered points. In this case, the number of points to be tracked will reach to tens of thousands. However, since the computation of the cost-volume and also the cost-volume aggregation operation in TAPTR increases sharply with the number of tracking points, with the number of tracking points increased, the advantage of TAPTRv2 will become more and more pronounced. As shown in the following second table, when the number of tracking points reaches 5000 (which is only 1.9% of the pixels in a 512x512 image), the advantage of TAPTRv2 in speed and resource consumption becomes much more significant (about 24% faster and 20% fewer computational resource requirements).

Discussion

Q4. Whether the performance advantage is achieved at the expense of increased computational cost.

We apologize for misunderstanding your question (Q3) and conflating it with other reviewers’ questions.

A4.1 Firstly, we have to thank the reviewer for the constructive suggestion. Indeed, comparing with more methods from the aspects of efficiency and speed can better reflect the advantage of TAPTRv2. Limited by the deadline of the discussion stage, as shown in the below tables, we compare the performance, computational cost, speed, and the number of parameters with two most widely recognized works in terms of performance and speed: CoTracker (from Meta, using its open-sourced implementation with about 2.2K stars) and PIPs (from CMU, using its open-sourced implementation with about 600 stars). The results show that, although these two methods have fewer parameters than ours, their computational cost is about three times larger than ours, resulting in much slower speeds. These substantial computational costs are due to their redundant designs in their multi-layer refinement process as we have discussed in Sec. 1. For example, they need to recalculate the correlation map between each tracking point and every image feature at the beginning of each layer in multi-layer refinement (6 layers in total).

A4.2 Affected by such redundant designs, their computational costs increase more rapidly with the number of points to be tracked. As shown in the second table, when the number of tracking points reaches 5000, CoTracker encounters an Out-Of-Memory error (OOM). Although PIPs does not experience the OOM error, its computational cost is about five times larger.

A4.3 Although these methods requires much more computational cost than TAPTRv2, TAPTRv2 still obtains the best performance. Especially, even when CoTracker is tested in its 'Single' mode, which deliberately tracks each single point at a time and will brings much more computational cost, CoTracker's performance is still inferior to ours. We believe these results can prove that the performance advantage is not achieved at the expense of increased computational cost.

(Since other methods are implemented with JAX and have not open-sourced their evaluation code (TAP-Net and TAPIR) or implemented with the on-the-shelf RAFT with numerous numpy computations on CPU (MFT), we need to spend more time measuring their computational cost. However, we believe that the comparison with CoTracker and PIPs is a solid support to our conclusion.)

4.4 At the same time, we hope that our insight in the analysis of cost-volume, the novel APU derived from the analysis, and the computational efficiency and performance superiority of TAPTRv2 over TAPTR can be considered as nontrivial contributions.