Response to Reviewer 6eUD

1. Lack of Comparisons with 2D Trackers (Track-On, CoTracker3, or BootsTAPNext) on Dynamic Replica

Our submission already includes quantitative comparisons with CoTracker3, including on the Dynamic Replica dataset, as in Table 2 of the paper. Per your request, we now include additional experiments with BootsTAPNext [a] on both Dynamic Replica and LSFOdyssey datasets as below. While BootsTapNext exhibits slightly better performance than CoTracker3 on certain metrics, our method still demonstrates superior results in both 2D and 3D metrics when ground truth depth is available.

Note: BootsTAPNext's performance on Dynamic Replica is inferior in the table. We consider this as normal, as mentioned in Section 4.3 of [a]: "We observe significant failure in long-term point tracking over the full length of videos greater than 150 frames.", while sequences in Dynamic Replica have 300 frames.

However, we wish to clarify that our goal is not 2D tracking. Accordingly, we focus evaluations on datasets that provide 3D point-track annotations, which allows us to compute 3D point tracking metrics. Note that [a] is public on arXiv only 1 month before the paper submission DDL. We will include more detailed comparison to BootsTAPNext [a] and Track-On in our final version.

[a] TAPNext: Tracking Any Point (TAP) as Next Token Prediction[J]. arXiv preprint arXiv:2504.05579, 2025.

2. Lower occlusion accuracy

We acknowledge this limitation, which stems from TAPIP3D’s 3D-centric design. The model operates entirely in XYZ coordinates using KNN-based attention, without explicit knowledge of pixel-space proximity or depth ordering. As a result, occlusion reasoning relies solely on neighbor features and their spatial offsets, making it more challenging, especially when depth reconstruction is unreliable.

This issue is particularly noticeable in the DriveTrack dataset, where occlusions between distant vehicles lead to significant depth estimation errors. In some cases, depth of remote objects becomes visually fused due to the low resolution of the underlying depth model, which results in misleading geometry and unreliable neighbor associations. Incorporating auxiliary 2D cues, such as depth-difference maps used in SpaTracker and DELTA, could help address this, which we leave for future work.

3. Clarification on Latency and Memory Use

We appreciate the reviewer’s request for a more detailed runtime and memory analysis. The reported 10 FPS refers to the tracking stage, assuming that depth and camera poses are available. It does not include the runtime of depth or pose estimation, as these are obtained from external sensors or off-the-shelf models (e.g., MegaSaM [9], CUT3R [10]) and thus can vary dramatically. Importantly, our model is not trained with any specific estimation method and can be flexibly combined with any future advancements in depth or pose estimation.

Below, we provide a breakdown of latency during one pass of forward tracking on a sequence from the Dex-YCB dataset on a single L40S GPU. The forward pass consumes a total peak memory usage of 3.6GB, while feature encoding uses up to 2.8GB.

We will include these in the revised manuscript.

4. On Question 2: Model Behavior and Fallback under Inaccurate Camera Extrinsics

Although TAPIP3D relies on estimated camera poses to transform points into a shared world frame, our training augmentations enable the model to generalize well under smooth camera motion. As a result, the model retains a degree of robustness even when the poses are inaccurate. As shown in Tables 1 and 2 of the paper, TAPIP3D-world and TAPIP3D-camera share the same model but use different pose inputs. TAPIP3D-camera effectively serves as a lower bound on the robustness of our approach, achieving competitive results even when camera extrinsics are highly unreliable, in which case the user can simply specify identity matrices as extrinsics.

5. Dependence on CoTracker Initialization

We incorporated CoTracker3 initialization early in the project and observed that it accelerates convergence; however, it should have minimal impact on the final performance after full training. We will include an ablation study in the revised paper to substantiate this observation.

6. Ablation on Depth Estimator Choice

We would like to clarify that our model is not specifically trained for any particular geometry estimation method. For evaluation, we selected MegaSAM due to its superior geometric accuracy and robust camera pose estimation.

Here, we additionally include results using UniDepth [a], a weaker depth estimation model with lower geometric and pose accuracy than MegaSAM. While the overall performance drops, our model still produces reasonable results.

[a] UniDepth: Universal Monocular Metric Depth Estimation. CVPR, 2024.

Thank you for the thoughtful follow-up. We address your remaining concerns as below:

1. Sensitivity to Camera Pose Estimation

We acknowledge that demonstrating robustness across different depth estimators alone could be insufficient. We provide two complementary experiments to assess pose estimation sensitivity, where our TAPIP3D is robust to varing pose estimation methods and independent angular camera noise:

Experiment 1: Alternative Pose Estimation Method
To provide a broader comparison beyond the MegaSAM + MoGe pipeline, we evaluate our method on LSFOdyssey using ground truth depth combined with camera poses estimated by various methods.

Note that TAPIP3D-camera represents our method's performance when pose information is completely unavailable, where we simply pass identity matrices as extrinsics.

[a] Continuous 3D Perception Model with Persistent State. CVPR, 2025.

Experiment 2: Injecting Angular Noise to Camera Poses We inject angular noise into GT camera poses on LSFOdyssey. Each frame's camera orientation is independently perturbed by rotating around a random axis, creating significant geometric inconsistencies across frames. Even with large angular noise like 2.0°, our method still significantly outperforms the baseline method DELTA.

*σ represents the standard deviation of the angular noise

2. TAP-Vid Evaluation

Following your suggestion, we conducted an evaluation on the TAP-Vid-DAVIS benchmark on the 2D tracking metrics (APD-2D), where CoTracker3 achieves 77.0, while our TAPIP3D-world reaches 71.2. The performance gap is primarily due to MegaSAM's complete failure in partial video scenarios (4/30), resulting in severely noisy depth estimates and unstable point clouds. If we removed this 4 MeGa-SaM failing DAVIS video, the 2D point tracking of CoTracker3 is 77.1 while ours is 73.4. We discussed these in the limitations section and the supplemental video, and we will include more detailed per-video experimental analysis in the final version of the paper.

We would also like to clarify that TAPIP3D is designed for two primary application scenarios and 3D point tracking:

1. RGB-D settings, using sensor depth, such as in robotics or autonomous driving (Table 3 of the paper, TAPIP3D achived significant boost);
2. Monocular videos, where the accuracy and robustness of MegaSAM-like depth and pose estimators are rapidly improving.

As noted by Reviewer Sbrq:

“I believe the field of camera pose and depth estimation is growing surprisingly fast, supported by recent works such as VGGT [1], CUT3R [2], and pi-3 [3]. I'm confident that these concerns will diminish in the near future, making this work even more important as TAPIP-3D stands to benefit greatly from those advancements.”

We would like to emphasize that this comparison is meant to highlight trade-offs and limitations, rather than to claim TAPIP3D's superiority over established 2D tracking methods.

3. Additional Runtime Analysis

We provide latency breakdown for two geometry estimation pipelines on a sequence from Dex-YCB using a single L40S GPU:

Response to Reviewer pGLy

1. On the Contribution and Necessity of TAPIP3D’s Architecture

We agree with the reviewer that the performance improvements of our camera space approach using estimated depth may appear modest in Table 1. However, we respectfully disagree with the suggestion that operating in a stabilized world frame is a straightforward extension of existing baselines or that our architectural contributions are incremental.

Most prior methods, including DELTA and CoTracker3, are fundamentally designed around 2D-centric architectures. They operate in UVD space, which is 2D coordinates with depth appended, and rely on grid-based correlation volumes defined in pixel space. These components are not naturally extendable to the unordered and irregular structure of world-frame 3D data, where no global 2D grid exists.

In TAPIP3D, we address this challenge with our 3D Neighborhood-to-Neighborhood (N2N) attention, which enables local feature matching and aggregation directly in a shared world coordinate space. To the best of our knowledge, there is no simple way to adapt 2D-based models like DELTA or CoTracker3 to this setting without significant architectural changes.

We also note that the performance gap becomes substantially larger when higher-quality geometric input is available. In Table 2 (GT depth), TAPIP3D outperforms prior methods by over 30% on the 3D APD metric. Similarly, Table 3 (DexYCB) shows that our architecture achieves significantly better performance than prior baselines when using sensor depth, which is often available in robotic scenario. This demonstrates that TAPIP3D is well-suited to leverage accurate geometric inputs, which is an increasingly important advantage as 3D foundation models for depth and pose estimation continue to improve.

2. On the Sensitivity of the method to Depth Input

While the reviewer notes that our 2D tracking performance using MegaSaM depth appears weaker, our primary contribution lies in robust 3D tracking. As shown in Table 2, TAPIP3D consistently outperforms all baselines in 3D metrics (AJ_3D and APD_3D) under both MegaSaM and GT depth, demonstrating the method’s robustness to depth quality. Moreover, the 2D performance of TAPIP3D improves significantly when higher-quality depth is available (e.g., GT depth), indicating that the 2D results are naturally coupled with depth accuracy rather than a limitation of the model itself.

In many real-world scenarios such as robotics or sensor-based capture, accurate depth maps are readily available from RGB-D cameras. As evidenced by the sensor depth results in Table 3, TAPIP3D achieves state-of-the-art 3D tracking in such settings, which highlights a key advantage of our approach. To further validate this, we provide evaluation results on a subset of DriveTrack [1] using sparse LiDAR depth, densified via a simple nearest neighbor approach. Despite the presence of noise and significant discontinuities in the unprojected geometry, our model achieves robust 2D and 3D tracking:

We additionally include results using UniDepth [a], a weaker depth estimation model with lower geometric and pose accuracy than MegaSAM. While the overall performance drops, our model still produces reasonable results.

[a] UniDepth: Universal Monocular Metric Depth Estimation. CVPR, 2024.

3. On the Sensitivity of the method to Camera Pose Estimation

Although TAPIP3D relies on estimated camera poses to transform points into a shared world frame, our training augmentations enable the model to generalize well under smooth camera motion. As a result, the model retains a degree of robustness even when the poses are inaccurate. As shown in Tables 1 and 2 of the paper, TAPIP3D-world and TAPIP3D-camera share the same model but use different pose inputs. TAPIP3D-camera effectively serves as a lower bound on the robustness of our approach, achieving competitive results even when camera extrinsics are highly unreliable, in which case the user can simply specify identity matrices as extrinsics.

4. Evaluation Clarifications

3.1 Discrepancy on the reported Average Performance of DELTA in Table 1

This is a typo, thank you for pointing this out! We incorrectly computed the DELTA average in Table 1. The per-dataset values are correct, and we have updated this average to its correct value from 22.3 to 26.3 in the manuscript. Our TAPIP3D-world still outperforms DELTA for 1.1 APD$_{3D}$.

3.2 Why the proposed method has a much better 3D tracking performance that Cotracker3 despite worse 2D tracking performance, since both methods use GT depth?

Thank you for the question. We would like to clarify a possible misunderstanding. In Table 2 (GT depth version), TAPIP3D-camera actually outperforms CoTracker3 on both 2D and 3D tracking metrics, not just in 3D.

• LSFOdyssey: APD-3D: 83.2 vs. 35.0; APD-2D: 91.2 vs. 88.0
• Dynamic Replica: APD-3D: 70.8 vs. 38.0; APD-2D: 84.7 vs. 80.0

The improvement appears numerically larger for 3D metrics because when using the same depth map, even small 2D pixel offsets—particularly for points near object boundaries—can lead to significant depth variation (e.g., a point jumping from foreground to background). This amplifies the effect in 3D space, making gains in 2D accuracy translate into larger improvements in 3D error.

[1] DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos. CVPR, 2024.

Response to Reviewer Sbrq

Thank you for your encouraging feedback and for recognizing both the conceptual ideas and technical contributions of our work.

1. Performance Comparison on TAPVid-3D with Ground Truth Depth

We appreciate the reviewer’s interest in whether TAPIP-3D’s superior 2D performance when using groundtruth depth generalizes to the TAPVid-3D dataset. However, TAPVid-3D does not provide ground truth depth annotations.

To partially address this question, we investigated the Aria Digital Twin (ADT) sub-dataset of TAPVid3D benchmark, where all objects in the 3D scenes are modeled with provided rendered depth. As in the below table, our TAPIP3D still outperforms CoTracker3 in 2D tracking on ADT dataset with GT depth.

2. Evaluation with Sparse Depth on the Waymo Dataset

We created a benchmark consisting of 50 sequences from the Waymo Open Dataset [3] (a subset of TAPVid-3D), paired with depth map annotations obtained by densifying sparse LiDAR depth using a simple nearest-neighbor approach, as proposed in the DriveTrack [4] paper. We report the performance of TAPIP3D-camera, DELTA, and CoTracker3 on this dataset.

3. Explanation for Results on LSFOdyssey

(1) Substantial performance gap between TAPIP-3D and its competitors in LSFOdyssey The LSFOdyssey dataset features challenging scenarios with many objects rotating and occluding each other, where knowing the accurate local 3d geometry help substantially.

(2) Why does SpatialTracker's performance in LSFOdyssey drop when using ground truth depth compared to MegaSAM predictions? In the provided GT depth, the background has large depth values, and SpatialTracker normalizes all valid depths to the range [0,1], which results in compressing the scene to a small region in SpatialTracker’s tri-planes. Our model is more robust to large depth ranges because it does not require depth values to fall within a fixed interval.

We further fixed the depth value issues of the background regions in SpatialTracker's code by setting it to zero (previously is a large value) to mark it as an invalid region and re-evaluate SpatialTrack using GT depth. The result on the LSFOdyssey benchmark is below. Although the AJ-3D performance of SpaTracker got improved, it still has a significant gap to our method.

4. Clarification on Evaluation Metrics (definition of APD)

Use depth-adaptive APD as a tracking metric, as introduced in TAP-Vid-3D

This is a misunderstanding. APD_3D stands for a depth-adaptive variant that scales the threshold based on object distance, precisely as introduced in TAPVid-3D [2]. So, we do use the metric you are suggesting. APD_2D is a fixed-threshold metric computed in pixel space, following [1].

We apologize for the confusion regarding APD metrics. We will include the formal definitions in the text to make this distinction clear.

The notation 'T' is used to denote the number of frames in a video and to represent the size of a local window.

We agree the notation could be clearer. In the final version, we will modify $T$ (temporal window length) to $W$ to avoid confusion with the total number of video frames.

5. DELTA Failure Cases (Figure 4)

DELTA is trained with depth augmentation. As a result, it sometimes “corrects” geometry even when it conflicts slightly with the input depth. By contrast, our model always respects the given geometry.

[1] TAP-Vid: A Benchmark for Tracking Any Point in a Video. NeurIPS, 2022.

[2] TAPVid-3D: A Benchmark for Tracking Any Point in 3D. NeurIPS, 2024.

[3] Scalability in Perception for Autonomous Driving: Waymo Open Dataset. CVPR, 2020.

[4] DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos. CVPR, 2024.

Response to Reviewer 4cTm

We thank the reviewer for the thoughtful and detailed feedback, and for highlighting the strengths of our decoupling strategy and 3D N2N attention mechanism.

1. Efficiency Comparison with Optimization-Based Methods

We acknowledge the concern regarding efficiency comparisons with optimization-based scene representation methods such as 4DGS [1] and appreciate the opportunity to clarify.

While 4DGS reports high rendering frame rates of 82 FPS, this only reflects only its rendering speed after training. The method requires time-consuming per-sequence optimization, with training times reaching up to 30 minutes per sequence on the HyperNeRF dataset, as reported in Table 2 of [1]. Similar limitations apply to other optimization-based methods such as NeRF.

In contrast, TAPIP-3D performs long-term point tracking in a fully feedforward manner, running at approximately 10 FPS without any per-sequence training (fitting).

Given the significant differences in computational cost and target applications, direct efficiency comparisons between these methods are not informative in practice. We will clarify this important distinction in the next version of the paper.

2. Robustness to Blur

2.1 Impact of Blur at Varying Degrees

Our method augments the training data by introducing motion blur and randomly applying Gaussian blur to both images and depth maps to improve robustness against blurry inputs. We also adopt all the 2D data augmentations from CoTracker3 as mentioned in L10 of the Supp. file, which includes gaussian blur to images. We will detail it in the paper. As a result, our model can tolerate some degree of blur. To evaluate the model’s robustness more concretely, we present a table below showing quantitative results of TAPIP3D-world + M-SaM on the LSFOdyssey dataset with varying levels of gaussian blur applied:

Note that in the table above, our model's 3D performance surpasses the baseline, DELTA, at every blur level, demonstrating competitive robustness. From the visualizations, we observe that with blurry inputs, the points align reasonably well with objets, while depth estimates of the same object become less consistent across frames, possibly due to blurry inputs being out-of-distribution for MegaSAM.

2.2 Tracking Blurry and Distant Objects

We acknowledge that low-resolution and distant objects may lead to degraded geometry, particularly around edges. To mitigate artifacts from edge interpolation caused by depth estimation, we apply a simple edge-removal preprocessing step before feeding the depth maps into our pipeline. This approach is commonly adopted in 3D visualization for depth models.

With this preprocessing in place, we observe that on DriveTrack (a subset of TAPVid-3D), where vehicles often appear small, blurry, and distant, our model is often able to track points on these objects despite the sparse pixel density after unprojection. However, we acknowledge some failure cases, particularly when two distant vehicles occlude each other. In such scenarios, the estimated depth geometry may become ambiguous, with disconnected objects appearing fused, making it hard for our model to distinguish between them. We believe this issue is related to the low resolution of the depth model (MegaSAM operates at 192×256 and is upscaled prior to use in our model). We expect that future improvements in foundational geometry (depth and camera pose) estimation models will help mitigate these challenges. We will include a discussion of these limitations in the final version of the paper.

---

References:

[1] Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4D Gaussian splatting for real-time dynamic scene rendering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024, pp. 20310–20320.

Thank you for the follow-up. We are very pleased to hear that you are considering raising your score.

1. Since we give all methods the same pose/depth estimators, this cannot be the source of relative improvement.
2. We are not sure what you mean by the detail about “augmented data”. The baselines (DELTA, SpatialTracker) use largely similar training time augmentations as TAPIP3D.
3. We can also add an experiment with multi-frame averaging, for a more accurate assessment of the robustness to motion blur. We agree this would be more accurate than Gaussian blur.
4. Potential refinements to MegaSAM: We agree! This would be exciting to explore in future work.