Fast Encoder-Based 3D from Casual Videos via Point Track Processing

We appreciate the reviewer's positive assessment of our method, particularly the recognition of our consideration of symmetry. We're pleased that the reviewer finds our approach solid and our evaluation results comprehensive. We'll now address the specific points and questions raised in the review.

Q: Limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes

A: Our model shows good adaptability to various scenarios. For video length, we trained on 20-50 frames and successfully tested on both 24 frames (Nvidia Dynamic Scenes Dataset) and 50 frames (Pet test set), with good performance in both cases. The encoder might be able to handle longer sequences, though we haven't tested this. Regarding scene complexity and object count, the Nvidia Dynamic Scenes Dataset includes a range of scenes, from single-object scenes like "Skating" to multi-object scenes like "Jumping," and various motion types such as "Truck" and "Umbrella". Tables 7 and 8 in the Appendix provide per-scene errors, demonstrating our method's robustness across these diverse scenarios, with Figure 4 offering visual examples. We'll include a more detailed discussion of these aspects in the revised paper.

Q: Changing the number of basis elements

A: We explored the impact of varying numbers of basis elements in our ablation study (Table 3, main paper), training models from scratch with different K values. Large K (30) led to increased depth error, indicating insufficient regularization, while small K (2) resulted in high pixel reprojection error, suggesting over-regularization and limited 3D representation of 2D motion. Our chosen K (12) balances depth regularization and accurate pixel representation. We found no significant differences with nearby K values (e.g., 11).

Q: Low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes.

A: We found K=12 basis elements effective for our evaluation set, balancing complexity reduction and motion representation. However, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. We'll add this point to our limitations and future work section.

Q: Robustness to noise or handling of occlusions. I'm curious how robust the method is against failing 2D point tracks.

A: We first note that the input point tracks extracted by CoTracker [18] are far from perfect, containing noise, outliers, and occlusions. To further evaluate our robustness to tracking errors, we performed three experiments by modifying CoTracker point tracks. First, to measure noise robustness, we added Gaussian noise to the tracks with different noise levels and measured overall accuracy. Second, to assess outlier robustness, we replaced a fraction of point tracks with uniformly sampled x,y points. Lastly, we repeated the second experiment but marked outlier points as occluded by setting o=0 (defined in L97 in the main paper). The results, presented in the attached PDF (Table 1), demonstrate that our method tolerates significant tracking errors and occlusions.

Q: Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?

A: Modeling dense point clouds would be an interesting direction for future work. Currently, our network can handle up to about 1000 point tracks in 50 frames in one inference step when running on an NVIDIA RTX A6000 GPU with 48GB of memory. A possible extension to handle denser point clouds could involve querying point tracks iteratively while maintaining a shared state, but this approach remains to be explored. We will include this discussion in the revised version of our paper.

Thank the authors for providing more insights into their work. I find this paper solid enough to lay a foundation for future long-sequence and dense point cloud tracking system. I would like to keep my original recommendation.

We appreciate the reviewer's recognition of the efficiency and the good performance of our approach.

Q: The permutation invariance of point sets has been discussed in early literature… and it might be better to connect…

A: We appreciate the reviewer's suggestion. While our architecture is based on a generalization of PointNet to sets of symmetric elements [Maron et al. 2020, ICML], which we reference multiple times in the paper, we acknowledge that PointNet [Qi et al. 2017] and DeepSets [Zaheer et al. 2017] are indeed seminal works in this area. We agree that explicitly discussing these connections would improve the clarity of our presentation. In the revised version, we will add references to PointNet and DeepSets, and briefly discuss how our approach builds upon and extends these foundational ideas in the context of our specific problem of processing point tracks from videos.

Q: I also don't really understand what is the relation of "linear equivariance" and cyclic group?

A: In geometric deep learning, it's customary to first analyze the symmetries of the input data and then derive suitable equivariant layers. Linear equivariant layers, such as convolutions and DeepSets [Zaheer et al., 2017], have proven effective and can be derived relatively easily. This approach is exemplified by convolutional layers for images, which are equivariant to translations, modeled as a product of cyclic groups (which formally correspond to an image on a torus).

In our case, we have a product of two group actions on the input point tracks:

1. Set symmetries (like in PointNet and DeepSets)
2. Time-shift symmetry (represented by a cyclic group)

Modeling temporal signals using cyclic symmetries is standard in the field, analogous to the image case, and forms the algebraic basis for transforms like the Discrete Fourier Transform (DFT).

We derive the linear equivariant layer structure using the DSS framework [Maron et al. ICML 2020]. However, we use this only as inspiration and replace convolutions with attention mechanisms and positional encoding in our final non-linear transformer-based architecture. This approach aligns with common practices in geometric deep learning [e.g., Bevilacqua et al. ICLR 2022]. We'll clarify this in the revised version.

Q: It seemed to me the final architecture is a standard transformer…

A: While our method incorporates transformer elements, it differs significantly from a standard transformer in its structure and processing: Our architecture operates on a 3D tensor of shape N × P × D (time steps × points × features) and processes it using two alternating operations:

1. Temporal Attention: A transformer with temporal positional encoding operates on each point track (N × D slices).

2. Set Attention: A transformer operates across all points at each time step (P × D slices).

This alternating row-column processing is inspired by the DSS linear architecture [Maron et al. 2020] but as mentioned above, replaces linear operations with more expressive transformer layers.

Regarding translation equivariance, we acknowledge that our final design is not formally equivariant to cyclic translations. Instead, it provides a strong inductive bias for temporal data through the use of temporal positional encoding. Our ablation studies (Table 3, original paper) demonstrate that this design significantly outperforms the strictly linear equivariant layers.

Q: 3D errors vs depth errors for evaluation.

A: Our evaluation focuses on depth metrics for lifted point tracks, as current state-of-the-art baselines addressing our case primarily predict dynamic depth maps. For our model, we also assess pixel reprojection error, which evaluates tracking accuracy in the other two dimensions (see Table 3, main paper).

Q: Is there any mechanism to handle the errors in co-tracker's 2d trajectory?

A: We note that the input point tracks extracted by CoTracker [18] are not perfect and contain noise and outliers. Nevertheless, as demonstrated in our paper, our model shows significant robustness to these errors. Our model handles imperfect input through two features:

• Static scene modeling: Ensures that only 2D motion that can be represented by a camera motion and truly static points is modeled statically. This makes our camera estimation robust to errors while pushing the non-modelled errors to the dynamic part.

• Dynamic scene modeling: Using limited basis elements for dynamic parts inherently resists outliers and extreme anomalies.

To quantify this robustness, we conducted tests by adding Gaussian and uniform noise to CoTracker points. The results (see Table 1 in the attached PDF) confirm that our method tolerates significant noise levels, further validating its effectiveness in handling imperfect input data. We will include this discussion in the revised version of our paper.

Q: Spatialtracker [B] is another relevant work.

A: SpatialTracker [53] is a concurrent work discussed in our related work section. Our approaches differ significantly: unlike SpatialTracker, which relies on depth estimation inputs or 3D ground-truth supervision, our fully self-supervised method requires only 2D point tracks as input, allowing us to train on a wider range of video data and fine-tune on test cases for improved accuracy (see Tables 1,2, Ours+FT in main paper). We have experimented with MiDaS-inferred depth as additional input but this hasn't improved performance. This remains an area for future investigation. We will add a discussion in the revised version.

Q: Training with more data

A: Please see the answer to Reviewer fqXP.

Q: Changing the number of basis elements

A: Please see the answer to Reviewer 6f3c.

We appreciate the reviewer's positive feedback on our paper's relevance, novelty, and clarity. We will now address your specific questions and suggestions raised by the reviewer.

Q: Limited Evaluation, MiDaS is quite old. Compare to video depth estimation methods.

A: Thank you for your feedback. We acknowledge that MiDaS has been around for some time. However, we'd like to clarify that our comparisons were made using MiDaS 3.1, specifically the dpt_beit_large_512 version (Birkl et al. 2023). This is an improved version of MiDaS that utilizes the DPT architecture, representing a more current state of the method. To strengthen this point we also add new experiments with the Marigold depth estimation method (CVPR 2024). Our experiments show that our model achieves higher accuracy compared to both MiDaS 3.1 and Marigold. We've included a table with additional comparison to Marigold in the attached PDF (Table 3) for your reference. To clarify, as the reviewer asked, we performed all these comparisons using lifted CoTracker tracks. We believe these additional evaluations address the concern about limited comparisons and demonstrate our method's effectiveness.

Regarding video depth estimation methods, our submission already includes comparisons with CasualSAM and RobustCVD. It's worth noting that while RAFT3D infers 3D scene flow from two depth images, our setup assumes RGB frames as input without depth information, making a direct comparison challenging.

Q: Small Test Set Size. Run evaluation on PointOddessey .

A: Following the reviewer's request, we ran an evaluation on multiple test cases from PointOdyssey and compared them with CasualSAM [59], which is the most accurate baseline. The results are presented in the attached PDF (Table 2). We observed that our method generalizes well to these cases while taking much less time than CasualSAM and maintaining high accuracy. We will make an effort to extend the evaluation on PointOdyssey to include more test cases in the final paper.

Q: Robustness to 2D Point Tracking Noise.

A: We first note that the input point tracks extracted by CoTracker [18] are far from being perfect, and contain noise and outliers. To further evaluate our robustness to noise, we added additional Gaussian noise to CoTracker points and measured the final error. We observed that our method tolerates a significant level of noise. See Table 1 in the attached PDF.

Q: Using a different point tracker at test time.

A: We followed the reviewer's suggestion and tested point tracks extracted by TAPIR [10]. Although we observed some degradation in accuracy, the results are still good especially after finetunning, demonstrating our method's robustness to different point trackers. We added these results to the attached PDF (Table 4) and will add them to the revised paper as well.

Q: Robustness to Speed of Dynamic Motion

A: To clarify, our limitation to the speed of dynamic motion comes from CoTracker's [18] limitation to track fast motion. In our evaluation set, we did not see such a limitation including in the Nvidia Dynamic Scenes Dataset [58] which contains a variety of motion characteristics.

Q: Automotive datasets that have ground truth depth

A: We observed that CoTracker [18] often fails when applied to automotive scenes due to high motion blur, especially on the road. This prevents us from running large-scale evaluations on automotive datasets with the current performance of the tracking methods. We will clarify this in the revised paper.

Q: Training with more data

A: We observe that our method, though trained only on pet videos, already generalizes well to scenes from the Nvidia Dynamic Scenes Dataset [58], as shown in the main paper. During the rebuttal period, following the reviewer's request, we also tested our model on a variety of samples from the large-scale PointOdyssey dataset (see Table 2 in the attached PDF). Our results demonstrate that the model already performs well on this dataset without additional training. Due to the dataset's large scale, we were unable to train on it during the rebuttal period. For the revised version, we will make an effort to further train our method on PointOdyssey samples and report the effects of this larger-scale training.

Q: Defining Out-of-Distribution data. Benchmark on data with different speed profiles

A:The Nvidia Dynamic Scenes Dataset [58] contains several speed profiles, e.g. 'Skating', 'Truck', and 'Umbrella'. Tables 7 and 8 in the Appendix show per-scene errors and demonstrate the robustness to dynamic motion levels. For example, our depth accuracy on 'Skating' is slightly better than on 'Truck'. We will discuss this in the revised paper.

Q: Contextualizing Runtimes. Tracking run times should be taken into consideration as well.

A: Our method's runtime in Tables 1 and 2 already includes the point tracking times. Table 4 in the Appendix shows the separation of tracking time that is done as a preprocess by [18] (8.6 seconds), inference time of our network (0.16 seconds), and Bundle Adjustment time (0.24 seconds). We will clarify this in the paper.

We appreciate the reviewer's positive feedback on our paper's style and its significance to the community. Below we answer specific questions and suggestions.

Q: Maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch.

A: We explored this suggestion by using MiDaS-inferred depth as additional input to our method and applying depth loss relative to MiDaS. Currently, we haven't seen performance improvements from these approaches. However, we agree that exploiting depth models and 3D-tracking techniques is an interesting direction that is worth further investigation in future work. This could potentially lead to enhanced performance compared to learning everything from scratch.

Q: Solving a single scene efficiently.

A: Yes, our model can be efficiently fine-tuned on specific scenes using our self-supervised loss terms. We demonstrated in the original paper, in Tables 1 and 2 (Ours+FT) that this approach further improves our accuracies. Training from scratch on a single scene, however, often failed to converge well after 30 minutes of training for some scenes. In contrast, feed-forward inference with our pre-trained model takes only 0.16 seconds. This significant time difference justifies our choice of a feed-forward modeling approach.