Response to Reviewer 9gnd:

Thank you for your acknowledgment of our work. We highly appreciate your time and feedback, and address your concerns below.

About Weakness 1, ‘Broader background context for a more readable paper’

Thank you for your valuable suggestions. We appreciate your recognition of the technical content in our paper. As the current submission cannot be modified, we plan to enhance the background information in the camera-ready version to facilitate a clearer understanding of our work. In particular, we will provide a more comprehensive introduction to the research field at the beginning of our paper. Furthermore, regarding the pre-trained PT model that serves as the foundation for our proposed patch-wise tracking filters, we will include a more detailed explanation in the main paper—or, if permitted by the page limit, in a dedicated preliminary section.

About Weakness 2, ‘Introduction about the pre-trained PT model’

Thank you for pointing it out. We plan to add a more detailed explanation of the pre-trained PT model in the main paper in the camera-ready version. The default pre-trained PT model on which our proposed method is built is CoTracker. CoTracker is a transformer-based model achieving dense 2D point tracking in long video sequences. Unlike other related works, it jointly tracks the 2D points so that the dependencies on each other can help improve the accuracy and robustness of the model. During inference, the vanilla CoTracker allows a set of 2D query points and an RGB video as inputs and directly outputs the 2D location and visibility of each 2D query point on each frame. Although it significantly improved the PT performance compared with the existing works, it still suffers from the tracking location drift across frames, especially when one point disappears from the camera frustum and reappears in the field of view, as shown in Figure 9 in the Appendix. Such observations motivate us to propose the patch-wise tracking filters to extract only the robust, reliable, and sparse tracking trajectories as our pseudo-supervision.

About Weakness 3, ‘Performance on static scene dataset’

Thank you for your suggestions. We agree with your words that, 'although it is not necessarily expected to outperform methods specifically designed for static scenes, such a comparison would serve as a useful reference and help assess the method’s generalizability'. We conducted additional experiments on the Tanks and Temples dataset (Table 1 below). Due to the character limit, for other baselines, please refer to Table 2 in CF‑3DGS. Compared with CF‑3DGS (SOTA static-scene method that uses GT intrinsics), although our approach does not surpass the SOTA methods specialized for static scenes with extra supervision, such low errors can also support our generalizability.

Table 1: Camera Pose Evaluation on static-scene Tanks and Temples dataset (ATE$\downarrow$/RPT trans$\downarrow$/RPE rot$\downarrow$)

We summarize below the reasons why meaningful comparisons with existing works on the 7‑Scenes and ScanNet datasets are not feasible:

1. Access to the ScanNet dataset requires an application process and approval, which takes additional time.
2. After recheck, we found that the existing 7‑Scenes evaluations (e.g., DUSt3R / Fast3R) are based on unordered images and pairwise pose metrics. In contrast, our method fundamentally relies on long, continuous video sequences and temporal consistency. Since the official protocol merges multiple short sequences into unordered inputs and does not provide a single long sequence, it is not possible to conduct a fair or meaningful comparison on this dataset.

About Weakness 4, ‘include the number of frames per scene when reporting runtime comparisons in Table 9 and other related tables’

Thank you for your helpful suggestions. Since the experimental section of our paper has already demonstrated the high efficiency property of our method, and due to the space limits of rebuttal, we only present here the runtime of our method on different datasets along with the corresponding number of frames, as shown in the following Tables 2, 3, and 4. We plan to incorporate these additional results into the runtime tables in the camera-ready version to make it more complete.

Table 2: Quantitative Runtime Results on DAVIS with Corresponding Frame Count

Table 3: Quantitative Runtime Results on iPhone with Corresponding Frame Count

Table 4: Quantitative Runtime Results on NeRF-DS with Corresponding Frame Count

About Question 1, ‘Do other point trackers work as well?’

Thank you for your question. As stated in Lines 141–143 of the main paper, although our proposed patch-wise tracking filters do an iterative process until there are $B$ tracked points in each frame, such filters do $\underline{not}$ conduct any $\underline{repeated}$ refinement or optimization of point tracking on the $\underline{same}$ tracking trajectory. This is the main reason why our runtime grows almost linearly with frame count, as shown in Figure 4 of the main paper. Thus, the pre-trained PT model on which our proposed filters are based must be a feed-forward prediction model, outputting point locations and visibility; we will clarify this further in the camera-ready version.

In Table 8 of the main paper, we show ablation experiments on 2 different popular pre-trained PT models, CoTracker and CoTracker3. The results show that our proposed method is nearly independent of building on the particular PT models, as long as these requirements are met.

Besides, Table 5 below summarizes all baseline PT methods reported in the CoTracker and CoTracker3 papers. CoTracker and CoTracker3 are the only two methods satisfying our requirements. Thus, other PT models cannot be incorporated in our proposed method due to architectural incompatibility.

Table 5: Properties of All Baseline Methods Reported in CoTracker and CoTracker3.

We are also happy to discuss any other proper pre-trained PT models to explore potential improvements on our current proposed method. Please feel free to let us know.

We hope our rebuttal has addressed your questions and resolved your concerns regarding our work. Please let us know if there is anything else we can clarify. We would greatly appreciate a brief reply to share your thoughts. Thank you very much for your time.

Thank you for your acknowledgment of our paper. The discussion period will end in several hours. We would like to kindly remind you to submit the Mandatory Acknowledgment. Thank you!

Response to Reviewer izms:

Thank you for your acknowledgment. We really appreciate your time and feedback, and address your concerns below.

About Weakness 1, ‘Abrupt introduction on COLMAP and Visual Odometry’

Thank you for your suggestions on paper writing. We plan to add more background information at the beginning of the paper, to help us make the introduction of visual odometry, and the state-of-the-art COLMAP more fluent.

About Question 1, ‘Results with a much weaker tracker’

Thank you for your question. As stated in Lines 141–143 of the main paper, although our proposed patch-wise tracking filters do an iterative process until there are $B$ tracked points in each frame, such filters do $\underline{not}$ conduct any $\underline{repeated}$ refinement or optimization of point tracking on the $\underline{same}$ point trajectory. This is also the main reason why our runtime grows almost linearly as frame count increases, as shown in Figure 4 of the main paper. Thus, the pre-trained PT model on which our proposed filters are based must be a feed-forward prediction model, outputting point locations and visibility; we will clarify this further in the camera-ready version.

In Table 8 of the main paper, we show ablation experiments on 2 different popular pre-trained PT models, CoTracker and CoTracker3. The results show that our proposed method is nearly independent of building on the particular PT models, as long as these requirements are met.

Besides, Table 1 below summarizes all baseline PT methods reported in the CoTracker and CoTracker3 papers. CoTracker and CoTracker3 are the only two methods satisfying our requirements. Thus, other PT models cannot be incorporated in our proposed method due to architectural incompatibility.

Table 1: Properties of all baseline methods reported in the CoTracker and CoTracker3.

We are also happy to discuss any other proper pre-trained PT models to explore potential improvements on our current proposed method. Please feel free to let us know.

We hope our rebuttal has addressed your questions and resolved your concerns regarding our work. Please let us know if there is anything else we can clarify. We would greatly appreciate a brief reply to share your thoughts. Thank you very much for your time.

Thank you once again for your time and thoughtful feedback. As the discussion period concludes in three days, we remain fully available to address any remaining concerns or questions regarding our work. Please don’t hesitate to let us know if any further clarification is needed. We sincerely appreciate your engagement.

Thank you once again for your valuable time and commitment. We would like to kindly note that the discussion period will conclude in approximately 48 hours. We greatly appreciate the opportunity to address any remaining questions or concerns you may have regarding our work. If any points require further clarification, please don’t hesitate to reach out. Thank you very much.

Response to Reviewer 79BF:

Thank you for your acknowledgment of our paper. We sincerely appreciate your time and feedback, and address your concerns below. As your concerns were not explicitly numbered or listed, we summarize them as follows.

About Weakness 1, ‘how many correct point trajectories..xxx..starting from a low to high amounts of moving outliers.’

Thank you for your question. We believe there must be a misunderstanding, and we should make it clear in the camera-ready version with the following clarifications.

Firstly, we do not define point trajectories by correctness. Both inlier and outlier trajectories can be correct or incorrect, and these datasets lack GT optical flow or tracking labels, so such a definition would be meaningless. Instead, we classify trajectories into inliers and outliers, which directly motivates our proposed outlier-aware joint optimization.

Secondly, the number of moving outliers does not affect trajectory accuracy. Our patch-wise tracking filters are built on CoTracker, which can robustly track trajectories regardless of the number of moving points, as discussed in the CoTracker paper. For further analysis on our model’s capacity to handle different amounts of outliers, please see our next response.

About Weakness 2, ‘What amount of outliers can be handled?’

Thank you for the question. By “amount of outliers,” we assume you mean the percentage of outliers among all trajectories. We want to clarify that model performance depends not only on this percentage but also on the appearance pattern of outliers. To illustrate, we provide a table comparing performance, outlier percentage, and appearance patterns across failure cases and one success case (motion masks for MPI-Sintel are manually annotated). As shown in Table 1 (below) and Fig. 21 (Appendix), a high outlier percentage (e.g., Ambush_6) can still yield good results, while lower percentages can fail when the appearance pattern is extreme. Thus, there is no strict threshold for the outlier percentage that leads to failure.

Table 1: Failure Case Analysis by Outlier Percentage and Appearance Pattern on MPI-Sintel dataset.

About Weakness 2, ‘Why not the Tukey estimator? Other estimator?’

Thank you for asking this question. To our knowledge, we believe what you mean by ‘the Tukey estimator’ is Tukey’s biweight estimator. In the weighting function of Tukey’s biweight estimator,

Tukey’s biweight completely discards residuals beyond the cutoff $c$, a hard threshold that must be tuned for each video. In contrast, our Cauchy weighting learns all parameters, allowing our outlier-aware joint optimization to generalize across videos.

We are also happy to discuss other estimators to explore potential improvements on our current proposed method. Please feel free to let us know.

About Weakness 3, ‘How would the proposed method perform with more GT supervisions?’

Thank you for your question. More GT supervision (depth, camera pose, motion masks, 3D point clouds, camera intrinsic) will change the architecture of our pipeline, which is specifically designed for RGB-only supervision. So under such circumstances, our model needs to be carefully redesigned. But, theoretically, like most other models, we believe our method will perform better with more GT supervision. We might leave this as a future research path.

Furthermore, we want to reclarify that our biggest contribution is our capability to accurately and efficiently estimate the camera parameters in dynamic scenes solely supervised by a single RGB video, without any other GT supervision.

About Question 1, ‘Analysis on $\Gamma^{raw}$'

Thank you for your suggestion. As shown in Table  7 of the main paper, replacing our two-stage strategy with a randomly initialized and learnable $\Gamma^{raw}$ leads to a clear performance drop. To address the request for a deeper analysis, we conducted additional ablations on $\Gamma^{raw}$ (see table below). 'Meaningless' indicates that when $\Gamma^{raw}$ is frozen in Stage 1, other value settings behave the same as setting $\Gamma^{raw}=1$, so ablations 2 and 3 duplicate the settings of ablation 1 and the full model.

Table 2: Analysis on $\Gamma^{raw}$ on NeRF-DS dataset. (Stage 1: 200 iterations; Stage 2: 50 iterations.)

Such comparisons show that our initialization strategy on $\Gamma^{raw}$ in different stages is very important; otherwise, the performance drops dramatically.

About Question 2, ‘more examples in different scenes, ranked by the level of difficulties, e.g., ..xxx... (low dynamic to high dynamic scenes)’

Thank you for your questions. To clarify, the difficulty of camera evaluation depends on multiple factors (outliers, camera motion, appearance pattern, scene dynamics, etc.). Analysis on parts of them is in our answers to About Weakness 2, ‘What amount of outliers can be handled?’. To our knowledge, there are no explicit quantitative or qualitative metrics to rank it. We will clarify this in the camera-ready version.

Unlike the related methods (Table 1 in the main paper), mostly supported by experiments on 30 - 50 videos, our results are supported by experiments on 5 popular public camera evaluation datasets covering about 70 videos across different difficulty levels. To make this clearer, we summarize the dataset characteristics (Section C of the Appendix) in the following table and will include it in the camera-ready version.

Table 3: Dataset Features

Due to the character limit, please refer to our response to Reviewer  9gnd, About Weakness 3, ‘Performance on static scene dataset,’ for a further investigation of the generalization ability of our method. Thank you.

About Question 3, ‘More qualitative and quantitative analysis on failure cases’

Thank you for your advice. We will add the following discussions to the paper in the camera-ready version. As shown in Fig. 21 (Appendix, L586–589), our method fails when videos are dominated by large, fast-moving objects (e.g., humans, dragons). We attribute it to two main reasons: (1) a high proportion of outliers and (2) extreme outlier patterns where large objects sweep and cover the screen. These factors hinder robust, sparse hinge-like relations extracted for accurate pseudo-supervision. More details about the reasons can be seen in our answers to About Weakness 2, ‘What amount of outliers can be handled?’.

We hope our rebuttal has addressed your questions and resolved your concerns regarding our work. Please let us know if there is anything else we can clarify. We would greatly appreciate a brief reply to share your thoughts. Thank you very much for your time.

Thank you once again for your time and thoughtful feedback. As the discussion period concludes in three days, we remain fully available to address any remaining concerns or questions regarding our work. Please don’t hesitate to let us know if any further clarification is needed. We sincerely appreciate your engagement.

Thank you once again for your valuable time and commitment. We would like to kindly note that the discussion period will conclude in approximately 48 hours. We greatly appreciate the opportunity to address any remaining questions or concerns you may have regarding our work. If any points require further clarification, please don’t hesitate to reach out. Thank you very much.

Thank you for your valuable feedback. Although the deadline has been somewhat extended, we would like to kindly remind you that the discussion period will end in approximately 48 hours. If you have any more questions, please let us know ASAP so that we have enough time to address your concerns. If you believe our rebuttal can help address your concerns, we sincerely expect you to consider improving your score as the final rating. Thank you!

PS: Please do not forget to submit the Mandatory Acknowledgement.

Thank you for your reviews! We would like to kindly remind you that the discussion period will end in approximately 24 hours. If there are any more concerns, please let us know so that we can address them ASAP. We also sincerely invite you to see our discussions with other reviewers if you share similar concerns with them. We sincerely expect you to consider improving your score as the final rating. Thank you again!

Response to Reviewer nBqP:

Thank you for your acknowledgment of our paper. We sincerely appreciate your time and feedback, and address your concerns below.

About Weakness 1, ‘The blurry words in the figures’

Thanks for your suggestions. After re-checking, we found that some figures were not correctly saved as vector graphics. Since the article cannot be modified at this stage, we plan to correct it in the camera-ready version.

About Weakness 2, ‘Words’ contradictory’

Thank you for pointing out this misunderstanding. What we meant is that our method is the first RGB-only supervised approach for dynamic scenes that is both accurate and efficient, as existing RGB-only methods are either slow or inaccurate. We will remove the word “first” in the camera-ready version.

About Weakness 3, ‘How patch size affects performance across different scene types?’

Thank you for raising this question. We would like to clarify that although patch size can influence performance, the scene type is not a factor in this process; thus, we do not adjust patch size based on scene types. Instead, as noted in Line 200 of our main paper, the patch size $w$ is determined solely by the frame size.

Specifically, when the input aspect ratio (long/short side) exceeds 1.5, we consider the image elongated. Camera motion in such cases causes obvious uneven scene changes across axes, so we adopt a smaller patch size to capture more valid high-texture patches. Otherwise, we use a larger patch size. The value of $w$ is chosen as a trade-off between memory cost and efficiency, but it should definitely be adjusted for extremely small or large frame sizes. We will add such discussions to the camera-ready version. Further sensitivity analysis on $w$ is provided in our response to About Question 1, ‘Sensitivity analysis on $w$ and $\tau_{var}$’.

About Weakness 3, ‘Sensitivity analysis of the texture threshold’

Thank you for your advice. Please see our subsequent answers to About Question 1, ‘Sensitivity analysis on $w$ and $\tau$’, for a detailed discussion about the sensitivity analysis of the texture threshold.

About Weakness 4, ‘The position of limitations’

Thank you for your suggestions. We will move the limitation section to the main paper in the camera-ready version.

About Weakness 4, ‘More discussion on when and why the method fails?’

Thank you for your advice. We should discuss more in the camera-ready version. As shown in Fig. 21 (Appendix, L586–589), our method fails when videos are dominated by large, moving humans and dragons. We attribute it to two main reasons: 1) high outlier percentage; 2) extreme outlier patterns where large objects sweep and cover the screen. These factors hinder robust, sparse hinge-like relations extracted as accurate pseudo-supervision. More details can be seen in our answers to About Question 2, ‘What percentage of moving pixels causes failure? ..xxx.. failure cases?’.

About Question 1, ‘How did you determine $w$=12/24 and $τ_{var}$=0.1?’

Thank you for your question. Please refer to our answers to About Weakness 3, ‘How patch size affects performance across different scene types?’ for a thorough analysis of $w$.

We sincerely apologize for the typo in the paper. The value “$τ_{var}$=0.1” should be corrected to “$τ_{var}$=0.01”. We will correct this typo in the camera-ready version.

We empirically set $\tau_{var}=0.01$ and use this fixed value for all videos. Eq. 1 in main paper selects relatively high-texture patches by keeping those whose intensity variance exceeds $0.01$ $\times$ $\sigma*$ ($\sigma*$ is the maximum patch variance in a frame). This removes textureless regions (e.g., plain backgrounds, white walls; see Fig. 9 in Appendix) while retaining useful information, as only patches with variance below 1% of the maximum are discarded.

About Question 1, ‘Sensitivity analysis on $w$ and $\tau_{var}$’

Thank you for asking this question. We conduct the sensitivity analysis on these two parameters, as shown in the subsequent two tables with detailed discussions. We plan to add the following two tables to the ablation study section of our main paper in the camera-ready version to make our paper more complete.

Table 1: Sensitivity Analysis on Patch Size $w$ on NeRF-DS dataset

Table 2: Sensitivity Analysis on Variance Threshold $\tau_{var}$ on NeRF-DS dataset

Tables 1 and 2 show that our model is more sensitive to $w$ than $\tau_{var}$. Such ablation studies also demonstrate our good value definition on these two parameters. However, too large $\tau_{var}$ makes the model fail due to a lack of useful information.

About Question 1, ‘What happens when there are insufficient high-texture patches?’

Thank you for asking this question. To clarify, what we mean by ‘high-texture patches’ is the relatively high-texture patches. We should make it clearer in the camera-ready version. Like our answers to About Question 1, ‘How did you determine w=12/24 and τ_var=0.1?’ above, in each frame, we only discard the relatively low-texture patches whose intensity variance is less than 1% of the maximum intensity variance. This efficiently avoids losing useful high-texture patches. However, we might share limitations with others in certain extreme scenarios, such as videos that predominantly feature plain walls with super-limited texture. We will leave it as our future research topic.

About Question 2, ‘What percentage of moving pixels causes failure? Is there a way to predict these failure cases?’

Thank you for asking this question. We would like to clarify that there is no strict threshold for the percentage of moving pixels that leads to failure. Because the model’s ability to handle outliers is also affected by the appearance pattern of outliers. Table 3 below compares the performance, outlier percentage, and appearance pattern in all failure cases and one sampled success case. We plan to add this table to the main paper of the camera-ready version. Since the MPI-Sintel dataset does not provide any GT motion mask, we manually annotate the motion masks.

Table 3: Failure Case Analysis by Outlier Percentage and Appearance Pattern

As shown in the above table and Figure 21 of the Appendix, the model’s performance is not strictly determined by the proportion of outliers: in ambush_6, the model performs well even with a large number of outliers, while in others, performance degrades despite a small proportion of outliers. The appearance pattern also affects the performance of the model.

Currently, there is no quantitative indicator available to predict whether the method will succeed or fail. But we highly suggest forecasting based on the appearance pattern.

About Question 3, ‘Testing on extremely long videos (>2000 frames)’

Thank you for your questions. To our knowledge, existing test datasets and prior methods do not provide extremely long videos (>2000 frames) nor report experiments under such conditions. So we select two sequences exceeding 2000 frames (footlab_3rd and scene_j716_3rd) from the Point Odyssey dataset. However, even after trimming the sequence to 1500 frames, our model still encounters RAM capacity limitations on our setup, where all experiments are conducted on a single NVIDIA A100 GPU (40 GB) with 256 GB of system RAM. The longest video in our current experiments contains 881 frames (bell in NeRF-DS). We plan to investigate methods for reducing GPU and RAM consumption as part of our future work.

We hope our rebuttal has addressed your questions and resolved your concerns regarding our work. Please let us know if there is anything else we can clarify. We would greatly appreciate a brief reply to share your thoughts. Thank you very much for your time.

Thank you once again for your time and thoughtful feedback. As the discussion period concludes in three days, we remain fully available to address any remaining concerns or questions regarding our work. Please don’t hesitate to let us know if any further clarification is needed. We sincerely appreciate your engagement.

Thank you once again for your valuable time and commitment. We would like to kindly note that the discussion period will conclude in approximately 48 hours. We greatly appreciate the opportunity to address any remaining questions or concerns you may have regarding our work. If any points require further clarification, please don’t hesitate to reach out. Thank you very much.

Thank you for your valuable feedback. Although the deadline has been somewhat extended, we would like to kindly remind you that the discussion period will end in approximately 48 hours. If you have any more questions, please let us know ASAP so that we have enough time to address your concerns. If you believe our rebuttal can help address your concerns, we sincerely expect your final rating. Thank you!

PS: Please do not forget to submit the Mandatory Acknowledgement.

Thank you for your acknowledgment of our paper. The discussion period will end in several hours. We would like to kindly remind you to submit both the final rating and the Mandatory Acknowledgment. Thank you!