PoseCrafter: Extreme Pose Estimation with Hybrid Video Synthesis

We thank the reviewer for acknowledging the effectiveness and practicality of our approach. Following the suggestion of the reviewer, we provide detailed responses to address the reviewer's concerns.

Q1: Clarification about novelty.

The novelty of our PoseCrafter lies elsewhere despite building on existing modules such as video interpolation, pose-conditioned novel view, and RANSAC filtering. Our main contribution is in the adaptation and integration of these components to holistically address the under-explored and challenging problem of pose estimation with small or no visual overlap.

• We are the first to propose a hybrid video generation (HVG) module that combines video interpolation and pose-conditioned novel view synthesis to generate high-quality intermediate frames for subsequent pose estimation.
• We introduce a Feature Matching Selector (FMS) that deterministically selects the most informative frames suitable for pose estimation. This eliminates the need for expensive statistical self-consistency scoring in existing works.
• Our method requires no additional training or fine-tuning, yet achieves significant performance improvements over existing methods. This not only greatly reduces training costs but also makes our approach a flexible framework, where components can be potentially replaced with more effective alternatives in the future.

We believe that the overall system design and its demonstrated impact across four benchmarks reflect a meaningful and practical innovation, which should be beneficial for the community and related researchers.

Q2: Memory Cost and Runtime Discussion.

We thank the reviewer for their important points regarding runtime and computational efficiency. To address this concern, we provide a detailed comparison of the runtime and memory usage of different stages of our pipelines and those of InterPose. For fair evaluation, we divide the pipeline into two stages: video generation and pose estimation. The video synthesis stage in both our method and InterPose is referred to as video generation. The subsequent process to obtain the final pose using self-consistency score in InterPose or feature matching selector in our method is referred to as pose estimation.

Since the original implementation of InterPose is not publicly available, we reimplemented its pipeline according to the details described in the paper. We refer to this reproduction as InterPose$^{\ddagger}$ for subsequent comparison. The results are summarized in the table below.

We can see that our approach significantly reduces the pose estimation time while achieving comparable or even higher pose accuracy. This improvement comes from the fact that InterPose generates 8 video sequences for each image pair and samples a subset of 11 frames from each video to compute the self-consistency score. In contrast, our approach employs a leaner and more efficient pipeline. We generate only a single hybrid video and apply a deterministic feature-based frame selection strategy.

Although our approach has a slightly higher memory footprint during the video generation phase, the pose estimation phase remains lightweight and practical in terms of memory and speed.

We will include this analysis in the revised manuscript to better demonstrate the computational efficiency and scalability of our approach.

Dear Reviewer,

The authors have replied to your concerns, and it would be very helpful if you could engage in a discussion to clarify your concerns.

Best,

AC

We thank the reviewer for the thoughtful and constructive feedback. We give our response to each concern in the following:

Q1: Would the variability of diffusion-based DynamiCrafter causes early point of failure?

We thank the reviewer for raising this important concern. To evaluate the impact of generative stochasticity in DynamiCrafter, we conducted experiments on the Cambridge Landmarks dataset under yaw changes of [50$^{\circ}$-65$^{\circ}$] using 5 different random seeds to generate intermediate video sequences. The results are shown in the following table. We can find that our method consistently outperforms other methods within a smaller range of variations. This indicates that while some variability is inherent to the diffusion-based generative process, our pipeline remains robust and stable across different runs. We will include this analysis in the revised version to better illustrate the reliability of our method.

Q2: Runtime and Memory Cost Discussion.

We thank the reviewer for highlighting the importance of efficiency. Although InterPose generates 8 video sequences for each image pair and samples a subset of 11 frames from each video to compute the self-consistency score, our approach adopts a more streamlined and efficient pipeline. Specifically, we generate only a single hybrid video and apply a deterministic and efficient frame selection strategy based on feature matching.

To address the reviewer's concerns, we provide a detailed comparison of runtime and memory usage at different stages. For fair evaluation, we divide the pipeline into two stages: video generation and pose estimation. The video synthesis stage in both our method and InterPose is referred to as video generation. The subsequent process using the self-consistency score in InterPose or the feature matching selector in our method is referred to as pose estimation. Since the original implementation of InterPose is not publicly available, we re-implemented its pipeline based on the details provided in the paper. We refer to this reproduction as InterPose$^‡$ for all subsequent comparisons. As shown in the table below, our approach significantly reduces pose estimation time while achieving comparable or even higher pose accuracy. Although our approach incurs a slightly higher memory cost during video generation, it remains affordable for practical usage. We will include this analysis in the revised version to clarify the computational efficiency and scalability of our approach.

Q3: Evaluation with RoMa [4] and LoFTR [5].

To evaluate the impact of different feature matching strategies, we conducted experiments on the Cambridge Landmarks dataset under yaw changes of [50$^{\circ}$-65$^{\circ}$], using RoMa and LoFTR as alternatives to ORB in our Feature Matching Selector (FMS).

Interestingly, we observed that each matcher led to the selection of a different subset of intermediate frames, ultimately affecting the pose estimation outcome. As shown in the table below, both RoMa and LoFTR resulted in inferior results compared to ORB. We believe the possible reason lies in the difference in robustness across methods. Learning-based methods such as RoMa and LoFTR may perform less reliably on our synthesized intermediate views. This is likely because they are trained on high-quality natural images. In contrast, ORB is more appropriate in our setting due to its strong robustness and higher efficiency. We will include these comparative results and analysis in the revised version to further demonstrate the robustness and efficiency of our frame selection strategy.

Q4: How would VGGT [3] perform in our context?

We sincerely thank the reviewer for the valuable suggestions. To evaluate the performance of VGGT in our pipeline, we conducted comparative experiments on the Cambridge Landmarks dataset under yaw changes of [50$^{\circ}$-65$^{\circ}$].

We first evaluate the two models in complex scenarios where only image pairs with small or no overlap were input. As shown in the second and fourth rows of the table below, DUSt3R consistently outperforms VGGT on our data with small or no overlap.

We then integrate these two models into our pipeline and evaluate their performance. As shown in the third and fifth rows, both configurations achieve significant improvements over their directly estimated counterparts. The version using DUSt3R achieves a higher accuracy than the version using VGGT. These results demonstrate that our method is compatible with different pose estimators and consistently enhances their performance.

Compared to VGGT, DUSt3R is better suited for pose estimation on small or non-overlapping image pairs. Furthermore, our method also yields more noticeable improvements when applied to DUSt3R. Although we believe that fine-tuning VGGT on small-overlap data or adjusting the hyperparameters of our pipeline could further improve the performance of VGGT, these efforts are beyond the scope of this work and rebuttal. We plan to explore them in future work.

Q5: Related works and citations.

We thank the reviewer for pointing out the relevant work [1, 2, 3, 4, 5]. Among them, [1, 2, 4, 5] explore different feature matching strategies and are highly relevant to our Feature Matching Selector module. We will include proper citations and discussion of these works in the revised version to better contextualize our contribution. Furthermore, we have already conducted experimental analyses of the matching methods proposed in [4] (RoMa) and [5] (LoFTR), as detailed in our response to Q3. These empirical comparisons will also be incorporated into the main paper. VGGT [3] is a recently proposed method designed for camera pose estimation and is also highly relevant to our task. We have included comparative experiments using VGGT in our response to Q4. , and will add the corresponding results and citations in the revised version.

[1] Zhang, Zichao, Torsten Sattler, and Davide Scaramuzza. "Reference pose generation for long-term visual localization via learned features and view synthesis." International Journal of Computer Vision 129.4 (2021): 821-844.

[2] Trivigno, Gabriele, et al. "The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Wang, Jianyuan, et al. "Vggt: Visual geometry grounded transformer." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[4] Edstedt, Johan, et al. "RoMa: Robust dense feature matching." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Sun, Jiaming, et al. "LoFTR: Detector-free local feature matching with transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

We thank the reviewer for the thoughtful and constructive feedback. Our responses to the concerns raised by the reviewer are as follows:

Q1: The proposed method is only tested on a single video interpolation model. Could authors provide experiments on more (recent) models?

Since our task focuses on input image pairs with small or no overlap, we require video interpolation models that can handle large motion or significant viewpoint changes.

To validate the generality of our pipeline, we additionally experimented with ToonCrafter [1]. This recent model is known for its efficiency in handling large-motion interpolation. We evaluated the model on the Cambridge Landmarks benchmark under yaw changes ranging from [50$^{\circ}$-65$^{\circ}$]. As shown in the table below, ToonCrafter [1] performs slightly worse than DynamiCrafter. This is likely because ToonCrafter is primarily designed for cartoon-style data. However, it still achieves clear improvements over directly using only the input image pair. These results confirm that our hybrid pose estimation framework is compatible with a variety of video generation backbones. We will include them in the revised version to emphasize the flexibility and robustness of our approach.

Q2: Evaluation on more recent feature matching methods such as RoMa[2].

We follow the recommendation of the reviewer to incorporate RoMa as an alternative feature matching method within our Feature Matching Selector and evaluated it on the Cambridge Landmarks benchmark under yaw changes of [50$^{\circ}$-65$^{\circ}$].

As shown in the table below, RoMa selected a different set of intermediate frames compared to ORB, while its performance falls behind ORB. One possible reason is that RoMa may not be robust enough to handle our synthesized intermediate views, as it was trained primarily on the natural high-quality images. In contrast, ORB demonstrates greater robustness in our scenarios and offers a significant advantage in computational efficiency. For these reasons, we adopt ORB for the Feature Matching Selector (FMS) in this work. We will include these comparative results and analysis in the revised version.

Q3: Inference time analysis.

We thank the reviewers for the valuable suggestions and have added a comparison of the inference time required for frame selection between our method and InterPose. For a fair evaluation, we define the pose estimation time as the duration taken to obtain the final pose. For InterPose, this includes the time spent computing the Self-Consistency Score. For our method, it includes the time spent using the Feature Matching Selector. We then compare the two methods based on this definition. Since the original implementation of InterPose is not publicly available, we re-implemented its pipeline based on the details provided in the paper. We refer to this reproduction as InterPose$^‡$ for subsequent comparisons.

As shown in the table below, our method offers a substantial advantage in computational efficiency. In contrast, InterPose requires the generation of 8 video sequences for each input image pair. From each sequence, it samples $m = 11$ frame subsets with 10 random samples and 1 uniformly spaced sample. It then computes pose estimates for each subset. A self-consistency score is evaluated across these subsets to determine the final pose. This process incurs significant computational overhead due to repeated inference and scoring steps.

In contrast, our method leverages a simple yet effective deterministic frame selection strategy based on feature matching. Features are extracted only once, and the most informative intermediate frames are selected directly for pose estimation. This significantly reduces time costs while maintaining highly competitive accuracy. These results highlight the practical benefits of our pipeline for large-scale applications. We will include this analysis in the revised version.

Q4: Alternative: Using DUSt3R confidence to rank video frames.

We thank the reviewer for suggesting the use of DUSt3R confidence scores as an alternative strategy for video frame selection. To evaluate its effectiveness, we conducted experiments by selecting top-ranked frames from hybrid videos based on four confidence thresholds (20%, 40%, 60%, and 80%). The corresponding results are shown in the table below.

Analysis shows that setting the threshold to 20% or 40% results in decreased performance of pose estimation compared to using the hybrid video sequence. Increasing the threshold to 60% or 80% improves the results, but the accuracy still lags behind our proposed feature matching selector (FMS).

In terms of computational efficiency, confidence-based selection incurs considerable overhead. Specifically, all video frames must first be processed by DUSt3R to compute the confidence map before any selection can be made. This step alone incurs significant time and memory costs. Furthermore, as the threshold increases, the number of selected frames increases, and the overall time continues to grow. In contrast, our FMS achieves superior efficiency and accuracy with only a single feature extraction step to immediately select the most informative frames. We will incorporate this analysis into the revised version to highlight the practical advantages of our approach in terms of computational efficiency and pose estimation performance.

Q5: Smaller gain on NAVI and DL3DV-10K.

We appreciate the reviewer’s observation. Our method is specifically designed for the challenging setting where the input images have small or no overlap. As a result, its improvements are particularly significant on outward-facing datasets such as Cambridge Landmarks and ScanNet, where the overlapping regions between widely spaced views are minimal. In contrast, center-facing datasets such as DL3DV-10K and NAVI present a different scenario. Although the camera angles vary significantly, the captured objects remain largely centered. This results in substantial visual overlap between frames. Consequently, baseline methods already perform relatively well on these datasets with little room for improvement. The smaller performance gains in these cases are therefore expected and consistent with the nature of the problem being addressed. We will include this explanation in the revised version to clarify the performance differences between dataset types and to better highlight the strength of our method in small-overlap scenarios.

[1] Xing J, Liu H, Xia M, et al. Tooncrafter: Generative cartoon interpolation[J]. ACM Transactions on Graphics (TOG), 2024, 43(6): 1-11.

[2] Edstedt J, Sun Q, Bökman G, et al. Roma: Robust dense feature matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19790-19800.

We thank the reviewer for their positive feedback regarding the clarity of our paper and the performance improvements demonstrated in our experiments. We provide our response to all concerns raised by the reviewer as follows:

Q1: Concerns about generalization and the heuristic nature of the relay frame selection strategy.

We thank the reviewer for raising this important point. We would like to clarify that although our relay frame selection strategy was initially inspired by empirical observations on the Cambridge Landmarks dataset, it is actually transferable to other data with different settings.

We have extended our evaluation beyond the Cambridge Landmarks dataset to include ScanNet, DL3DV-10K, and NAVI. These datasets feature diverse scene types and motion patterns. The updated results shown in the table below indicate that the selection of only the start and end frames along with their immediate neighbors {${I₀, I₁, I_{T−1}, I_T}$} as the four relay frames consistently produces the lowest mean rotation error and the highest stability. This trend holds across different datasets and settings. It supports our intuition that using more frames may introduce geometric inconsistencies, and too few relay frames can lack sufficient information for accurate pose estimation.

Although the technique to select relay frames seems to be simple and deterministic, it has proven to be surprisingly robust and effective across datasets with varying settings. We can further explore its principled proof in our future work. We will also include these new experimental results and add more clarification in the revised version.

Q2: The method appears to be a composition of off-the-shelf components.

Despite building on existing components, our method addresses a fundamentally challenging and under-explored problem: camera pose estimation under small or non-overlapping images. Most existing methods fail under these conditions due to the lack of reliable visual correspondences.

We further clarify our core contributions in the following:

• We couple the video interpolation model DynamiCrafter with the pose-conditioned NVS model ViewCrafter. This combination forms our Hybrid Video Generation (HVG) framework. Our HVG synthesizes high-quality intermediate frames to support more accurate subsequent pose estimation.
• We incorporate robust feature descriptors to propose the Feature Matching Selector (FMS) module. Our FMS deterministically selects the most informative frames for pose estimation. This eliminates the need for expensive statistical self-consistency scoring used in existing methods.
• By integrating these components, we propose a simple yet effective pipeline for reliable pose estimation between image pairs with small or even no overlap. Experiments confirm that our method consistently outperforms existing approaches in these challenging scenarios.

Our work goes beyond a pure engineering effort. It reshapes the role of generative models in supporting geometric tasks and extends the applicability of existing pose estimators. We believe that our contributions to this work can benefit the community and inspire further research in related areas.

Q3: The description of the ablation (Table 1) is unclear.

We thank the reviewer for pointing out the ambiguity. We apologize for the lack of clarity in Table 1 and will revise the table caption and description in the main text for better readability. Specifically, the setting #Frames=6 corresponds to {${I₀, I₁, I₂, I_{T−2}, I_{T−1}, I_T}$}, and #Frames=8 corresponds to {${I₀, I₁, I₂, I₃, I_{T−3}, I_{T−2}, I_{T−1}, I_T}$}.

Dear Reviewer,

The authors have replied to your concerns, and it would be very helpful if you could engage in a discussion to clarify your concerns.

Best,

AC