LEAP: Liberate Sparse-View 3D Modeling from Camera Poses

We appreciate the highly positive feedback! Thanks for acknowledging that our paper is well-written and the method is novel, as well as the experiments are convincing. We will address your comments as follows.

1. [Cross-dataset evaluation] Thanks for pointing this out. Cross-domain generalization is important for real-world applications. We appreciate your insight! We study the cross-data transfer capability as shown in Table 1 of the paper, where we train on 13 ShapeNet categories (Kubric-ShapeNet-seen) and test its generalization on 10 novel categories (Kubric-ShapeNet-novel). LEAP demonstrates robust generalization in this test with ~3 dB PSNR higher than the next-best method FORGE. To further address your concern, we include another cross-domain evaluation, where the model is tested on Google Scanned Objects (GSO) [1], collected from another domain. LEAP demonstrates better performance than FORGE in this setting (24.12 v.s. 22.78 PSNR). We will add more discussion in the revision accordingly.

[1] Downs, Laura et al. “Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items.” ICRA 2022.

2. [Minors] Thanks for pointing out the related work. We will cite the paper and add discussion. RUST resolves the necessity of target camera poses by prompting the model with a partial target image, which further makes SRT to be more applicable in real applications.

We highly appreciate the detailed feedback! Thanks for acknowledging that the paper provides convincing results. We will address your concerns as follows. (1/2)

1. [Comparison with SRT] Thanks for the suggestion. We described the limitations of LEAP on unbounded scenes on page 9 of our paper (asked by the reviewer). We further clarify the differences with SRT: i) LEAP targets at 3D modeling -- it uses a 3D representation that can export the density field and render depth (with evaluation results shown below). Dealing with unbounded scenes is a long-standing problem with 3D-aware methods [1], which is not our current focus. SRT targets at novel view synthesis based on 2D representations. It can be applied to unbounded scenes, but it cannot produce 3D geometry directly. To address your concern, we will specify the scene-level results to bounded scenes as you suggested; ii) Moreover, the absence of SRT in DTU experiments is because SRT requires large training data and it cannot adapt to the small DTU data – It is a property of SRT rather than intentionally excluding it. In contrast, LEAP works better with limited training data. All comparisons of objects and bounded scenes are fair and extensive, which is acknowledged by all other reviewers and demonstrates the effectiveness of LEAP. We will better clarify these points in the revision.

[1] Barron, Jonathan T. et al. “Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.” CVPR 2022.

2. [The pose-free approach] We never claim we are “the first” in the first paragraph of page 4 (asked by the reviewer). What we claim is “a novel pose-free paradigm” – LEAP is indeed different from SRT as it is based on 3D representations for 3D modeling. We will make this more clear. We give credit to SRT, which is the first pose-free work and it performs novel view synthesis based on 2D representations.

3. [DTU details] Pose estimation on DTU is not solved. As shown in Fig 7 of SPARF, the pose errors are still large with sparse inputs.

4. [Fig 8] The location of the point visualized in the volume clearly matches the GT views. Simply placing the density near the mean value will not match the GT. Besides, the reviewer mentions “estimating pose given images of a single dot is impossible”. However, LEAP doesn’t estimate any poses. To further address your concern, another piece of evidence is in Fig 16, where LEAP shows large depth errors with one input, and it is almost resolved with two inputs, verifying LEAP uses multiview information to resolve depth ambiguity.

5. [Camera pose during training] We described the related content in the introduction. We will make it clearer to avoid misunderstanding. Thanks for your detailed feedback.

6. [Coarse-to-fine 2D-to-3D mapping] Your understanding is correct. It is a learned behavior. We will clarify it.

7. [Typos and figure color] We will revise them. Thanks for the detailed review!

8. [Related work] We will cite and add more discussion on the mentioned works. RUST resolves the necessity of target camera pose by prompting the model with a partial target image. VMRF and GNeRF focus on the dense-view setting.

9. [SRT training details] We strictly follow the training protocol with officially verified code (4M iterations). The visualization results are comparable to the ones in the SRT GitHub repo. Besides, we clarify the difference between the object-centric datasets that this work and the SRT paper use. The NMR data, which is used in SRT paper, is rendered by only 24 fixed camera poses. This leads to better performance on the fixed views but the trained model cannot generalize to other camera poses. The problem is reported in the SRT repo. In contrast, the camera poses we use to train LEAP and SRT are randomly sampled, which introduces more difficulties. Besides, the dataset that we experimented with contains more complicated real-object textures and geometry.

10. [Rendering features] We observe a 0.72 PSNR drop with rendering RGB directly, showing the effectiveness of rendering features.

We thank the reviewer for the positive acknowledgment of the novel idea, the clarity of writing, and the diverse experiments. We will address your comments as follows.

1. [Performance upper bound] Your understanding is correct. Using the pose-free approach is a better strategy for handling pose estimation errors. However, we argue that the pose-free approach does not limit its accuracy, as LEAP is better than prior generalizable NeRFs, i.e., PixelNeRF and FORGE when they use estimated poses. It’s reasonable that the performance is upper-bounded by the results with GT poses, and actually LEAP is better than PixelNeRF+GT poses in some cases. The mentioned blurry results in Fig. 6 are caused by another problem, and we discuss it as follows.

2. [Reconstruction quality] We appreciate your detailed observation. We note that the result in Fig. 6 (DTU experiments) is a test of LEAP on handling small-scale training data. In detail, the DTU dataset only contains 88 scenes for training, and all test scenes are unseen during training. Thus, the performance in Fig. 6 is not so “perfect”, which is caused by the limited training data scale rather than the pose-free framework itself. When using training datasets with normal scales (e.g. > 1K training instances), LEAP demonstrates good enough performance, e.g., shown in Fig. 4, Fig. 5, Fig. 12, and Fig. 13 for Omniobject3D, ShapeNet, and Objaverse, respectively. Improving the performance with limited training data is an open problem for deep learning-based methods. We believe that applying better transfer learning strategies on models pre-trained with normal-scale or larger-scale data can improve the performance in this setting. Besides, we believe applying more geometry-level regularizations on the refined neural volume and predicted radiance field can improve the performance.

3. [Refine results with more incoming images] We appreciate the valuable suggestion. The mentioned setting of refining the prediction with more incoming images is a very practical problem, as is the incremental reconstruction [1]. To address your concern, we provide a study using 5 existing views with 1 incoming view. Our solution is i) reconstructing the neural volume using the 5 existing views as an initialization, ii) using the multiview encoder to propagate information between the canonical view and the incoming view, and iii) refining the existing neural volume using the image features of the incoming and canonical view by performing 2D-3D information mapping attentions. Using the refined neural volume for predicting the radiance field demonstrates 0.37 PSNR novel view synthesis improvement, compared with the results from the existing 5 views. For reference, direct inference on 6 images observes a 0.45 PSNR improvement. The slight gap shows the feasibility of the solution and the potential of LEAP in handling the incremental reconstruction problem. Besides, we believe the mentioned task is a promising research direction where a lot of details should be considered (while it is beyond the current scope of this paper). For example, whether the input views are temporally correlated, i.e., video frames, or whether the lighting conditions have been changed in incoming views. We thank you for the insightful suggestions and we plan to explore it further in the future.

[1] Yuan, Ze-Huan and Tong Lu. “Incremental 3D reconstruction using Bayesian learning.” Applied Intelligence 2012.

4. [Inference on different numbers of inputs] Our method is able to work on different numbers of inputs. As shown in Fig. 8 (bottom), LEAP is able to work with fewer images (2-4 views) when the model is trained with 5 views. We further evaluate the results using 7 and 10 images (as shown below), which reach the upper bound of the number of views in prior sparse-view research [1]. Under the dense view setting (>100 images as mentioned), the pose estimation accuracy will not be a problem, and, thus, we can perform pose-aware reconstruction, e.g. COLMAP. For extending LEAP to dense views, as you mentioned, it is necessary to propose hierarchical information aggregation techniques, as the aggregation from all input views can be computationally expensive. We note that working on dense views is beyond the current scope of LEAP and it's an interesting open research problem to unify sparse-view and dense-view reconstruction methods. We appreciate your insightful comments!

[1] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022.

We appreciate your valuable comments. We thank you for acknowledging the extensiveness of our experiment and the originality of our idea. We will address your comments as follows.

1. [Render input views during training/optimization] Different from the original NeRF-style works, where the model is trained and tested on the same instance, LEAP deals with modeling novel instances captured as sparse & unposed images at inference. It is trained and tested on different object/scene instances, and all test instances are unseen during training. As described in the third paragraph of page 2, we use ground-truth poses of input images of training scenes to render the predicted input views. Note that these ground-truth poses are only used during training for learning the 2D-3D information mapping, and using ground-truth pose during training is a common setting in prior generalizable NeRF works, e.g. PixelNeRF, SRT, and FORGE. Besides, we note that the ground-truth poses used during training are easily accessible, as they are provided in the existing datasets, e.g. OmniObject3D, Objaverse, and DTU dataset, annotated using virtual cameras for rendering or complicated camera calibration techniques. After LEAP is trained, it directly inferences on novel objects/scenes captured as unposed images with zero-shot generalization. We will better clarify the point in the revision.

2. [Improved speed] The fast inference speed of LEAP originates from its ability to predict the radiance field in a single feed-forward step during inference, without any test-time optimization (as we described in the first paragraph of page 2). In contrast, most prior works require some per-scene optimization at test time (SPARF, RelPose, and FORGE), which are slow to converge. We will clarify it in the revision.

3. [Scale prediction] We perform all experiments (both training and evaluation) with a normalized scale. This is a common setting in RGB-based reconstruction methods [1]. As scaling up/down the size of the scenes as well as the camera translations jointly will lead to the same RGB observations at these camera viewpoints, it requires depth information to resolve the scale ambiguity. It is an ill-posed problem to recover the scale when only RGB images are available (if without any category-level prior, e.g. human category average height). We will clarify it in the revision.

[1] Schönberger, Johannes L. and Jan-Michael Frahm. “Structure-from-Motion Revisited.” CVPR 2016.

4. [Local consistency] LEAP learns the local consistency between 2D-3D. As shown in Fig. 11 (a) and (b), the two neighbor on-surface 3D voxels are associated with consistent 2D regions. We generally agree with your point that incorporating more regularizations can improve the 2D-3D association accuracy.

3. [Increasing number of views] As shown in Fig. 8 (bottom), we provide the performance of LEAP compared with prior works using 2-5 input views, where all methods are trained with 5 images of each scene/object. LEAP demonstrates better performance than prior works when more images are available. Besides, we note that LEAP focuses on sparse-view reconstruction, where we usually have only 2-5 input views. To further address your concern, we experiment with using 7 and 10 input views, which reaches the upper bound of the number of views in prior sparse-view research [2]. As shown below, the performance of the best baseline FORGE drops with more than 7 views. We conjecture that the reason is the compounding pose estimation error. In contrast, LEAP continually demonstrates better performance with more inputs.

[2] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022.

Thanks for the positive comments, and your acknowledgment that our experiments and ablation studies are extensive. We will address your comments as follows.

1. [NVS with given camera pose] We use “relative” camera poses for both training and inference. The relative camera pose specifies the rotation and translation of a target view with respect to the canonical view (first image) [1, 2]. Thus, during testing, you can control the target NVS viewpoints by specifying small or large relative camera transformations. This definition of target NVS camera pose will not be influenced by the different definitions or absence of “absolute” camera pose of input views, as the reconstructed radiance field is defined in the local frame of canonical view. Furthermore, we follow prior works [3] to perform reconstruction with normalized scales, which eliminates the scale ambiguity of camera translations caused by RGB-only observations. We will include more details in the revision and we thank you for your helpful feedback. Additionally, we show 360-degree visualization in the supplementary by controlling the relative poses of target NVS views.

[1] Zhang, Jason Y. et al. “RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild.” ECCV 2022. [2] Jiang, Hanwen et al. “Few-View Object Reconstruction with Unknown Categories and Camera Poses.” 3DV 2024. [3] Schönberger, Johannes L. and Jan-Michael Frahm. “Structure-from-Motion Revisited.” CVPR 2016.

2. [Comparison with optical flow] We generally agree that more explicit use of fine-grained cross-view correspondences/matching can facilitate learning. However, the existing correspondence models are not suitable for finding correspondence in the sparse-view setting. For example, the mentioned optical flow methods typically work on nearby frames in a video, which observes slight visual appearance differences. In contrast, input images in the sparse-view setting usually have extremely large variations, caused by wide-baseline cameras. The current optical flow methods cannot work well in this setting [4]. Besides, one of our baseline methods, SPARF, is based on dense correspondence from off-the-shelf methods, which shows strong artifacts when correspondence is not accurate and suffers from slow convergence. Finally, we believe pixel-level dense correspondence and sparse-view reconstruction with feature-level correspondence can benefit from each other. Exploring their synergy to improve the performance of both, i.e., using a joint learning framework, can be a promising future direction. We appreciate your insightful comments.

[4] Chen, Qiao and Charalambos (Charis) Poullis. “Motion estimation for large displacements and deformations.” Scientific Reports 12 2022.

3. [Comparison with unposed NeRF] We clarify that this paper focuses on sparse-view reconstruction, where only 2-to-5 unordered images are available. In contrast, the current unposed NeRFs usually assume either dense-view (~50) or sequential (video) inputs [5,6]. This line of work demonstrates limited performance in the sparse-view setting, as they still estimate camera poses internally. We run experiments on the DTU dataset with Nope-NeRF [5] under the sparse-view setting, where its PSNR is 11.32. This performance gap (compared with their satisfying performance using dense-view inputs) verifies the necessity of this work. Besides, one of our baselines, FORGE, is an unposed & generalizable NeRF variant designed for sparse views, and LEAP demonstrates better performance compared with FORGE. We will add more discussion on this in the revision.

[5] Bian, Wenjing et al. “NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior.” CVPR 2023. [6] Fu, Yang et al. “MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Poses.” ICML 2023.