PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction

We thank the reviewers for the valuable feedback. We will include the missing references and fix typos in a revision. We address your major concerns below.

Evaluating RelPose++ and Forge: we used the pretrained checkpoints from RelPose++ and Forge to evaluate their cross-dataset generalization. We would like to emphasize that we focus on cross-dataset generalization in this work as it is vital for practical applications; our model was not trained on any of the evaluation datasets either. Moreover, we would like to highlight that we also reconstruct NeRF while RelPose++ doesn’t. This joint prediction of 3D and pose is very helpful for boosting pose prediction accuracy in our experiments, as shown in Tab. 4.

Missing Scene representation transformer (SRT) baseline: In our humble opinion, we can not consider SRT as a fair baseline to be compared, since there’s a clear distinction in objectives and scopes between our work and SRT. PF-LRM targets 3D reconstruction and pose estimation, while SRT only focuses on novel view synthesis. It means that SRT does not estimate poses for input images which is one of the main goals of this paper and densely measured. Poses are important for practical applications like robotic navigation, etc. Also, geometry can be naturally extracted from our reconstructed NeRF, while SRT is a geometry-free method that does not directly predict geometry. However, we thank the reviewer for bringing this work to our attention, we will cite SRT and discuss it in a revision.

Input-view PSNR: We will include novel-view psnr in a revised draft before the rebuttal deadline. The current input-view psnr is to show that our predictions (both NeRF and poses) are well-aligned with the input images, as we render our predicted NeRF using our predicted poses (in Fig. 3); we would like to highlight that we do well in this important objective of matching re-renderings with inputs for joint 3D and pose prediction, while baselines struggle with this goal. We also show extracted geometry in Fig. 3, 4 and interactive NeRF viewers on our project page to demonstrate that we did not cheat by predicting degenerate solutions. In Fig. 4, we show re-rendering of our reconstructed NeRF using ground-truth pose, to demonstrate that removing pose prediction from our model leads to lower reconstruction quality. This is to highlight the synergy between NeRF reconstruction and pose estimation, which was not shown in prior work like SRT.

Comparison with RelPose++ on Co3D: as shown in Tab. 8 of the appendix, we outperform RelPose++ on the majority of the Co3D categories, even though our model is not trained on Co3D while RelPose++ was trained on it. As to why we are worse on certain categories, we attribute this to the noisy in-accurate masks provided by Co3D, which we used to remove background before running our method. A clear performance drop can also be seen in Tab. 8 when we test RelPose++ on such background-free images. We think handling background in our framework is an interesting future direction.

Move Co3D results to the main paper: We think it’s a good idea to move Co3D results to the main paper. By the time of submission, we did not have time and resources to train our model to full convergence on a mix of Objavese and MvImgNet data; hence we put the non-convergent model’s results in the appendix. We will include the results of our convergent model in the main paper in a revision.

Missing real-data results in the paper: We respectfully disagree with this claim. We have shown our model’s results on the real DTU dataset (Fig 5 and Tab 9 in appendix), OmniObject3D dataset (Tab 1), Co3D dataset (Tab 8 in appendix). Please refer to our main paper and appendix for details.

Dear Authors,

thank you for the kind and detailed reply.

Comparison to RelPose

I do not think that your argument for evaluating with the pre-trained checkpoints for RelPose is valid. If I understand correctly, you are basically saying "we are evaluating cross-dataset generalization, therefore, it is reasonable to evaluate the pre-trained checkpoint for RelPose.". I don't think that is a valid argument at all. You are claiming that your method outperforms RelPose on pose prediction, as you literally say in your abstract:

"When trained on a huge amount of multiview data, PF-LRM shows strong cross-dataset generalization ability, and outperforms baseline methods by a large margin in terms of pose prediction accuracy and 3D reconstruction quality on various evaluation datasets".

If you want to claim that you outperform either of these methods on pose estimation accuracy, then you have two options: 1) you train your method on the same dataset as RelPose and then show that you are doing better or 2) you are training RelPose on the same dataset that you used for training and then show that you do better on the corresponding test set.

Otherwise, you have not supported your claim. In other words, you would be putting out a paper that might lead many scientists down the wrong path: You are suggesting that anyone who is doing pose estimation ought to use your method, when in practice, it might well be that RelPose would in fact outperform your method when trained on the same dataset.

IMO, it is standard procedure in our field to train the baseline on the same dataset as the proposed method to allow for a fair comparison.

Other points

Thank you for clarifying the other points, they clarified my other questions!

We sincerely understand your concerns of fair comparison with baselines. We spent the rebuttal period scaling up the training of the baseline RelPose++ on the Objaverse dataset, and report the performance of the re-trained baseline in the following table. We can see that our method (trained on exactly the same data) still outperforms the retrained baseline by a wide margin, demonstrating the superiority of our proposed method. We also attempted to scale up the training of baseline FORGE, but its complex multi-stage training (6 stages) seems non-trivial to scale up in our trials. We leave it to future work to investigate the scalability issue of FORGE.

---

OmniObject3D

---

GSO

---

ABO

---

Updated draft: we have updated (in-place) the abstract, introduction, related work, conclusion sections. We’ve mentioned in the updated abstract that we used multi-view posed data for training. We’ve cited SRT paper and discussed the scope difference between our work and SRT. We also added citations to the Convolutional Occupancy Networks paper and FlowCam paper, as the reviewer suggested. For the experiment section, we did not modify it in-place because our modifications will change the figure/table numbers, hence increasing the reading difficulty for the reviewers. Instead, we put the updated experiment section in Appendix B. We plan to integrate it upon acceptance. We are also happy to integrate it now, if the reviewers don’t think this increases reading burden. You can find our novel-view PSNR in Tab. 14 and our novel-view renderings (along with ground-truth) in Fig. 9. We also include a detailed limitation section in Appendix C.

We thank the reviewer for the valuable feedback. We will add the suggested references ([a],[b]) in a revised draft soon before the rebuttal deadline. Please find our answers to your questions below.

Input-view PSNR: We will include novel-view psnr in a revised draft before the rebuttal deadline. The current input-view PSNR is to show that our predictions (both NeRF and poses) are well-aligned with the input images, as we render our predicted NeRF using our predicted poses; we would like to highlight that we do well in this important objective of matching re-renderings with inputs for joint 3D and pose prediction, while baselines struggle with this goal. We also show extracted geometry in Fig. 3, 4 and interactive NeRF viewers on our project page to demonstrate that we did not cheat by predicting degenerate solutions.

We thank the reviewer for the positive feedback about our empirical evaluation, technical soundness, and potential impact on the 3D generative AI. We address your questions below.

View selection: For the evaluation on ABO and GSO dataset, we randomly select 4 sparse views such that viewing angles between any two views are at least 45 degrees to ensure sparsity. This is to test our method’s test-time performance on extreme inputs. But our method can work on different viewpoint settings, as during training, we did not enforce any restriction in selecting input viewpoints. The bunny and frog in Fig. 1 shows that our method can work well even when the two input views have little overlap. Our model can also handle the easier case where viewpoint changes are small.

We thank the reviewer for the valuable feedback. Please see our answers to your questions below.

Limited pipeline novelty: We respectfully disagree with this point. We would like to reiterate the contributions of our work: first, we propose a novel single-stream transformer architecture for joint NeRF and pose prediction, which is different from the encoder-decoder one used in the LRM, Instant3D work; the single-stream architecture not only allows the NeRF tokens to be contextualized by image tokens for reconstruction purpose, but also allows the image tokens to be contextualized by NeRF tokens for predicting per-view coarse geometry used for solving poses. Second, as far as we know, we are the first to apply the differentiable PnP solver to the task of jointly predicting 3D and pose from unposed sparse views, though the solver itself is inspired by the EPro-PNP work. This is in stark contrast to most prior work, e.g., RelPose++, Forge, that uses direct pose regression from images. Meanwhile, unlike the 6DoF pose estimation and 3D object detection tasks shown in EPro-PNP, we don’t have access to ground-truth 3D points/models during training (we only use multi-view posed images as training data); hence we propose the novel solution of supervising the sparse point predictions by distilling the predicted NeRF geometry in an online manner.

Overfitting concern: As shown by Fig. 1, we have tested our method on the generated multi-views from Zero1-to-3, MVDream, SyncDreamer etc. These images are unlikely to be in the Objaverse dataset, as they are generated by 2D diffusion models, but our method still works well on them, showcasing our model’s strong cross-dataset generalizability. Moreover, we’ve also demonstrated our method’s generalization to the real scanned datasets OmniObjects3D, as shown in Fig. 3 and Tab. 1. This dataset was published later than Objaverse, making it unlikely to be covered by Objaverse.

Scaling laws: we have shown scaling laws for model sizes in Table 1,2,3 and 4, where bigger models clearly outperform smaller models. Due to limited compute resources, we do not show scaling laws for data sizes using our largest model (as the scaling law of data is not insightful under smaller model and with inadequate compute [1]); but we’re happy to include one in the final draft. [1] Scaling Laws for Neural Language Models

Comparison with sparse NeRF methods: In this work, we focus on the challenging case of sparse unposed images as inputs, for the practical concern of estimating poses from sparse unposed images is a very challenging task itself. Hence, most sparse NeRF methods (e.g., [1,2,3] as the reviewer pointed out) requiring poses cannot work on such unposed inputs, especially when overlapping between input views is minimized (e.g., the frog and bunny in our Fig. 1).