PRM: Photometric Stereo based Large Reconstruction Model

$**Question1**$

One key motivation of this paper is unconvincing.

$**Answer1**$

We would like to clarify that our key motivation is not online rendering. Our primary objective is to demonstrate that utilizing photometric stereo images rendered under varying materials, lighting conditions, and camera poses can lead to significant improvements in existing LRMs. These improvements include enhancing detailed local geometry and increasing robustness to variations in image appearance.

The split-sum approximation serves two purposes: first, as a tool for enabling online rendering, offering greater flexibility for ground-truth rendering; and second, as a predictive rendering method. In this context, split-sum approximation is essentially a PBR technique that excels at modeling the specular component, which facilitates improved reconstruction performance on glossy surfaces, as shown in prior per-scene optimization methods such as NeRO [1]. We have also conducted an ablation study to validate the effectiveness of using PBR instead of directly predicting RGB, as shown in Table 5 under the setting "w/o PBR."

We validate this motivation through extensive experiments, including both qualitative and quantitative evaluations. We have further added more results in the revised PDF. We believe the results are convincing enough to support our motivation.

[1] L iu Y, Wang P, Lin C, et al. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images[J]. ACM Transactions on Graphics (TOG), 2023, 42(4): 1-22.

$**Question2**$

Comparison with some stronger baselines, such as Mesh-LRM.

$**Answer2**$

We have added a comparison with Mesh-LRM using their released online demo. However, since the demo does not provide a download interface, we included a qualitative comparison in Figure 20 of the revised manuscript. Our method still demonstrates superior performance.

It is also worth noting that Mesh-LRM acknowledges certain limitations in their work in the ArXiv version, stating: ''Since our model employs surface rendering and assumes only Lambertian appearance without performing any inverse rendering, it is not sufficiently robust when the input images contain complex material'' and ''However, this requires sufficient training data with accurate material annotations.''

We believe our proposed approach addresses these limitations through the use of photometric stereo images for both input and supervision, as well as differential PBR for predictive rendering. We hope the reviewer can recognize and appreciate the efforts we have made to overcome these challenges.

$**Question3**$

The ablation studies are incomplete.

$**Answer3**$

Since each trial incurs a high computational cost, it is infeasible for us to include all training objects in the ablation study. Instead, we used a subset of 10k training objects for the study. The quantitative results have been added to Table 3 in the revised manuscript.

Dear Reviewer FrKV:

We sincerely appreciate the time and effort you have dedicated to reviewing our paper and for providing such thoughtful feedback! As the discussion period concludes in two days, we kindly ask if our rebuttal has addressed your concerns to your satisfaction. If there are any remaining points requiring clarification or further improvement, please do not hesitate to let us know—we are fully committed to addressing them promptly.

Thank you once again for your invaluable contributions to improving our work!

Warm regards, The Authors

Thanks for your response.

To begin with, I hope the authors can clarify the question I asked in the previous reply "Are both the predicted results and gt results rendered with split-sum approximation? Or are only the training data rendered with a split-sum approximation?"

Second, I don't think using photometric stereo images to improve the performance of LRMs is a significant contribution. It is pretty straightforward. Everyone knows that training data of higher rendering quality can help improve the model's performance. The contributions of data augmentation will come from how to do data augmentation, i.e. how to improve the data quality. This paper chooses to use split-sum approximation to render the photometric stereo data online as the training data, which makes using split-sum approximation to render the data online one of the key technical contributions of this paper. This is why I am questioning the necessity of online rendering, which takes a large portion of this paper's technical method description. If rendering training data online is not truly necessary, rendering photometric stereo images offline with a longer rendering time but higher rendering quality will be better for the final quality.

Third, as I said, the model will only be trained with finite steps, so training data distribution has no difference when you do the online rendering or predefined offline rendering because you can always use offline rendering to render the same data created by online rendering, as long as using the same sampling strategy. You mentioned that DINO-V2 also uses online data augmentation. It's true that online data augmentation is common in 2D vision tasks, but this is because it is very efficient to do data augmentation for data like 2D image data or text, and storing the augmented 2D images will be too costly considering the large amount of training 2D training data. However, things are very different in 3D fields and most 3D papers try to avoid rendering the training data online. The reasons include but are not limited to (1). Loading 3D objects online and rendering are both more costly than just loading the 2D rendered images. Even though you can make the rendering time to be approximately 0 seconds. Loading 3D objects online will still be very high IO demanding. (2). Most 3D reconstruction models will only train the model for about 100k iterations, so the total data storage won't be very high when storing the offline rendered data.

Fourth, NeRO uses split-sum approximation to render the predicted results only. I can understand using the split-sum approximation to render the predicted rendering results during optimization. However, my question is the necessity to render the training data online for 3D reconstruction tasks.

For the first question, the answer is yes—both the predicted and ground-truth results are rendered using the split-sum approximation. We have already addressed this in our previous two responses.

In the first response, we stated: "The split-sum approximation serves two purposes: first, as a tool for enabling online rendering, offering greater flexibility for ground-truth rendering; and second, as a predictive rendering method." In the second response, we further clarified: "It also plays a crucial role in specular surface reconstruction as a form of predictive rendering."

I hope the reviewer can review this information more carefully instead of merely repeating the same question. Reiterating the same concerns without engaging with the provided clarifications is not constructive and does not contribute to improving our manuscript. I kindly request that the Area Chair pay closer attention to this point.

Second, based on the reviewer’s feedback, it seems that the reviewer may not have a thorough understanding of photometric stereo. The reviewer stated, "Everyone knows that training data of higher rendering quality can help improve the model's performance." However, in reality, the split-sum approximation, as a real-time rendering method, simplifies the rendering equation and may degrade rendering quality compared to more accurate methods like Blender rendering. Despite this, our method still delivers significant improvements. Have you considered the reasons behind this?

In our work, we aim to explore what constitutes good data for LRM. The key factor is not the rendering quality of each image, as mentioned by the reviewer. We found that photometric stereo images are highly effective data for LRMs, and extensive experiments validate this finding, even though the quality of each individual rendered image may be slightly lower. We believe this insight will advance the development of the field and contribute new perspectives.

I think the reviewer has completely missed the point, which is not only unhelpful but also detrimental to advancing the research. The reviewer seems fixated on the quality of individual images. However, the choice between Blender and split-sum approximation for ground-truth rendering is irrelevant. Our method can easily work with offline-rendered photometric stereo images as well. But rendering these images offline with numerous combinations is computationally prohibitive. For example, rendering 150k samples with 60 views per case using an 8-GPU A800 cluster took nearly two months. Additionally, since each sample is trained multiple times in a batch, offline pre-rendering with Blender would multiply the rendering time even further, making it several times less efficient. In contrast, our online rendering approach dramatically improves efficiency by generating data dynamically during training, which is far more practical and scalable.

Regarding your concern about I/O overhead when loading 3D objects during online rendering, we have proactively addressed this issue by preprocessing the data, which significantly accelerates the loading process and effectively mitigates the I/O bottleneck. Furthermore, we plan to open-source all our code, giving you the opportunity to directly test our approach and verify the efficiency gains for yourself.

We discuss the split-sum approximation extensively in our manuscript, as it not only enables online rendering, but also enables predictive rendering. Unlike previous LRMs that directly predict the final RGB color for rendering, we use the split-sum approximation for predictive rendering, which effectively models the specular component, thereby improving the reconstruction performance of specular surfaces. If the process is not clearly illustrated, other readers may not fully understand the rendering method. For instance, reviewer UZVs raised a question about the split-sum approximation and requested further clarification.

Overall, we do not understand why the reviewer seems to disregard these other important contributions of our work.

And there is one point I would like to raise: How did you know that the Gradio demo was released by the first author of MeshLRM, as it is explicitly marked as an unofficial demo? I believe and hope that the review process for top conference ICLR can remain fair and unbiased, free from any malicious competition among peers.

For the second question, since the Gradio demo does not provide a mesh download interface, we are unable to offer a quantitative comparison. Additionally, given that the discussion period is approaching, we are unable to add new figures to our manuscript at this point.

We recommend referring to Figure 6 and Figure 7 of the MeshLRM arXiv version for comparisons on diffuse objects. You can download our results in video. The surface normals of MeshLRM clearly exhibit slight concavity, while our method achieves superior single-view reconstruction results.

(Click the "video" link to view the results in the anonymous version.)

$**Question1**$

Limited discussion of computational overhead compared to simpler approaches.

$**Answer1**$

Our network mirrors the structure of InstantMesh [1], and the computational overhead is similarly efficient. The online split-sum approximation approach takes only 0.008 seconds to render a single image, ensuring it does not introduce any additional computational burden.

[1] Xu J, Cheng W, Gao Y, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models[J]. arXiv preprint arXiv:2404.07191, 2024.

$**Question2**$

The 50% probability threshold for material/lighting consistency seems arbitrary.

$**Answer2**$

Due to the high training cost for LRM, we did not carefully fine-tune the probabilities. The motivation for setting the threshold is that, in most cases, we input images captured under the same material and lighting conditions for reconstruction. Therefore, it is essential for the model to effectively handle this scenario as well.

$**Question3**$

Results appear sensitive to multi-view diffusion model quality.

$**Answer3**$

Yes, this is a common issue for 3D generation models, which highlights the importance of training a robust multi-view diffusion model. However, it is important to note that this limitation is not specific to our proposed method. Our work focuses on improving the reconstruction quality of LRM, given the generated multi-view images or real-captured sparse multi-view images.

$**Question4**$

Some failure cases (e.g., with lacking depth information) could be analyzed more thoroughly.

$**Answer4**$

The lack of depth information can lead to failures in generating multi-view images using a pretrained multi-view diffusion model. However, it is important to clarify that this is not a limitation of our model. Our work focuses on improving the reconstruction quality of LRM, given either generated multi-view images from multi-view images or captured sparse multi-view images. That said, you present an intriguing idea.

We tested an advanced single-image depth estimation model [1], and the results are shown in Figure 16 in the revised PDF. If correct depth information is incorporated into the multi-view diffusion model, it could potentially enhance the accuracy of the generated multi-view images. We have discussed this potential solution in our manuscript. Thank you for your suggestion. I believe you have raised a meaningful question for improving existing multi-view diffusion models.

[1] Yang L, Kang B, Huang Z, et al. Depth Anything V2[J]. arXiv preprint arXiv:2406.09414, 2024.

$**Question4**$

How does the method perform on real-world images with unknown lighting conditions?

$**Answer4**$

Our method is robust to variations in materials and lighting, as it utilizes photometric stereo images for both input and supervision. Figure 13 illustrates several examples. The object in the second row is from DTU [1], and the objects in the last four rows are from NeRO [2]; all objects are real captures under unknown lighting conditions. Please note that ground-truth lighting was only used during training. However, during inference, our method requires no additional information beyond the RGB images.

[1] Jensen R, Dahl A, Vogiatzis G, et al. Large scale multi-view stereopsis evaluation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 406-413.

[2] L iu Y, Wang P, Lin C, et al. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images[J]. ACM Transactions on Graphics (TOG), 2023, 42(4): 1-22.

Dear Reviewer noz1:

We sincerely appreciate the time and effort you have dedicated to reviewing our paper and for providing such thoughtful feedback! As the discussion period concludes in two days, we kindly ask if our rebuttal has addressed your concerns to your satisfaction. If there are any remaining points requiring clarification or further improvement, please do not hesitate to let us know—we are fully committed to addressing them promptly. Thank you once again for your invaluable contributions to improving our work!

Warm regards, The Authors

$**Question1**$

Details and concerns about split-sum approximation.

$**Answer1**$

The split-sum approximation is a widely used method in real-time rendering, as introduced in [1]. This approach is also extensively applied in per-scene optimization methods [2, 3]. In our manuscript, we adopt this method for photometric stereo image rendering due to its efficiency and fast computation speed. Specifically, this method accelerates rendering by approximating the integral in PBR. We provide some rendered examples in Figure 9, demonstrating photorealistic rendering quality. Notably, this method is a general-purpose real-time rendering technique that does not rely on any special assumptions or requirements. For further details, we recommend referring to [1]. Due to the complexity and length of the formula derivation, we are unable to include all the details in our manuscript. But do not worry about this issue, all of our code will be released.

[1] Karis B, Games E. Real shading in unreal engine 4[J]. Proc. Physically Based Shading Theory Practice, 2013, 4(3): 1. [2] Munkberg J, Hasselgren J, Shen T, et al. Extracting triangular 3d models, materials, and lighting from images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8280-8290. [3] Liu Y, Wang P, Lin C, et al. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images[J]. ACM Transactions on Graphics (TOG), 2023, 42(4): 1-22.

$**Question2**$

How are the hyperparameters tuned?

$**Answer2**$

As you mentioned, training LRM involves high computational costs, making it impractical to carefully fine-tune the hyperparameters. For the weights of the losses, we follow the approach used in InstantMesh. For the newly added losses, we adjust their weights based on the magnitude of the loss values to maintain balance.

$**Question3**$

what happens on scenes with real, non-trivial backgrounds?

$**Answer3**$

Most existing LRMs assume white background images as input, which can be easily obtained using Segment Anything (SAM) [1]. We experimented with multi-view images that include natural backgrounds as input, and the results are shown in Figure 19 in the revised PDF. Interestingly, the background is also reconstructed. While our model is not designed to handle images with backgrounds, your suggestion presents an intriguing idea. If input images with backgrounds were included during training, it is possible that the model could learn to handle such scenarios as well.

[1] Kirillov A, Mintun E, Ravi N, et al. Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4015-4026.

$**Question6**$

L015: for what purposes?

$**Answer6**$

We mentioned this in the last sentence: "Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision." The purpose here is to highlight the use of images as both input and supervision.

$**Question7**$

Need to introduce the full name of PBR before first use of the abbreviation.

$**Answer7**$

We have modified this in the revised version.

$**Question8**$

Need to clarify 'dependence on images rendered under fixed and simple lighting conditions' of previous methods.

$**Answer8**$

Most previous methods render images using only fixed, pre-set area or point lighting (simple lighting), with consistent lighting across all rendered images for each object. Examples of such methods include OpenLRM [1], SyncDreamer [2], and InstantMesh [3]. Notably, OpenLRM and SyncDreamer have provided their rendering scripts in their GitHub repositories for checking. In contrast, we render photometric images for both input and supervision. This means that each 3D model are rendered under varying lighting conditions for each view. Such images enhance the precision of local details by providing rich photometric cues and increase the model’s robustness to variations in the appearance of input images. This is also the key motivation of our work.

[1] He, Zexin, and Tengfei Wang. "Openlrm: Open-source large reconstruction models." 2023, [2] Liu Y, Lin C, Zeng Z, et al. Syncdreamer: Generating multiview-consistent images from a single-view image[J]. arXiv preprint arXiv:2309.03453, 2023. [3] Xu J, Cheng W, Gao Y, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models[J]. arXiv preprint arXiv:2404.07191, 2024.

$**Question9**$

what is 'a richer set of equations'?

$**Answer9**$

In traditional photometric stereo methods, each lighting condition corresponds to a specific rendering equation, as shown in Eq. (6). Observing the object under multiple lighting conditions provides several such equations, which imposes stronger constraints for surface normal recovery.

$**Question10**$

Additional evaluation results on images of complex lighting and materials.

$**Answer10**$

In Figure 9 (original version), we present rendered examples generated by our framework, which are not comparisons under complex lighting and materials.

In fact, we show such cases in Figure 13. In the revised version, we have added comparisons on these objects with InstantMesh, the state-of-the-art baseline. Additionally, as noted in Table 1 and Table 2, we have rendered images under complex lighting for qualitative comparisons, as described in Line 367. Moreover, we have included qualitative results on selected objects including both synthetic and real-captured objects with spatially varying materials, as shown in Figure 18.

$**Question1**$

What does the model estimate w.r.t. the PBR parameters? Does it only estimate albedo?

$**Answer1**$

Yes, we apply an albedo MLP for albedo prediction using the interpolated features from the triplane, similar to how previous LMRs perform RGB prediction. This is mentioned in Lines 308 and 733 of the manuscript. Predicting albedo instead of RGB is more reasonable because the interpolated triplane features represent positional information without accounting for view-dependent appearance. Albedo is a global, inherent, and view-independent property, making it a more suitable choice in this context. We have added a figure to illustrate the detailed network architecture in Figure 9 in the revised PDF.

$**Question2**$

Do we apply the metallic and roughness globally?

what happens if the original CAD model is already associated with spatially-varying BRDF maps?

$**Answer2**$

Yes, we apply metallic and roughness properties globally. This approach stems from the challenge of assigning spatially-varying BRDFs, as we lack precise information on which parts of the object should be assigned specific BRDF values. For 3D models with spatially-varying BRDF maps, we replace these maps with our global values.

$**Question3**$

Does this strategy diminish the model's generation ability towards real images with complex SV materials?

$**Answer3**$

Through experiments, we found that this strategy does not compromise the model's ability to generate objects with complex spatially-varying materials. We provide several examples in Figure 18 in the revised PDF, showcasing results on both synthetic and real-captured images.

$**Question4**$

Does it benefit the training to also predict ground truth BRDF without manually sampling and enforcing global roughness and metallic?

$**Answer4**$

Most objects in Objaverse are characterized by low metallicity and high roughness, which causes the diffuse term to dominate their appearance. If we do not adjust the metallic and roughness properties, specular surfaces would fall outside the domain of our model. We conducted an ablation study, as shown in Figure 8 to illustrate the importance of varying metallic and roughness.

Predicting ground truth BRDF properties, such as roughness and metallic values, falls within the domain of inverse rendering. This task is typically more challenging, as demonstrated by previous per-scene optimization methods [1, 2]. Incorporating these additional predictions would potentially increase the complexity of optimization, as a single triplane would need to simultaneously account for geometry, albedo, metallic, and roughness. Previous works [3] has proven that disentangled triplane can better model the textured 3D model.

In this work, our primary focus is on improving the geometry reconstruction performance of LRMs. However, we are considering extending this research to develop an LRM framework for inverse rendering in the future. Achieving this would require a detailed understanding of each part of the object to assign appropriate materials to different regions. We sincerely thank you for inspiring us.

[1] Zhang Y, Sun J, He X, et al. Modeling indirect illumination for inverse rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 18643-18652. [2] Jin H, Liu I, Xu P, et al. Tensoir: Tensorial inverse rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 165-174. [3] Liu Q, Zhang Y, Bai S, et al. DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6881-6891.

$**Question5**$

Is the split-sum approximation only applied to synthesizing estimated image from estimated representations, or it is also used to render ground truth images? Are ground truth images rendered on-the-fly for each batch in training?

$**Answer5**$

Yes, we also used the split-sum approximation for ground truth image rendering during training in an on-the-fly manner. Using Blender for ground truth rendering would be computationally prohibitive. For each 3D model, there are infinite possible camera poses, 121 material combinations, and 600 environment maps. Rendering all these combinations with Blender would take several months. We depicted this in Line 80- Line 82.

Thanks to the authors for the response. There are a few further clarifications and questions that I would like to raise:

Follow up to Question 1: just to confirm: does the model estimate roughness and metallic? In other words, are the estimated diffuse and specular color maps rendered with GT roughness and metallic, or estimated ones? The revised Fig. 9 shows only albedo is estimated. What are the obstacles to also estimated roughness and metallic, and go a step further to estimated SV roughness and metallic?

Acknowledging the response from the authors, SV metallic and roughness are not estimated, and as a result the model only produce a 3D mesh with diffuse textures only? In this the case, it will inherently limit applications such as relighting where the original roughness and metallic are not captured, as a result specularities cannot be reproduced. It would help clarify the contributions by explicitly stating that the method is only focused on geometry and albedo, and adding additional discussion on this limitation and how the model can possibly be extended for full inverse rendering.