IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations

W-1. Experiments on Real Data

Thank you for highlighting this concern. In response, we have included results on two real-world datasets: MIT-Intrinsic [1] and Stanford-ORB. Additionally, we have added StableNormal and IntrinsicNeRF for comparison, as recommended by reviewer GJXm.

We also provide qualitative results in the supplementary material: Figure R1 of rebuttal.pdf (MIT-Intrinsic); video.mp4 (Stanford-ORB).

Additionally, we include results on the Internet real-world data (Figure R6 of rebuttal.pdf) and the NeRD-real dataset [2] (Figure R7 of rebuttal.pdf and video.mp4).

W-2. Clarity and Details in Method Section

Thanks for the constructive feedback and suggestions.

1. Modifications to Figure 2: We have update Figure 2 in response to the provided feedback:
   1. Clarification of D=3: We have explicitly mentioned in the figure caption that $D=3$, which corresponds to the number of intrinsic components.
   2. Color Contrast: We have adjusted the colors to provide stronger contrast, making it easier to distinguish between them.
   3. Attention Matrix: We use different colors to represent the two attention matrix, as they correspond to different attention maps. We have updated the illustration of attention block to enhance understanding.
   4. Image Latents and Noise: For the top half of Figure 2, we clarify that the image latents are concatenated with the noise, which is also explained in Line 246 of the main text, where we specify that each triplet latent is channel-concatenated with Gaussian noise for the denoising process.
2. Illumination-augmented training:
   1. In real-world phenomena, lighting conditions can vary significantly due to factors like time of day, weather, or even the age of images (e.g., images of the Statue of Liberty taken over several years). This results in in-the-wild images of the same object having varying lighting conditions. Previous works [2-4] have also explored reconstruction under multiple illuminations.
   2. By leveraging illumination-augmented training, we enhance our model's ability to handle such extreme lighting variations. This approach further allows the model to learn how to utilize photometric cues from different lighting conditions to infer intrinsic components. Figure R7 of rebuttal.pdf demonstrates how our model achieves stable predictions under varying lighting conditions.

W-3. Missing Citations

Thank you for pointing this out. We have added relevant works in Section 2.1 and 2.2 of the main paper.

[1] Grosse, Roger, et al. "Ground truth dataset and baseline evaluations for intrinsic image algorithms." 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009.

[2] Boss, Mark, et al. "Nerd: Neural reflectance decomposition from image collections." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Martin-Brualla, Ricardo, et al. "Nerf in the wild: Neural radiance fields for unconstrained photo collections." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[4] Jin, Haian, et al. "Tensoir: Tensorial inverse rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

W-1. Performance on Scenes

Thanks for the suggestion. We have evaluated our model on both indoor and outdoor scenes and present the results in Figure R3, R4, R5 of rebuttal.pdf in the supplementary material. Figure R3 and Figure R4 show predictions on outdoor and indoor scenes from the Mip-NeRF 360 dataset [1] respectively, while Figure R5 presents results on outdoor and indoor scenes derived from Internet images.

Despite not being explicitly trained on such datasets, our model demonstrates notable generalization ability in these domains. However, the intrinsic components of objects and scenes differ significantly in their distributions. While our model exhibits generalization to scene-level data, retraining on scene-specific datasets is necessary for achieving stable and superior performance.

W-2. Compare Pros/Cons against Unsupervised Methods

Unsupervised methods, such as DyNeRFactor and IntrinsicNeRF, often require dense multi-view inputs (e.g., DyNeRFactor needs a 10-second, 30 FPS video) and can be computationally expensive (e.g., IntrinsicNeRF requires 10 hours to optimize a single scene). In contrast, supervised methods, like ours, predict intrinsic components in a feed-forward manner, which is much faster—taking only seconds or minutes. However, supervised methods require a dataset with ground truth, which can be challenging to acquire for real-world data. When using synthetic data, these methods may face domain gaps when transferred to real-world.

Supervised methods can aid unsupervised approaches by providing a reliable prior. For example, DyNeRFactor leverages a pre-trained model on the MERL dataset as a material prior to ensure that the predicted intrinsic components are physically plausible.

W-3. Limited Comparison with Other Methods

Thanks for pointing this out. We have included quantitative results on two real-world datasets:

1. MIT-Intrinsic dataset [4]: We compare our method with IntrinsicAnything on albedo prediction.

2. Stanford-ORB dataset [2]: We compare our method with StableNormal on normal prediction and IntrinsicNeRF on albedo prediction.

Qualitative results on MIT-Intrinsic dataset is presented in Figure R1 of the rebuttal.pdf in the supplementary material. Qualitative results on Stanford-ORB dataset [2] and NeRD-real dataset [3] are in video.mp4 in the supplimentary material.

W-4. Additional Evaluation Results

Thanks for the valuable feedback.

1. We understand your concern regarding the use of synthetic data for comparison. To address this, we have included additional comparisons on real-world data (as discussed in W-3). Moreover, we provide results on scene-level data (Figure R3, R4, R5 of rebuttal.pdf in the supplementary material), demonstrating the strong generalization ability of our model and confirming that it is not overfitted to the training data.
2. Training on synthetic datasets with ground-truth supervision has been widely explored in this domain (e.g., [5-8]). Since obtaining ground-truth annotations for real-world data is challenging, some works resort to unsupervised methods. However, these approaches are often limited to specific object categories [9] or employ simplified image formation models [10] that lack physical correctness. Therefore, we render a large-scale synthetic dataset and leverage supervied training. Additionally, we argue that the domain gap between synthetic and real-world data is significantly alleviated through the use of a pre-trained diffusion model.

W-1. Limited novelty

While similar attention designs have been explored in other works such as GeoWizard, adapting such mechanisms to intrinsic decomposition is far from trivial. (1) While GeoWizard is limited to predicting single view image, our method supports arbitrary input views without requiring camera poses since our cross-view attention module effectively captures spatial relationships on its own. (2) Whereas GeoWizard employs a domain switcher to control generation across domains, we guide the process using specific text prompts via cross-attention modules. (3) Furthermore, we developed a large-scale, high-quality dataset tailored to the multi-view, varying-illumination setting. (4) Additionally, we devised a specialized training strategy to address both single-view and multi-view predictions along with a shifted noise scheduler tailored to better adapt to the material domain.

W-2. & Q-1. Reproducibility

We have made a part of the ARB-Objaverse dataset available at this anonymous link. We promise to release the complete dataset, along with the corresponding model and code, upon acceptance of this work.

W-3. & Q-2. Generalization to Other Domains

Thanks for pointing this out. We have evaluated our model on both indoor and outdoor scenes, with the results presented in Figure R3, R4, R5 of rebuttal.pdf in the supplementary material. Despite not being explicitly trained on such datasets, our model demonstrates notable generalization ability in these domains.

W-4. Clarifications on the Attention Block in Figure 2

Thank you for your valuable feedback! We have revised Figure 2 of the mian paper:

1. We added captions to explain what each image corresponds to in the cross-component attention and cross-view attention.
2. We refined the visualization of the attention modules to present the operations in a clearer and more intuitive manner.
3. We added arrows to represent the order of operations within the attention block: cross-component attention → cross-view attention → cross attention → feed-forward.

W-5. Consistency in Figure 7

Thank you for your suggestion. We have updated Figure 7 of the main paper.

W-6. Missing Citations

Thank you for pointing this out. We have added relevant works such as NeRD, NeROIC, and Neural-PIL in Section 2.1 of the main paper (Page 2, Line 105).

I thank the authors' effort in answering my issues and revising the paper, especially showing a demo of the datasets. I have no more questions, and am happy to raise my score. I'm looking forward to seeing the full release of the datasets in the future.

W-1. Limited novelty

Adapting such attention mechanisms to intrinsic decomposition task is not trivial. (1) While previous works are limited to generating or predicting predefined views, our method supports arbitrary input views without requiring camera poses since our cross-view attention module effectively captures spatial relationships on its own. (2) Whereas Wonder3D and GeoWizard employ a domain switcher to control generation across domains, we guide the process using specific text prompts via cross-attention modules. (3) Furthermore, we developed a large-scale, high-quality dataset tailored to the multi-view, varying-illumination setting. (4) Additionally, we devised a specialized training strategy to address both single-view and multi-view predictions along with a shifted noise scheduler tailored to better adapt to the material domain.

W-2. & Q-1. Limited evaluation samples and Re-rendering results

Thank you for your valuable suggestions! We have included more challenging samples and their corresponding re-rendering results in rebuttal.pdf and video.mp4 in the supplimentary material.

1. High specularity cases

We include high-specularity examples such as a motorcycle, model car, trumpet, and knife on a table in Figure R6 of rebuttal.pdf. Our model effectively removes highlights and accurately identifies metallic parts, demonstrating its robustness in handling reflective surfaces.

2. Multi-view images with extreme lighting variation

We evaluated our model on three scenes from the NeRD dataset [1], captured under varying illumination at different times of day. Results for the StatueOfLiberty and MotherChild scenes are provided in Figure R7 of rebuttal.pdf, while results for the Gnome scene are included in video.mp4. Our model consistently produces plausible predictions, even under these challenging lighting conditions.

3. Scenes with few views

We tested our model with varying numbers of input views:

• Single View Input: Results on real-world data are shown in Figure R1 and Figure R6.
• Four Views Input: Results are presented in Figures R3, R4, and R7.
• Dense Input Views (25 views): Results are showcased in video.mp4.

Additionally, numerical results for 1, 2, 4, and 8 input views are provided in Figure 7 and Tables 4–7 of the main paper.

4. Re-rendering results

We present the Re-rendering results along with relighting results of challenging cases in Figure R6 of rebuttal.pdf.

W-3. Evaluation on more datasets

Thank you for your suggestion. We present qualitative results for both indoor and outdoor scenes in Figure R3, R4, R5 of rebuttal.pdf. Despite not being explicitly trained on such datasets, our model demonstrates notable generalization ability in these domains.

However, we argue that the intrinsic components of objects and scenes differ significantly in their distributions. While our model exhibits generalization to scene-level data, retraining on scene-specific datasets is necessary for achieving stable and superior performance. Similarly, models trained on scene datasets often experience performance degradation when applied to object-centric scenarios, as illustrated by the results of IntrinsicDiffusion in Figure 4 of the main paper.

Q-2. Failure Cases

Thanks for pointing this out. We provide several failure cases in Figure R2 of rebuttal.pdf.

• Outdoor Scenes (First row). Our model struggles with outdoor scenes, as it is primarily trained on object-centric data. While the model exhibits some generalization capability, its performance degrades in these scenarios.
• Textual Structures (Second Row). When the model is faced with text, the decomposition fails to recover the correct text structures.
• Over-Simplified (Third Row). In some cases, the model produces overly smooth outputs, failing to preserve subtle material details, such as the metallic features of a telephone. This issue arises from the synthetic training data, which often contains simpler material variations, leading the model to overly simplify fine-grained material properties.

[1] Boss, Mark, et al. "Nerd: Neural reflectance decomposition from image collections." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

W-1. Discussion about Wonder3D

Thank you for pointing this out. This is a very valuable suggestion. Our initial inspiration came from GeoWizard, and we inadvertently overlooked that cross-domain design in GeoWizard originates from Wonder3D. We sincerely apologize for this oversight and have added the citation (Section 2.3, Page 3, Line 144) and the discussion (Section 3, Page 5, Line259) in the main paper.

Although our attention block shares similarities with Wonder3D, adapting this model design to intrinsic decomposition tasks is far from trivial. (1) Unlike Wonder3D, which is limited to generating six predefined views, our method supports arbitrary input views without requiring camera poses since our cross-view attention module effectively captures spatial relationships on its own. (2) Whereas Wonder3D employs a domain switcher to control generation across domains, we guide the process using specific text prompts via cross-attention modules. (3) Furthermore, we developed a large-scale, high-quality dataset tailored to the multi-view, varying-illumination setting. (4) Additionally, we devised a specialized training strategy to address both single-view and multi-view predictions along with a shifted noise scheduler tailored to better adapt to the material domain.

W-2. Novelty of Dataset

While the technique of rendering object is indeed widely used, no existing dataset provides both diverse lighting renderings and their corresponding intrinsic components. Our dataset addresses this gap by offering 5,700,000 images of RGB rendering, albedo, normal, and material maps, generated at the cost of 1,900 GPU hours.

We have uploaded part of our ARB-Objaverse dataset at this anonymous link and will release the full dataset upon acceptance. As acknowledged by reviewers cXk1, GJXm and PUK6, we believe this contribution will be a valuable resource for the community, facilitating research in the field of inverse rendering.

W-3. Additional Examples in the Supplementary Video

Thank you for bringing this up. We have updated the supplementary video to include more comparisons on the Stanford-ORB[1] and NeRD-real[2] datasets. Additionally, we provide quantitative comparisons on MIT-Intrinsic[3] and Stanford-ORB[1] dataset.

Q-1. Elaboration on 3D Reconstruction

Thank you for your suggestion. We have revised this section in the main paper (Page 10, Line 514) to provide additional details. Rather than directly reconstructing the 3D shape from the predicted intrinsic maps, we use the predicted intrinsic components as material priors to enhance existing 3D reconstruction methods. Specifically, we select NVDiffRecMC [6] to validate the effectiveness of our priors. During each iteration, we introduce an additional L2 loss between the intrinsic components reconstructed by NVDiffRecMC and those predicted by our method. This significantly alleviates color-shifting issues in albedo reconstruction and ensures the physical plausibility of the reconstructed materials.

Q-2. Explanation of the Cross-View, Cross-Component Attention Mechanism

Thanks for the suggestion. We have updated Figure 2: Overview of IDVI and its caption (Page 4, Line 182) to improve clarity.

For cross-component attention, we extend the original self-attention module to operate across different components. Specifically, the input tensor is reshaped from $[(B,V,D),H,W,C]$ to $[(B,V),(D,HW),C]$ before attention calculation, where $B$, $V$ and $D$ represent batch size, number of views and number of intrinsic components.

For cross-view attention, the input is reshaped from $[(B,V,D),H,W,C]$ to $[(B,D),(V,HW),C]$ before applying self-attention.