IllumiNeRF: 3D Relighting Without Inverse Rendering

About efficiency

We acknowledge that our approach currently prioritizes quality over real-time performance, as generating new samples with the RDM and optimizing a NeRF for each new lighting condition is computationally intensive (L252-254).

A potential solution to mitigate this limitation is to adopt an amortization strategy, similar to [a]. Specifically, we could train a single latent-NeRF conditioned on both latent codes and a latent representation of the given lighting. This design would allow us to train a single latent-NeRF model capable of handling multiple target illuminations. As a result, when presented with a new lighting condition, the trained model could directly generate the 3D relit version without requiring further optimization.

[a] Lorraine et al., ATT3D: Amortized Text-to-3D Object Synthesis. ICCV 2023.

Number of samples used for Tab. 1 and 2

We use 16 samples per view for all benchmarking evaluations.

Why is Z = 0?

It is unclear how we could use the optimal Z that best matches the actual material besides optimizing Z using the test set images, which does not seem fair. We found that setting Z to 0 yields good results across both our synthetic and real-world benchmarks (see Tab. 1 and 2, and Fig. 5 and 6). We'll clarify this in the final version of our paper.

We provide more qualitative results for different latent codes in Fig. 1 (see PDF). These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials.

Typo

Thanks a lot for pointing this out. We will correct it.

Fig. 4.(a)’s generations are unrealistic

"More specularity than what is actually present in the hotdog scene" does not imply unrealistic results. Without known source lighting, the RDM samples from the entire distribution of plausible relighting outcomes. We showcased random samples from this distribution, some of which exhibit higher specularity on the hotdog. However, due to ambiguities between materials and lighting (L127), even these seemingly less-likely images are plausible. The subsequent latent-NeRF optimization process focuses on finding the most probable results.

Averaged results in Fig. 4.(b)

This outcome is to be expected: with a sufficiently large sample size, we accurately capture the underlying material distributions. Latent-NeRF maintains consistent appearance across all samples while mitigating inconsistencies. As shown in Fig. 4(b), it preserves the plate's specular effects seen in most samples from Fig. 4(a) while reducing the bread's glossiness, keeping mild specular effects on the sauces.

Lack of detail in Fig. 6

Our method surpasses all inverse rendering approaches in capturing high-frequency details, except for Neural-PBIR. Notably, for the salt can (column 2), our method clearly recovers the ingredient list, which most baselines miss. Similar improvements are evident for the toy car (column 6) and gnome (column 7).

Compared to Neural-PBIR, our results often match or surpass its fine details, see the teapot (column 1), salt can (column 2), toy car (column 6), and gnome (column 7). The cactus (column 4) is an exception, due to UniSDF's inability to produce high geometric quality, as noted in the main paper (L251).

Our method's results will inherently improve with advances in geometry reconstruction, as our RDM trains on perfect synthetic geometry. Neural-PBIR's code is unavailable, but its isolated geometry reconstruction stage (Sec. 3.1 in [39]) would allow us to directly leverage better geometry.

Our results lacking shadows in Fig. 6

Our method consistently produces shadows that accurately match the ground truth. Although Neural-PBIR shows more pronounced shadows on the toy car (column 6), these deviate significantly from the ground truth.

Shadows depend on geometry and illumination, modeled through our radiance cues, leading to realistic shadow placement (Fig. 4.(a)). This accuracy is maintained through Latent-NeRF optimization (Fig. 4.(b)).

Consistency of samples from RDM

To clarify, we do not claim that our RDM generates "consistent" relit images. As our RDM performs single-image relighting, we anticipate diverse relit images across samples for the same view, as well as across different views (see Fig. 4.(a)). The role of the latent-NeRF is to reconcile these inconsistencies into a cohesive 3D relit result.

Questioning claim that light probe unfairly favors Neural-PBIR

We appreciate this valid concern and would like to clarify.

Stanford-ORB provides estimated per-image illumination (L235 and [21]’s Fig. 4 and 5), moving a light probe for each image (PDF’s Fig. 2). Ideally, fixed illumination per object would match reality, but aligning the object and light probe is challenging. This limitation in the Stanford-ORB benchmark can significantly affect results, especially in areas with specular highlights (PDF’s Tab. 1). Consequently, there is no “correct” way to do relighting in Stanford-ORB: our results use fixed illumination, while competitors use per-image illumination.

We rendered the ground truth geometry and materials, obtained by Stanford-ORB in a controlled studio environment ([21]’s Sec. 3.2.2). Each view was rendered under fixed illumination (consistent environment map) and per-image illumination (unique environment map per view). Fig. 3 in the PDF shows both renderings alongside the corresponding real image. Note the significant variations, especially in areas with specular highlights (see marked regions and PSNR).

We computed metrics for each method using both reference renderings as ground truth (Tab. 1 in PDF). Our method excels under fixed illumination but performs worse with per-image illumination. Competitors show the opposite trend, doing better with per-image illumination.

Neural-PBIR did not release code or full results, limiting comparisons to 8 images in [39]’s Fig. 10, where it also performs better with per-image setup. We outperform Neural-PBIR in most metrics:

Computational limitations prevented us from employing per-image illumination, as it would have required optimizing a latent-NeRF for each per-image illumination. Given the trends observed for competitors, we believe our method would demonstrate improved performance if this were feasible.

More qualitative results for Fig. 4

See Fig. 1 in the PDF.

About view-dependent effects

Our latent-NeRF can maintain view-dependent properties corresponding to each generation. Please refer to the cactus (4th row), pepsi (5th row), grogu (6th row), and teapot (7th row) scenes in Fig. 1 in the PDF.

Ablation on #roughnesses

We empirically found that using four radiance cues yielded strong results (Tab. 1 & 2, Fig. 5 & 6). Given the computational cost of training the RDM (L452), we did not conduct further ablations. However, we are happy to provide additional ablation studies in the final version if the reviewer would like us to.

Why not compare against all baselines?

We used the official benchmark results from [17] and [21] to ensure authenticity. We also obtained genuine qualitative results directly from the benchmark.

The only exception is Neural-PBIR, as its code is not publicly available, preventing us from applying it to the TensoIR dataset.

We thank the reviewer for their time in reading our rebuttal and providing the feedback.

We are pleased that the reviewer appreciates the concept and formulation of our method.

Our primary objective was to introduce a novel paradigm with the potential to replace hand-crafted inverse rendering techniques. Although our current method may not significantly outperform all existing baselines, we believe it lays a strong foundation for future advancements in the field.

We now address each question below:

Response to Point [1]

Diffusion model input

There seems to be a misunderstanding with respect to the number of views used as input to our relighting diffusion model (RDM). The RDM model (L168 - 172) is a single-image/view model. It has never been exposed to “multiview images” (see Eq. (4)).

We also agree that it would be interesting to further improve the RDM model by conditioning on multiple views.

Unrealistic relit images | little to no chance that the bun is actually of specular material

We want to emphasize that our RDM models the entire distribution of plausible materials given a single observation of an object without known source lighting.

While material roughness is still ambiguous when the source lighting is unknown, we agree that the samples with specular hot dog buns are less likely, and would hope that a future multi-view RDM model could improve this.

Different appearances for Hotdog’s bun between main paper and rebuttal

For Fig. 4(a) we chose to show a wide spectrum of plausible samples generated by the diffusion model.

For the rebuttal we chose to show more samples, which included less specular buns.

This further demonstrates the wide range of possible combinations between lighting and material. The final reconstructions recover the most likely materials (filter out the shiny bun samples) and do not exhibit any specular highlights for the bun, see Fig. 4(b).

Response to Point [2]

On the benchmark with Stanford-ORB

We agree that in the presence of noisy GT lighting the analysis does not favor any method and will rephrase the related language.

Response to Point [3]

We show qualitatively that our method is the only one that somewhat-consistently recovers correct specularities on this dataset, while all other methods exhibit no specular highlights, apart from Neural-PBIR for a single example (the ball in the rightmost column of Figure 6). For the task of relighting this is perhaps one of the most important features.

Unfortunately, Neural-PBIR only provides 8 qualitative result images (no source code is currently available), hence it is hard to properly compare our results with their approach, and we are happy to clarify our claims in the paper and limit them to only refer to specular highlights.

We also admit that our method sometimes misses details due to the UniSDF reconstruction, and are happy to clarify this in a revised version.

Overall

We are happy that the reviewer recognizes the value of our proposed 3D relighting paradigm, which we consider the core contribution of our work (L42 and 256).

Our method significantly outperforms all baselines except Neural-PBIR. However, due to limited results, unavailable code, and challenges with Stanford-ORB, it's difficult to determine which method is superior.

We also agree that each component of our pipeline has the potential for further improvements. Since the components are independent of each other, improving one should improve overall performance.

We are optimistic that future research will address these areas and build upon our framework.

Why is Z = 0?

It is unclear how we could use the optimal Z that best matches the actual material besides optimizing Z using the test set images, which does not seem fair. We found that setting Z to 0 yields good results across both our synthetic and real-world benchmarks (see Tab. 1 and 2, and Fig. 5 and 6). We'll clarify this in the final version of our paper.

We provide more qualitative results for different latent codes in Fig. 1 (see PDF). These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials.

Result quality in Fig. 6

Our method consistently surpasses all inverse rendering approaches in capturing high-frequency details, except for Neural-PBIR. Notably, for the salt can (column 2), our method clearly recovers the ingredient list, which most baselines miss. Similar improvements are evident for the toy car (column 6) and gnome (column 7).

Compared to Neural-PBIR, our results often match or surpass its fine details, see the teapot (column 1), salt can (column 2), toy car (column 6), and gnome (column 7). The cactus (column 4) is an exception, due to UniSDF's inability to produce high quality geometry, as noted in the main paper (L251).

Our method's results will inherently improve with advances in geometry reconstruction, as our RDM is trained on perfect synthetic geometry. Neural-PBIR's code is unavailable, but its isolated geometry reconstruction stage (Sec. 3.1 in [39]) would allow us to directly leverage better geometry.

About more examples for Fig. 4 (b)

We provided more examples in Fig. 1 of the rebuttal PDF. We are happy to include these in a final version.

Method is limited to NeRF

Our approach introduces a novel 3D relighting paradigm, employing a Relighting Diffusion Model (RDM) as prior to optimize a relit 3D representation. This approach is adaptable to various 3D representations like NeRFs or Gaussian Splats, since both can be conditioned on latent embeddings. For example, Wild Gaussians [a] (Sec 3.2) uses latent embeddings for a similar purpose in a different context (modeling lighting variations in internet photo collections with Gaussian Splatting).

[a] Kulhanek et al., WildGaussians: 3D Gaussian Splatting in the Wild. ArXiv 2024

Advantages over concurrent work

1. The concurrent work focuses on forward-facing indoor scenes, while ours addresses full 360 degree 3D object-centric relighting.
2. Our method can handle intricate environment map-based lighting, whereas the concurrent work is limited to single-source directional illumination, a subset of what IllumiNeRF supports.
3. The concurrent work’s diffusion model relights images only to 18 known light directions in the dataset (Sec. 3.2).
4. Their latent code is per-view, while ours is per-sample (See Eq. 3 and Fig. 8)

Note that EGSR (July 3-5) took place after the NeurIPS submission deadline (May 22).

The pipeline seems brittle

Our presented numerical metrics and qualitative visualizations either produce SOTA results (Tab. 1 and Fig. 5) or highly compelling results (Tab. 2 and Fig. 6).

While we acknowledge that our approach relies on high-quality geometry estimates from UniSDF (L249), it is crucial to note that our diffusion model's training relies solely on synthetic data. This design choice completely decouples the training process from the inference pipeline depicted in Fig. 2. Consequently, our method will benefit from future improvements in geometry reconstruction as well as improvements with regard to the relighting model or the latent NeRF optimization.

Simply relighting the extracted mesh

We are unsure exactly what the reviewer is suggesting.

If the reviewer suggests simply relighting the output from UniSDF, it's important to emphasize that UniSDF generates only geometric information and outgoing radiance, not material properties. Therefore, direct relighting of the mesh is not possible.

If the reviewer proposes to perform inverse rendering based on the geometry predicted by UniSDF, we would like to clarify that one of the main benefits of our method is that it does not require inverse rendering (L42 - 46).

Abstract is misleading

Thank you for the suggestion! We will modify the final version of the abstract to include reconstruction and conditioning.

Use of UniSDF is not justified

UniSDF is a SOTA method for surface/SDF-based NeRF approaches (L387) on the Shiny Blender benchmark [43] (see MAE in [47]’s Tab. 1). We build on top of it, as other non-surface methods would make it harder to obtain high-quality geometry for the radiance cues computation (L249, 388 - 390). Our framework can easily be applied to any NeRF/SDF approaches that recover better geometry in the future.

About highlights in Fig. 6

We will integrate the changes as suggested.

About normalization for Fig. 6

The qualitative baseline results were kindly provided by the authors of Stanford-ORB [21]. Additionally, we utilized the benchmark's official code (https://github.com/StanfordORB/Stanford-ORB/blob/962ea6d2cc/scripts/test.py/#L36) to perform the normalization.

Why focus on objects?

Good question. There are multiple reasons for focusing on objects:

1. Object-centric 3D relighting is vital in various downstream applications, e.g., AR/VR, robotics, and game development.
2. Scene-centric lighting is a much harder task to solve. It is hard to set up realistic illumination for scenes. Conditioning on lighting is non-trivial as lighting representations such as environment maps are not easily applicable to scenes.
3. Established synthetic and real-world object-centric benchmarks like TensoIR and Stanford-ORB enable clear evaluation and comparison.
4. Large-scale object-centric synthetic datasets like Objaverse [10] are readily available for training the relighting diffusion model.

Efficiency of method

We acknowledge that our approach currently prioritizes quality over real-time performance, as generating new samples with the RDM and optimizing a NeRF for each new lighting condition is computationally intensive (L252-254).

However, we believe this focus on quality is crucial as a first step. Our extensive evaluations and thorough numerical experiments demonstrate competitive results, offering a "promising alternative for relighting" (mZi1). We hope our work "inspires future works using diffusion models to hallucinate lighting" (q4ex), with a gradual focus on improving efficiency.

We see parallels in the NeRF community, where the initial MLP-based NeRF [30], despite its impressive results, was computationally demanding. Subsequent research, like instant-NGP [31] and Gaussian Splatting, significantly improved efficiency. We envision a similar trajectory for our approach.

We thank the reviewer for their time in reading our rebuttal and providing feedback.

We are encouraged that the reviewer recognizes the quality improvements and novel contributions introduced by our method.

We would like to clarify that every existing 3D relighting baseline also requires geometry reconstruction.

Our current method did not focus on improving speed, but rather introducing a novel 3D relighting paradigm. It was designed as a general framework that can be further developed to substantially improve speed in the future, e.g. by using Gaussian Splatting representation instead of NeRF as well as using a faster diffusion sampler.