ROGR: Relightable 3D Objects using Generative Relighting

Q1: Visualize more concepts in Section 3.1 and make Figure 6 larger.

Thank you for the suggestion. We will update Figure 2 and integrate the concepts of Section 3.1 into it. We will also update Figure 6 with improved visibility for differences. Note that compared to (b) and (e), our final model produces more realistic specular highlights in the regions marked by the inset (specifically note the “ketchup”). Although (a) appears visually similar, our method achieves slightly better quantitative metrics as shown in Table 3.

Q2: Add discussion in the results section—why baselines fail or why our method performs better.

Methods based on inverse rendering such as NeRFactor, TensoIR, and R3DGS rely on accurate decomposition of geometry and materials, making them susceptible to compounded errors that lead to unrealistic relighting under novel illumination. In contrast, our method bypasses this decomposition by directly generating relit images. Compared to generative relighting methods like IlluminNeRF and Neural Gaffer, our approach leverages multi-view relighting diffusion, which produces more view-consistent outputs for relightable NeRF optimization. Additionally, our environment map-conditioned relightable NeRF more effectively utilizes lighting information for radiance prediction, resulting in higher-fidelity and physically-plausible relighting. We will add the result discussion into the revision.

Q3 : Limitation discussion.

We provided a limitation discussion in the supplementary material, but we are happy to move it into our main paper in the final revision.

Q4: Use of ‘However’ in Line 80

Thank you. We will revise "[...] generalization. However [...]" to "[...] generalization, and [...]".

Q5: Figure reference issue in Line 137

Thank you for pointing this out. We will fix the sentence to: “The conditioning signals are illustrated in Fig. 3.”

Q6: Clarification on the relit dataset

Correct. We will emphasize in Section 3.2 that the relit dataset D_{\text{relit}} is constructed for a single target object.

Q7: Inputs and outputs of the NeRF model.

The inputs to our relightable NeRF include 3D coordinates (x,y,z), ray direction, general conditioning, and specular conditioning. The geometry MLP predicts roughness, normals, and density, while the color MLP outputs the RGB values. We will explicitly describe these components in Section 3.2 to improve clarity.

Q1: Quantitative metrics to evaluate the multi-view diffusion in the relighting task.

Good idea! The table below shows the effect of increasing the number of views input to the diffusion model. For less reflective objects, the inconsistency from 4-view diffusion was already small enough that adding more views did not have a major impact and the NeRF model could reconcile the inconsistency. We observed qualitatively that increasing the number of views for shinier objects was more important, such as the car and sewing machine objects shown in the supplemental “in the wild” capture setup.

Q2: Can the method handle shadows and specular highlights? What strategies are used to handle them?

Shadows and reflections are two challenging phenomena that our model is designed to handle in somewhat different ways. Shadows are strongly lighting-dependent, but they are not view-dependent. In order for our model to render correct shadows, the conditioning signals (and in particular the general conditioning) must encode the lighting in a way that allows the NeRF model to render them. This is enabled by our multi-view relighting model being trained to output correct shadows. Our model’s ability to handle shadows correctly is demonstrated in our supplementary material—see for example the lego scene with the rotating environment map at the top of the main page (index.html) or the car scene of StanfordORB (in stanford_orb.html).

Specular highlights are even more challenging since they are both strongly lighting-dependent and view-dependent. For rendering specular highlights, the diffusion model must be able to correctly relight the input images, and the NeRF must be able to use the conditioning signals (and in particular the specular conditioning) to create fast variations in appearance as the view direction changes. The strong dependence on view direction is achieved through the specular conditioning being concatenated to NeRF-Casting’s reflection feature, which was originally designed to enable rendering highly specular environments. We show that this pipeline is indeed effective in our results—see for example the hotdog scene at the top of the main page (index.html) and under additional lights in the TensoIR page (tensoir.html), and multiple objects like teapot and pepsi from StanfordORB (in stanford_orb.html).

Q3: Discussion with RelitLRM and DIFFUSIONRENDERER. RelitLRM is a concurrent work to ours and directly recovers the 3D geometry and appearance of an object. While their method produces similar results to ours, their method cannot guarantee to produce the same geometry for different environment maps. Our two-stage strategy enables reconstruction of constant geometry under different illumination conditions.

Compared to DiffusionRenderer, which uses a video diffusion model to generate relighting results, our method enforces 3D consistency by design, since we explicitly model the geometry and rendering, whereas DiffusionRenderer does not guarantee 3D consistency. Additionally, our model enables fast inference as it avoids the long sampling times typically required by diffusion-based generation.

Q1: Comparisons with NeRO, Neural-PBIR, and Generative Multi-view Relighting

The table below shows a comparison of our method with NeRO and Neural-PBIR on the TensoIR dataset. We will include these results in our revised version. As pointed out by the reviewer, the setting for Generative Multi-view Relighting (GMR) is different than ours: we aim to relight objects from an unseen environment map, whereas GMR expects an observed image from the target lighting. Furthermore, GMR did not release their source code.

Q2: Additional analysis of the geometry / normal map quality of the relightable NeRF.

The quality of our method’s normals is comparable to prior inverse rendering techniques like TensoIR, as well as novel view synthesis approaches like NeRF-Casting which explicitly encourage geometry to be more surface-like. We will add normal visualizations to our final supplemental appendix.

Q3: Analysis of the quality of the multi-view relight diffusion results.

As suggested, we compare the output of the multi-view relighting model with renderings from our full pipeline which includes the relightable NeRF. Note that the metrics are computed over the test views of the TensoIR dataset. While the NeRF results only use the relit training images, meaning that our pipeline never observed the test images, the evaluation of the relighting model required providing the test views to the relighting model as input. Even though our pipeline does relighting and novel view synthesis, it still performs only slightly worse than a model that gets access to the test views and only needs to relight them.

Q4: Discussion on choosing the hyper-parameter of specular blur kernel size.

​A 2D Gaussian kernel with a radius of 20 pixels (resulting in a 40 × 40 kernel) corresponds to an angular range of (40 / 512) × 360° = 28.125° centered around the reflection ray. A 2D Gaussian kernel with a radius of 40 pixels, aggregates lighting information within an angular range of (80 / 512) × 360° = 56.25°centered around the reflection ray. Based on empirical findings, this hyperparameter selection captures the light transport behavior of most reflective surface materials.

Q5: Number of images used to train the relightable NeRF.

Given N cameras and M environment maps, we use a total of NxM images for training the relightable NeRF.

Q6: The reason for choosing 111 environment maps.

We chose roughly 100 environment maps because our method’s quality increases with the number of environment maps and starts saturating around that point. In the table below we present relighting metrics on the hotdog scene from TensoIR for a varying number of environment maps used.

Q7: Cactus result from Neural-PBIR in Fig.5

Thank you for noticing this! We will fix the cactus visualization issue of Neural-PBIR in Fig.5.

Q1: Additional ablation study on per-illumination optimization using multi-view relit images and the original NeRF-Casting method.

Thank you for the suggestion! See the results of the proposed ablation study in the table below, where we optimize NeRF-Casting directly for each illumination. We will add it to our final revision. While our method performs on par with NeRF-Casting, it has the obvious advantage that it performs relighting in a feed-forward manner and does not require optimization per illumination.

The experimental setup is the same as Table 3 in our paper, which is from the hotdog scene from TensoIR. See the text and Fig. 6 for corresponding qualitative results and additional explanations.

Q2: The performance gap between TensoIR and Stanford-ORB.

The Stanford-ORB benchmark provides illumination captured using a light probe that was repositioned for each image in the dataset, and is not colocated with the object. As such, their reported “environment maps” don’t capture the actual lighting condition at the object's position. The dataset's inaccuracies mean that any metrics derived from this benchmark should be interpreted with caution. This was also pointed out by IllumiNeRF [1] (Section 4.2) and Eclipse [2] (Section 5 and Section S1). This issue causes mismatches between the provided images and the relighting results obtained by every method, which effectively introduces noise to the metrics. This explains why it’s easier to see our method’s quality boost in the TensoIR data compared with Stanford-ORB.

[1] IllumiNeRF: 3D Relighting Without Inverse Rendering

[2] Eclipse: Disambiguating Illumination and Materials using Unintended Shadows

Q3: Ablation study on the effect of varying the number of views in diffusion models for relightable NeRF reconstruction.

Good idea! The table below shows the effect of increasing the number of views input to the diffusion model. For less reflective objects, the inconsistency from 4-view diffusion was already small enough that adding more views did not have a major impact and the NeRF model could reconcile the inconsistency. We observed qualitatively that increasing the number of views for shinier objects was more important, such as the car and sewing machine objects shown in the supplemental “in the wild” capture setup.

Q4: No discussion about limitations

We provided a limitation discussion in the supplementary material. If the reviewer would like us to, we can move it into our main paper in the final version of the paper.