Latent Intrinsics Emerge from Training to Relight

We appreciate Reviewer n7pv's recognition of the novelty and usefulness of our work, particularly in training models for relighting without supervision on ground-truth intrinsic properties. We are also grateful for the positive feedback on the clarity and straightforwardness of our presentation, and the comprehensiveness of the methodology and experiments.

Regarding Ablation of regularization terms

Thank you for suggesting additional ablation experiments. In response, we trained a model using our optimal configuration ($\alpha=0.5$) for the relighting experiment, excluding the regularization terms in Eq. 6 and Eq. 9 from the training objectives. The results, presented in the attached tables, clearly show that the proposed regularization terms significantly improve performance in both relighting and albedo estimation tasks.

_

We appreciate Reviewer t1Cn's feedback and are grateful for their recognition of the simplicity and effectiveness of our proposed method, the thoroughness of our experiments, and our approach to unsupervised training. We are pleased to see the acknowledgment of our design choices, the clarity and straightforwardness of our paper's writing, and the practical implications of our method for downstream tasks. We now respond to the reviewer's questions.

Regarding Consistency of Our Albedo Predictions

We conducted an experiment to evaluate the consistency of albedo predictions under varying lighting conditions. Specifically, we estimated albedo for the same scene under different lighting settings, as provided in the MIT dataset. Qualitative visualizations are included in the attached PDF. For comparison, we also evaluated the state-of-the-art supervised intrinsic estimator, Intrinsic Image Diffusion (IID) [1]. Notably, IID requires 10 independent estimations followed by averaging to produce a single image estimation. Due to the computational expense of this process, we assessed albedo consistency under only five different lighting settings.

Our method, on the other hand, produces consistent results with just a single forward pass. Our findings demonstrate that our approach achieves stable albedo estimation despite not utilizing any albedo-like maps as supervision. Also, while IID often exhibits significant color drift due to its reliance on synthetic training data, our method maintains color fidelity, as shown in our qualitative figures. To quantify the albedo stability under varying lighting conditions, we report the mean deviation and standard deviation on albedo map, normalized per scene,

Our method produces more stable albedo estimations under varying lighting conditions compared to the supervised IID approach.

[1] Kocsis et al., Intrinsic Image Diffusion for Indoor Single-view Material Estimation. CVPR 2024.

On Missed citations

Thank you for bringing these papers to our attention. We will ensure that these citations are added to the related work section in our final draft.

We appreciate Reviewer VUQQ's feedback and are grateful for their recognition of the novelty of our proposed method, our approach to unsupervised training, and the quality of our results and paper writing. We now address the specific concerns and questions raised by the reviewer.

Concern: Lack of Fine-Grained Control in Relighting because of latent lighting representation

While it is true that using latent lighting representation for relighting may limit the ability to achieve fine-grained control over specific lighting parameters such as intensity and direction, this limitation is not unique to our approach. Current explicit parametric methods, like spherical harmonics or spherical Gaussians, also face challenges in providing precise control over these aspects, often due to the complexity and computational cost involved.

Our approach, however, offers a significant advantage by simplifying the relighting procedure. By learning a high-dimensional latent representation, we bypass the need for detailed manual tuning of lighting parameters, making the process more efficient and accessible. Additionally, we have demonstrated the ability to interpolate in the latent space to achieve various lighting effects, as shown in Figure 4. This capability suggests that while explicit fine-grained control is limited, our method provides a practical and scalable solution for many real-world applications where such precision may not be critical. Future work will explore ways to enhance control, potentially by mapping latent representations to explicit lighting parameters for users who require finer-detailed adjustments.

Concern: Generalization Ability

We respectfully disagree with the reviewer's assessment regarding the generalization capabilities of our model. Our paper demonstrates two key aspects of the model's generalization ability, which we detail below:

1. Generalization to Unpaired Images: While our model is trained on paired images captured under the same scene, we evaluate its performance on unpaired images. This is illustrated in the first two columns of Figure 3, where the input and reference images originate from different scenes. We opted for paired training data due to practical scalability considerations, as using unpaired data necessitates the calibration of exact extrinsic lighting conditions across different scenes. Our model significantly outperforms comparable approaches in accurately rendering relit patterns while preserving environmental details.

2. Generalization to Out-of-Distribution Images: We trained our model on the Multi-Illumination dataset and directly tested it on the IIW dataset for albedo estimation, without any additional training or prior exposure to the IIW dataset. Despite the considerable distribution shift between these datasets, our method consistently outperforms other approaches in albedo estimation, even without using any albedo-like supervision.

To further substantiate our model's generalization capability, we provide additional results as suggested by the reviewer. We tested the relighting performance of models trained on different datasets, including demonstrating plausible relighting of StyLitGAN and IIW images using models trained on the MIT Multi-Illumination dataset. These experiments show the model's ability to generalize to unseen images using two sources of extrinsics: (a) estimating extrinsics from in-distribution images and (b) estimating extrinsics from out-of-distribution images. Visualizations of these results are included in the attached PDF. While these results demonstrate plausible relighting, we acknowledge that they are not perfect, primarily due to the distinct nature of the distributions. The MIT Multi-Illumination dataset does not feature visible luminaires or light sources, whereas StyLitGAN images may include visible luminaires, with the quality dependent on StyleGAN's generative capabilities. We believe that expanding our training dataset will enhance rendering quality, as our approach can easily scale due to its lack of reliance on supervision.

We appreciate the reviewer's feedback and questions, and we will incorporate these clarifications into the final version of our manuscript for improved clarity.