Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport

(Q5) Background conditioned inference

When using the background as a condition, the environment branch can be zeroed out (by completely removing the connection to the neural network and not only setting environment as zero), similar to [1]. We plan to release the models with and without the environment branch for various use cases.

[1] Relightful Harmonization: Lighting-aware Portrait Background Replacement

(Q6) Averaged channels for normal estimation and influence of specular terms

For generating normal maps, we do average all color channels to calculate the intensity. Handling specular terms may involve unique considerations. For example, when comparing shading maps (without hard shadow) under left-to-right versus right-to-left lighting, the two maps should ideally be exact negatives of each other in log space. Any discrepancies or offsets between the two maps often indicate specular reflections. Such offsets could potentially be extracted as specular components using more sophisticated formulations. We are also working on exploring more advanced methods to improve the decomposition formulations.

(Q7) How well does the normal align with a real 3D normal map?

In our study, the normal maps computed by dividing appearances visually resemble more to those captured via light-stage photography metrics (like those normals in [2]) than those from 3D-rendered normals (like Objaverse normals). This is likely due to the hard mesh boundaries (though high-quality 3d normals would have internal local normal textures) and less detailed hair/fur in 3D normals. We find that the appearance of the division-based normals aligns more closely with empirical measurements from light stage setups.

[2] Total Relighting: Learning to Relight Portraits for Background Replacement

(Q8) Temporal consistency in changing the environment map

Temporal consistency is not addressed in this work, and as a result, the model may not produce explicitly optimized temporal coherence when changing conditions. We are taking a close look at latest video models like CogVideo [3] and actively investigating the potential to train IC-Light on those models. For example, recently dimension-X [4] showed that video models can be tuned to produce a short video representing scene rotation, this is likely to indicate that illumination transitions like environment map rotations can be learned in a similar way. This is one promising way to consider future experiments.

[3] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

[4] DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion

(Q9) Imposing lighting consistency on foreground

Light transport consistency is masked by foreground mask $M$ in Eq.4 and Eq.5.

(1) Material ambiguity in the single input image

When the input images contain ambiguities (like materials), the final effect is jointly influenced by the base model, training data, and user prompts. Empirically, we find that stronger base model like Flux and larger, more diverse, higher quality training data are helpful for learning a more accurate mapping (and more responsive user edits from prompts). We will revise the related exposition.

(2) Revising normal estimation formulation and coordinate system

Thanks for pointing out this! The Eq 8 should be minus indeed. We are also revising the coordinate systems.

(3) Potential artifacts in the image results

Thanks for the observations and we are revising some related discussions.

(Q1) Environment map FOV handling and visibility/occlusion handling

In this work, all data on the input side are from methods chosen randomly (and/or with randomized degradations) as data augmentation to make the model more robust, whereas all data on the output side are of the highest quality as possible, like original raw in-the-wild images filtered by CLIP metrics. All randomness and errors on the input-side data are seen as data augmentation so that the model will work even when the input-side distribution is not strictly aligned with the distribution of real-world inputs.

Regarding illumination environment maps's FOV handling, though methods like DiffusionLight does include "generative" patterns for small FOV images, we place a stronger emphasis on the correspondence between the environment lighting map and the pixel colors in the original image. For example, we also use an environment-from-normal approach to project shading colors with their normal direction to a environment ball (supp, L68-74). Using mixed environment map estimators helps the cases when FOV correspondences are relatively weak.

The visibility and occlusion relationships between objects are handled with similar principles to allow certain amounts of error and noise to serve as data augmentations for training robust models. For example, in the environment-from-normal approach, if a position in the normal sphere is mapped with multiple colors from different pixels in the shading maps, we take averages. While this may introduce some physical inaccuracies (it can be less accurate than explicitly modeling occlusion/visibility), it is effective in establishing a learnable correspondence between the lighting map and pixel values, and the extra randomness may also facilitate model robustness. We will add related discussions.

(Q3) CLIP for sample filtering

The CLIP-filtering is mainly for getting rid of data that has no relationship to illumination. For example, most in-the-wild text-image datasets have large numbers of flat vector graphs, icons, UI designs, logos, and so on. Performing intrinsic decomposition on those data are not meaningful for our problem and those data need to be removed. Although CLIP vision is not perfect in text-image alignment, in our tests, it is already powerful enough to get rid of almost all those non-related data.

(Q4) Image-based rendering pipeline

The image-based rendering pipeline prioritizes speed (using pure pytorch matrix dot products) to generate more diverse illumination appearances over rendering quality. This is mainly because the rendered images are only used as degradation images, and their distribution diversity and quantity is much more important than individual quality and fidelity. The image-based rendering pipeline does not include global illumination or inter-reflection modeling between objects. However, it does have random hard shadows (L239).

Thank you for the careful and constructive comments in this evaluation effort. We first discuss some considerations in the light transport consistency and then discuss each question in detail.

MLP in light transport consistency (LDR vs. HDR, functionality as proxy, stability in x0 interpolation for extreme tSNR)

Thanks for the question about LDR vs. HDR. This is indeed an important consideration. Actually in our work the challenge is a bit beyond LDR vs. HDR domains and extends to handling the images in latent space. The MLP acts as a bridging component, learning an implicit conversion that allows for smooth appearance interpolation in any of those domains (LDR / HDR / latent). As a result, the MLP is always necessary whenever the training data are not HDR images, no matter whether the diffusion model is pixel diffusion or latent diffusion. We will revise the related expositions.

We also appreciate the insight to view this consistency as a proxy for numerical sum, and want to point out that the consistency is a bit more than "without any other regularization". The consistency is learned together with vanilla diffusion objective (Eq.6 in main paper) and the vanilla diffusion objective is a strong regularizer that makes sure that the model output follows the desired diffusion objective type. The consistency itself does not have any diffusion objective in it and cannot be used independently to train a diffusion model.

Finally, thank you for bringing up the stability in x0 interpolation for extreme tSNR. This is a really inspirational way to consider the problem. Lets consider a simple interpolation of $f(k, x_a, x_b)=k x_a + (1 - k) x_b$ where $x_a$ and $x_b$ are two different $x_o$ estimated using two independent model inferences with different conditions. The stability in extreme tSNR can be seen as a case where the matrices of $x_a$ and $x_b$ have very large data range (or standard deviation) because they have been premultiplied by very large ddim alphas_cumprod (or divided by very small k-diffusion sigmas). The question is whether approximating a model's output to $f(k, x_a, x_b)$ will influence numerical stability.

Some related experimental evidence is the wide existence of CFG-distilled models. For any real number $k > 0$, we have $f(k, x_a, x_b)=k x_a + (1 - k) x_b = x_b + k (x_a - x_b)$ if $x_b$ is estimated from empty condition, then $f(k, x_a, x_b)$ is the exact CFG formulation on conditional $x_a$ with guidance $k$. If we train a model to approximate this $f(k, x_a, x_b)$, we will obtain a CFG-distilled model. Note that in this case, this numerical range of $k > 0$ is even more aggressive than that in our paper (between 0 and 1). Considering the widely available of CFG-distilled models like Flux-1.dev and many LCM/Lighting models (many timestep-distilled models are also CFG-distilled), the terminal SNR does not seem to influence the model quality when approximating $f(k, x_a, x_b)$.

In our work, the $f(k, x_a, x_b)$ is not receiving $x_a$ and $x_b$ from positive/negative conditions like CFG, but receives two different illumination environment maps as conditions. This should empirically produce an even smaller disparity between $x_a$ and $x_b$ and even less numerical instability than CFG distillation models. Considering that the tSNR instability is not frequently observed in CFG distillation literature, the interpolation for illumination conditions is likely to have similar stable training. We will add some related discussions to supplementary materials.

Thank you for the insights in the evaluation and the appreciation in the application of image illumination editing tool.

With regard to the model and data, our current plan of the release timeline is (1) SD1.5 models and demos, and then (2) demos for latest models like Flux, and then (3) more ablative models for SD1.5 and some related data, and then (4) more weights and inference codes for latest models like Flux, and then (5) more ablative models for latest models like Flux, and then potentially (6) more related training implementations and data processing. We are also considering having a standard standalone software.

Thank you for the insights in the evaluation and the hints for revising manuscripts.

(1) More discussions for literature in computational photography and low-level vision communities, especially in terms of intrinsic image decomposition.

Thank you for this suggestion. We will include more discussions (a new section) about works in intrinsic images, computational photography, and related low-level vision methods.

(2) More quantitative experiments.

Thank you for this feedback. We are working on improving quantitative presentations, and will add dataset examples in the supplementary materials.

(3) Revising expositions for some figures and fixing some typos.

We will revise the manuscripts, captions, and figures according to the comments.

(Q1) What about general scene-level relighting?

Thank you for the insights about scene-level relighting. The scene-level illumination poses additional challenges, especially when accounting for internal light sources (that cannot be represented by environment maps), occlusions, and complex hard shadow interactions. These elements require more sophisticated data synthesis or inference pipeline. A possible future direction is decomposing the scene into objects/components, applying individual illumination effects, and blend components with shadow generation, inter-reflections, and other types of harmonization. This is definitely a promising and challenging open direction, and we will include it in the revision as well as references to the mentioned related paper.

(Q2) Consistency in high-quality lighting effects and hyper-parameters

The framework is designed to be robust to diverse user inputs and lighting conditions. Figure 1 is designed to display typical use cases like hard shadow (the woman), changing light direction (the man), transparency (the perfume), special effects (the neon car), etc. Consistent high-quality results are relatively easy to obtain in simple cases like altering the illumination color or direction (like the perfume bottles and man). For more complicated cases like obtaining a specific shape of hard shadow casted by Venetian blind patterns, more prompts are usually needed for achieving the very customized effects.

Thank you for the insightful comments and evaluation.

(1) It would be great to highlight more which datasets are used for which parts of the training.

For in-the-wild images, degradations are used as training inputs $I_d$. For 3D data and light stage data, degradations are not needed and we directly use random illumination environment maps to render random appearances as training inputs $I_d$ (L249). To be specific, 3d data are rendered by the rendering pipeline and light stage data are rendered by weighting one-light-at-time images. We will revise the paper to clarify this.

(2) Additional citations and discussions and better bold highlight in tables.

We will incorporate the additional references and clarify the tables with bold highlights in the revision.

(Q1) More details would be helpful on the synthesis of L1 and L2 (L.319), probably even some visuals in the supplementary.

Thank you for this suggestion and we are working on adding visualizations to the supplementary material.

(Q2) More qualitative samples from the dataset with all the methods used to generate the synthetic components.

We appreciate this suggestion and we are working on adding some dataset examples in the supplementary material.

Thanks for the public comment. The “below section” around supplement L44 refers to the section around supplement L67-73. The shading map computation does not use gamma correction, but image rendering often have a step from HDR images to LDR and that step may have an effect similar to gamma correction. About multiplying intrinsic components and processing normal vectors, a good tutorial is “An L1 Image Transform for Edge-Preserving Smoothing and Scene-Level Intrinsic Decomposition” from Bi et al, with many fundamental tutorials. Another great tutorial is “Shape, Albedo, and Illumination from a Single Image of an Unknown Object” from Barron et al. 2012 for processing environments with image-space shading and normals

Thank you for the interesting paper and the cool results!

Here are some things I noticed that you might want to fix in the final version of the paper. I hope this helps:

• lines 289 and 290: the number 2307 should probably be $2304 = 3 \cdot 768$
• line 296 (Eq. 1): $\mathcal{E} \to \boldsymbol\varepsilon$; $t$ should not be bold as it's a scalar; delete one extra closing parenthesis
• line 335 (Eq. 4): missing closing parenthesis inside the norm
• line 337: delete one “same”
• line 341: (Eq. 5): $\mathcal{E} \to \boldsymbol\varepsilon$; $t$ should not be bold as it's a scalar; delete one extra closing parenthesis in the middle; the 1/2 subscripts for $\mathbf{L}$ should not be bold
• line 347: $\lambda_\text{relight} \to \lambda_\text{vanilla}$

Please consider discussing IntrinsicDiffusion (SIGGRAPH 2024) in your new related work section on intrinsic images; it seems quite relevant.

It would also be good to clearly define the projection format of the lighting environment (which direction each pixel maps to) as square aspect ratios are unusual.

Thanks for the comment! In the latest version, some expositions should relate to this: (see also new supp.fig.4) ... A typical full environment map is usually of ratio 2:1, with size 64x32 when convoluted. We use the front half (facing the image) of the convoluted environment map, which is 32x32. Using the front half makes normal-based environment extraction easier (since the image-space normals often do not have any pixels facing to the back half) ...

Thanks for providing the additional sources for references!

Normal map and environment map will create a shading map (that can be applied to appearances) with that environment, and yes, it is better to always call it shading map. Hard shadow are from shadow materials, whereas soft shadows are from normal and environment maps. About randomized specularity, the synthesizing is by increasing the specular of random areas (for Phong it is Specular term; for Cook-Torrance and other bsdf one can also use roughness and/or metallic etc).

Thanks for this comment! This is a typo and should be fixed soon: The $I_{L_1}$ and $I_{L_2}$ in Eq. 5 should be $I_{L}$ (The “1” and “2” are typo). And here $L=L_1+L_2$. (see also Fig.3-(b)) Thanks again for finding this!

Congratulations on the amazing work! I have a confusion about derivation of line 324 in page 7. If the $I_{L_1}$ and $I_{L_2}$ in Eq. 5 is $I_{L}$, we apply Eq. (3) as $I_{L_1+L_2} = I_{L_1} + I_{L_2}$ and get the following equations:

$

\frac{I_{\sigma_t}^{L} - \epsilon_{L_1+L_2}}{\sigma_t} &= \frac{I_{\sigma_t}^{L} - \epsilon_{L_1}}{\sigma_t} + \frac{I_{\sigma_t}^{L} - \epsilon_{L_2}}{\sigma_t} \\ I_{\sigma_t}^{L} - \epsilon_{L_1+L_2} &= I_{\sigma_t}^{L} - \epsilon_{L_1} + I_{\sigma_t}^{L} - \epsilon_{L_2} \\ \epsilon_{L_1+L_2} &= \epsilon_{L_1} + \epsilon_{L_2} - I_{\sigma_t}^{L}

$

which conflict to the $\epsilon_{L_1+L_2} = \epsilon_{L_1} + \epsilon_{L_2}$ in line 325. Could you please explain my little confusion? Thank you!

Thanks for the comments. The $I_{L_1+L_2}=I_{L_1}+I_{L_2}$ measured by real-world validation is for HDR images. The images for training are LDR images and need MLP as an implicit conversion to make use of this equivalence. Scaling HDR images by any global scaler does not influence its LDR appearance (if tone-mapping has intensity pre-norm).

When denoising with Fig.3 (b), the clean latents for the two $L_{1,2}$ are unknown. Fig.3 (b) should be seen as a big model as a whole to denoise noisy latents from known clean latents $I_L$ using conditions $I_d, L_1, L_2$ and prompts.