DreamLight: Towards Harmonious and Consistent Image Relighting

We sincerely thank your insightful comments and valuable feedback. To the concerns expressed in Weaknesses and Questions, our response is as follows:

W1: The potential effect of SFF.

Thanks for your kind question. Since high-frequency components are primarily related to the detailed textures of the subject, the modifications made by SFF to high-frequency components are expected not to affect effects such as shadows. This is based on the assumption that the high-frequency components of an image correspond to the pixels varying drastically, such as object boundaries and textures, while the low-frequency components correspond to the high-level semantic information such as the color and light. Could you kindly describe what the "incorrect shadow effects" are? We are a bit confused about where the issue lies. Typically, our model can produce desirable light interaction effects in diverse scenarios. This is also confirmed by non-cherry-picked user study and extensive evaluations experiments. Certainly, there exists failure relighting cases in some complex scenarios for our model, but it may be unrelated to SFF. It is mainly because the initial relighting results generated by the diffusion model may have errors. We believe constructing more high-quality data in complex scenarios can further enhance the performance of our model.

W2: Discussion about SFF.

SFF is designed to preserve details of the foreground subject (such as the identity of small human faces). Actually, it can also be trained using real relighting data. Theoretically, if real relighting data has very strong subject consistency, SFF trained on it would achieve good performance. However, we do not use actual relighting data because they may contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although we have utilized VLMs to filter low-quality data, there are still cases where foreground details have altered. Using such data for training may have adverse effects on SFF. Therefore, we directly use a self-supervised manner (color transformations) to construct pseudo relighting pairs with strongly consistent details for training SFF and find it works well.

Following your kind suggestion, we print the numerical distributions of $\alpha$ and $\beta$ and compare them with the high-frequency part. Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide the distribution visualization images of these parameters for demonstration. Therefore, we present the statistical data of their distributions in the table below. $HQ _ {in}$ and $HQ' _ {in}$ denote the input and the adjusted high-frequency component, respectively. $HQ' _ {in} = HQ _ {in}*\alpha + \beta$. Mean, Min, and Max indicate the distribution of each parameter. It can be seen that $\alpha$ and $\beta$ work together to adjust the distribution of high-frequency components, so as to adaptively enhance the details of the subject without affecting the relighting effects. Visual ablations of Figure 8 in the paper also illustrates the effectiveness of our SFF.

W3: The meaning of $\beta$.

The design philosophy behind our SFF being engineered to output both $\alpha$ and $\beta$ is inspired by neural network layers. Since neural network layers often contain weights and biases, we intuitively adopted two predictive parameters and defined: $HQ' _ {in} = HQ _ {in}*\alpha + \beta$. Ideally, $\alpha$ is utilized to control the influence degree of initial high-frequency components and $\beta$ contains the adjustment information. Besides, the modulator is somewhat like a style transferrer, and the commonly used AdaLN operation also has scaling coefficients (our $\alpha$) and offset coefficients (our $beta$). We assume that these commonly used operations contain two parameters is because such approach can facilitate model learning.

W4: Performance metrics confusion.

There appears to be some misunderstanding here. The values you compared are likely from Table 4 (the ablation study on SFF) and Table 1 (the main results) in the paper. However, their test settings and metric calculation are actually different. As described in the paper from L304 to L307: "SFF primarily works on critical small regions. While the refinement of these regions is important for visual quality, it has limited impact on global metrics. Thus, to better test the refinement effect, in Table 4 we crop small face regions for metric calculation."

Actually, when calculating the metrics for the entire image, the performance of the model without SFF (using only PGLA) is numerically close to that of the final version and significantly outperforms previous methods. The corresponding results are shown in the table below.

W5: Inference latency.

Thanks for your valuable advice! We provide the inference cost (FPS and memory utilization) of the proposed modules in the table below. All the value are obtained on a single H20 GPU. For comparison, we have also listed the relevant cost of ICLight.

Moreover, to further demonstrate the inference latency of different parts of our model, we have counted the inference time of each module, and the results are as follows.

We will update these results in the manuscripts.

Thanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you.

Dear Reviewer 6h7r,

We sincerely thank you for your efforts in reviewing our paper and your valuable comments. May I kindly ask if we have fully addressed your concerns? If there are any remaining concerns, please feel free to raise them and we will try our best to explain.

Sincerely,

Authors

We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:

W1&Q1: Light queries

Thanks for your valuable question! We set the light directions as "up, down, left, right" because the lighting information is derived from a 2D background image. In our view, the light sources in a 2D image tend to align with horizontal and vertical axes. Therefore, these four directions serve as a natural "basis" for representing lighting orientations in 2D space. Since light queries are high-dimensional features and there are multiple queries responsible for each direction (by default, 4 per direction, totaling 16), they are capable of representing diverse light information. Besides, the biased perception of each direction changes continuously (for example, the decay map responsible for leftward light gradually changes from 1 at the far left of the image to 0 at the far right). Direction-biased masked attention is a soft constraint rather than a hard distinction. High-dimensional light queries still have the ability to perceive multiple light sources and the foreground also interacts with all basic direction light features under the guidance of position. As a result, more complex lighting patterns can be captured by organizing signals from these four base directions. To prove that, we construct images featuring colors in 6 different directions as conditional background images for testing. Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide the generated results for illustration. We observe that the output relighted results can perceive such complex light source.

We hope this explanation offers greater clarity. If you have any remaining questions, we would be glad to elaborate further. We will also refine the corresponding representations in the manuscript to make it clearer. As for extending the PGLA mechanism to support more flexible or adaptive directional queries, we consider this an interesting direction for exploration. It may contribute to more significant and natural light interaction effects in complex scenarios. However, how to enable them to obtain ideal information is quite challenging, which will be one of the directions for our future work. Thanks!

W2&Q2: Training set and evaluation of SFF

Thanks for your great question! SFF is designed to preserve details of the foreground subject (such as the identity of small human faces). Actually, it can also be trained using real relighting data. Theoretically, if real relighting data has very strong subject consistency, SFF trained on it would achieve good performance. However, we do not use actual relighting data because they may contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although we have utilized VLMs to filter low-quality data, there are still cases where foreground details have altered. Using such data for training may have adverse effects on SFF. Therefore, we directly use a self-supervised manner (color transformations) to construct pseudo pairs with strongly consistent details for training SFF and find it works well.

All our evaluation of SFF are conducted on real relighting scenarios. On the one hand, the quantitative evaluations in the paper are all performed on real relighted images. The demonstration figures in the paper (both image-based and text-based ones) are also results processed by SFF, which demonstrates that SFF can capture real light. On the other hand, we have also tested the relighting effects of images from real-world daily scenes, and SFF can maintain the details of the subject without compromising reasonable relighting effects.

W3&Q3: Computational cost and inference speed.

Thanks for your valuable advice! We provide the inference cost (FPS and memory utilization) of the proposed modules in the table below. All the value are obtained on a single H20 GPU. For comparison, we have also listed the relevant cost of ICLight.

Moreover, to further demonstrate the inference latency of different parts of our model, we have counted the inference time of each module, and the results are as follows.

We will update these results in the manuscripts.

W4: Relative importance of PGLA and SFF.

PGLA and SFF have entirely different design purposes and functions. Specifically, PGLA is intended to enable the model to generate more reasonable light interaction effects in image-based relighting scenarios (serving to enhance relighting effects). In contrast, the role of SFF is to maintain the consistency of the subject, especially in regions with high information density such as small human faces (serving for ID preservation). They function in different aspects, so there is no comparison of which is more important. For instance, if only PGLA is used without SFF, the output image would have good relighting effects but the human face may be distorted. Conversely, if only SFF is utilized without PGLA, the model may produce results where foreground details are well preserved but the light effects are unreasonable.

Q4: Discussion about failure cases.

Typically, thanks to the PGLA module, which enables reasonable light interactions between the foreground and background, our model achieves great relighting effects even in scenes with strong lighting and shadows. However, there are indeed some failure cases in extremely complex scenarios. For example, when sunlight shines through multiple gaps in tree canopies, our model may struggle to generate all fully plausible light spots. We consider these scenarios to be highly challenging, and incorporating 3D-related representations might help address such cases. This will be one of the directions for our future improvements.

Thanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you.

Dear Reviewer Vqyj,

We sincerely thank you for your efforts in reviewing our paper and your insightful suggestions. May I kindly ask if we have fully addressed your concerns? If there are any remaining concerns, please feel free to raise them and we will try our best to explain.

Sincerely,

Authors

We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:

W1&Q1: Implementation details of SFF.

Thanks for your valuable advice! Specifically, we combine the high-frequency foreground $HQ _ {in}$ and low-frequency relighting result $LQ _ {out}$ to form a single image as the input. To generate two modulation parameters, we modify the output channels of the last layer of the VAE to 6, and during initialization, the new channels copy the weights of original channels. We will revise the description of this part in the manuscript to make it clearer.

W2: Evaluation dataset.

Thanks for your valuable question! We understand your potential concerns. However, there are indeed no widely used public benchmarks for relighting based on purely natural images. Existing public test sets are mostly based on environment maps. To enable testing on such dataset, we render the corresponding 2D background images of each view from the HDR panorama maps based on the relevant information (FOV, camera extrinsic parameters) provided by the dataset. Take TensoIR as an example, we calculate the metrics on all test views with the scenarios of provided HDR maps and the results are shown in the table below. In fact, the locally rendered background images (just patches in the scenarios) in this way are actually unable to provide sufficient light information (there is a huge gap compared with the information that HDR panorama maps can offer). Therefore, there is a discrepancy between the prediction results of the evaluated models and the ground truth provided by the dataset. However, we observe that under such background conditions, previous methods also expose some problems, such as color bleeding of ICLight and weak light changes in INR. Generally, our model performs better in most cases.

Besides, our test set is indeed not deliberately cherry-picked. When creating the test set, manual intervention only aimed to ensure diverse scenes (for evaluating generalization ability) and natural ground-truth results, without referring to the outcomes of our model. Additionally, we will make our test set publicly available to benefit the community.

W3: Ablation gap.

Thanks for your helpful suggestion! We have experimented with the ablation of IP-Adapter with filtering. Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide visual images for demonstration. Therefore, we present the results of quantitative comparisons below and describe the visualization comparison through text. Specifically, as shown in the results in the table below, applying the filter without the light queries (i.e., IP-Adapter with filtering) does not bring significant performance improvement. Visualization results also show that the relighting effects of IP-Adapter with filtering are similar to those of the vanilla IP-Adapter, with no obvious light interaction effects. This is because although filtering is introduced to enhance the representation of high-level light information in the background, IP-Adapter with filtering, like the vanilla IP-Adapter, performs unguided interactions between the subject and the background, resulting in light tending to have a general effect. In contrast, through the utilization of direction-biased masked attention and light queries, our PGLA selectively transmits background lighting information, enabling the foreground to acquire lighting that is harmonious with the background.

Q2: Text prompt.

Thanks for your valuable question! As mentioned in L118 in the main paper, the text prompt is set to "blend these two images" for image-based relighting. This is the same for both training and inference by default. We will make the corresponding descriptions clearer. In addition, our model also has the generalization ability for other text prompts, such as "natural light". However, since "blend these two images" is used during training, this prompt yields the best results in most inference cases.

Q3: Reflections or subject-light interactions in background.

Great question! In fact, this is somewhat related to the definition of the task. For image-based relighting, we follow the definition of this setting in previous works: only modifying the foreground region to adapt to the background while keeping the background region unchanged. The training objectives and corresponding training data are also organized in this setting. Hence, in examples like the one in Figure 1 of the supplementary materials you mentioned, there is no reflection of the dog in the background. In contrast, for text-based relighting, since the background is also generated by the model, there are interaction effects between the foreground and background (such as shadows), as demonstrated in the examples in Figure 3 of the supplementary materials. Our pipeline and the proposed SFF do not suppress interactions such as reflections or shadows in the background region.

Besides, we believe this question highly insightful. If the background region in image-based relighting could also exhibit reasonable interaction effects, the overall result would be more natural and realistic. However, this also poses additional challenges, such as how to obtain such training data in large quantities. This will be a direction for our further optimization in the future.

Thanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you.

We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:

W1&Q1: Evaluation on public datasets.

Thanks for your valuable advice! Our model and the compared works all rely on pure natural images rather than HDR maps for providing light information. To the best of our knowledge, there are indeed no widely used public benchmarks for relighting based on purely natural background images. The benchmark you recommended (TensorIR) also relies on HDR panoramas to provide light information rather than mere 2D natural background images.

To enable testing on this dataset, we render the corresponding 2D background images of each view from the HDR maps based on the relevant information (FOV, camera extrinsic parameters) provided by the dataset. We calculate the metrics on all test views with the scenarios in "high_res_envmaps_1k" folder and the results are shown in the table below. In fact, we find that the locally rendered background images (just patches in the scenarios) in this way are actually unable to provide sufficient light information (there is a huge gap compared with the information that HDR panorama maps can offer). Therefore, there is a discrepancy between the prediction results of the evaluated models and the ground truth provided by the dataset, leading to generally low overall scores. However, we observe that under such background conditions, previous methods also expose some problems, such as color bleeding of ICLight and weak light changes in INR. Generally, our model performs better in most cases.

Furthermore, we understand your potential concerns, yet our test set is indeed not deliberately cherry-picked. When creating the test set, manual intervention only aimed to ensure diverse scenes (for evaluating generalization ability) and natural ground-truth results, without referring to the outcomes of our model. Additionally, we will make our test set publicly available to benefit the community.

Q2: Relighting effect on general objects with different materials.

Thanks for your great question! Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide relevant images of relighting results for demonstration. The reason why the paper mainly presents portraits is that some of the comparison methods are mainly trained on portrait data. Actually, our model has great generalization ability for general objects with different materials.

On the one hand, as shown in Figure 5 of the supplementary materials, our training data contains, and even includes a large amount of, samples of general objects. Specifically, the ratio of portrait data to non-portrait data is about 3:2, which ensures the effectiveness of our model in relighting general objects. On the other hand, the benchmark (TensoIR) mentioned in Q1 is related to general objects of different materials, and the relevant results demonstrate the superiority of our method over previous approaches in handling general objects. In addition, we have also constructed a new test set consisting of 100 images of pure general objects (without portraits) and compared the performance. The results below also prove the effectiveness of our method.

W2: Spectral Foreground Fixer seems to be less tailored for relighting.

Thanks for your insightful question. SFF is designed to preserve details of the foreground subject (such as the identity of small human faces) while minimizing the impact of reorganizing frequency components on relighting effects. Thus its design also takes into account factors related to relighting effects. Specifically, directly combining the high-frequency components of the input foreground with the low-frequency components of the initial relighting result may cause artifacts and biased lighting in some scenarios. To generate more consistent and natural results, we propose to train a modulator to adaptively adjust different parts. This enables SFF to have good generalization ability for diverse light.

High-frequency light effects (such as specular reflections) often appear in regions with low information density (e.g., non-facial regions). Since SFF can adaptively adjust the influence degree of the high-frequency components, it will make more use of the relighting information in such cases, thus achieving desirable results. We have tested such cases where smooth object surfaces are exposed to intense light. The results show that the relighting outcomes are satisfactory, and SFF does not have a negative impact on such scenes. However, in some cases (such as when there are high-frequency light spots on a small face), SFF may affect the relighting and soften the light effect. This is because there exists a trade-off between them. Users can control this by adjusting the kernel size of the wavelet transform. How to better improve both aspects simultaneously for such challenging scenarios is also an important direction for our future work.

W3&Q3: Further explanation about PGLA.

The reason for setting "up, down, left, right" light directions is that the light information is provided by a 2D background image. We assume that the light sources in a 2D image are mainly in the horizontal and vertical directions. Therefore, the four directions (up, down, left, right) can form the "basis" for the light directions in a 2D image.

Besides, the biased perception of each direction changes continuously (for example, the decay map responsible for leftward light gradually changes from 1 at the far left of the image to 0 at the far right). The foreground also interacts with all basic direction light features under the guidance of position. Thus, other types of light patterns can be automatically perceived through the combination of information from the up, down, left, and right directions. For instance, a light source in the upper-left corner corresponds to strong perceptions of leftward and upward light. We wonder if this explanation is clearer. If you have any further doubts, we are happy to provide further clarification.

W3&Q4: Time cost of SFF.

Thanks for your valuable advice! We provide the inference cost (FPS and memory utilization) of the proposed modules in the table below. All the value are obtained on a single H20 GPU. For comparison, we have also listed the relevant cost of ICLight.

We will update these results in the manuscripts.

W3&Q5: Training data of SFF.

Thanks for your great question! We do not use actual relighting data because they contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although it undergoes screening, there are still cases where foreground details have altered. Using such data has adverse effects on SFF, causing its learning objectives to deviate. Therefore, we directly use a self-supervised manner to construct pseudo pairs with strongly consistent details for training SFF.

Typo

Thanks! We have carefully checked the manuscript and corrected the typos.

Thanks again for your valuable time and comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you.

Dear Reviewer X4Rx,

We sincerely thank you for your efforts in reviewing our paper and your valuable feedback. May I kindly ask if we have fully addressed your concerns? If there are any remaining concerns, please feel free to raise them and we will try our best to explain.

Sincerely,

Authors