Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model

Q6. Performance in extreme conditions

Three extreme cases can lead to suboptimal Retinex decomposition: (1) Severe environmental degradation (blurred texture details), (2) Extreme lighting conditions (uneven lighting), and (3) Situations where the Retinex components of the degraded image are nearly identical to those of the ground truth.

• In the first extreme case, our method addresses this by extracting highly compact Retinex priors from low-quality inputs, enabling robust enhancement despite complex degradation. To validate its effectiveness, we assess performance on the underwater image enhancement (UIE) task, which inherently suffers from noise, color casts, and distortion. As shown in the table, our method achieves state-of-the-art results. Additionally, we employ the advanced image quality assessment method DA-CLIP [2] to identify the 15 underwater images with the lowest quality and most severe illumination issues, forming two sub-datasets: LSUI_Extreme and UIEB_Extreme. Performance on these sub-datasets further underscores our method's leading capability to handle extreme degradation scenarios.

• In the second extreme case, despite interference from highly imbalanced or inconsistent lighting, our Retinex priors effectively capture valuable information from degraded inputs, providing stable guidance for enhancement. This is demonstrated by our superior performance in the backlit image enhancement task, where images suffer from significant illumination imbalance. By analyzing pixel distribution histograms, we select the 15 most challenging images with extreme and complex lighting, creating the BAID_Extreme dataset. As shown in the table, our method continues to outperform competitors, achieving leading results even under such demanding conditions.

• In the third extreme case, when the Retinex components of the degraded image closely resemble those of the ground truth, our method remains effective. Results in Table 7 and Fig. S1 highlight the superiority of our Retinex priors in guiding the enhancement process, ensuring robust performance even in these challenging cases.

[2] DA-CLIP, ICLR 2024.

Q3. Difference between low-quality image and GT at the prior level

We further utilize the Mean Squared Error (MSE) metric to quantify the differences between the Retinex priors extracted from low-quality images and those derived from the ground truth. As shown in the table, the observed differences are minimal, underscoring the effectiveness of our method in directly extracting high-quality priors from degraded inputs. Additionally, to validate that our extracted priors are more aligned with those obtained from the ground truth, we introduce two baselines: $\mathbf{Z} _ \mathbf{R}$-Ablation1 and $\mathbf{Z} _ \mathbf{L}$-Ablation2. $\mathbf{Z} _ \mathbf{R}$-Ablation1 represents the MSE between $\mathbf{Z} _ \mathbf{R}$ extracted from the ground truth and the ablated version of "w/ $\mathbf{Z} _ \mathbf{R}$" in Table 7, while $\mathbf{Z} _ \mathbf{L}$-Ablation2 denotes the MSE between $\mathbf{Z} _ \mathbf{L}$ from the ground truth and the ablated version of "w/ $\mathbf{Z} _ \mathbf{L}$" in Table 7. The results verify that our Reti-Diff can extract high-quality priors directly from the low-quality inputs.

Q4. Convergence analysis

We perform ablation experiments to evaluate convergence across different priors. After confirming network convergence, we record the average batch loss over the subsequent ten epochs under identical training conditions, with the results summarized in the table. The findings reveal that our method, which integrates the Retinex prior, achieves a lower loss compared to approaches utilizing the RGB prior or no prior. These results underscore the superior convergence performance of our Reti-Diff model when guided by the Retinex prior. Notably, as both the image prior and Retinex prior employ customized loss functions, the comparison is restricted to the overlapping components of these functions to ensure fairness.

Q5. Visualization and analysis of Retinex decomposition

• We provide a visualization of the Retinex components of the low-quality input and the ground truth decomposed by D(⋅) in Fig. S6. As shown, the Retinex components extracted from the low-quality input exhibit significant degradation compared to those from the ground truth. However, as demonstrated in the table provided in response to Q3, our Retinex priors show minimal differences from the ground truth priors. This finding confirms that our method does not heavily depend on the quality of the Retinex decomposition.

• Furthermore, as illustrated in Table 7 and Fig. 7, swapping our Retinex priors with those derived from the ground truth results in only marginal improvements. This observation underscores that our Reti-Diff framework is capable of directly extracting high-quality priors from low-quality images.

W1&Q1: Why use LDM rather than Transformer or CNN?

We introduce an LDM to extract compact Retinex priors, aiming to directly estimate high-fidelity priors from low-quality input images. Using an LDM enables more accurate prior estimation compared to other structures, such as CNNs, thereby enhancing the robustness and generalizability of image enhancement in extreme degradation scenarios. To validate this, we replaced our LDM prior estimator with alternative structures of similar computational cost and found that our LDM achieves superior performance.

W2: Clarifying the Retinex operation mechanisms and more experiments.

• Our RGformer uses cross-attention to model the Retinex theory with the purpose of refining and aggregating features modulated by reflectance and illumination priors, ensuring enhancement performance. To achieve this, we originally employed two different attention mechanisms to model the Retinex theory. The first mechanism denoted as A1, involves setting $\mathbf{Q}=\mathbf{W} _ Q \mathbf{F} _ {\mathbf{R}}$, $\mathbf{K}=\mathbf{W} _ K \mathbf{F} _ {\mathbf{R}}$, and $\mathbf{V}=\mathbf{W} _ V \mathbf{F} _ {\mathbf{L}}$. The second mechanism, termed A2 (Ours), is a cross-attention. The multiplication of $\mathbf{Q}\mathbf{K}$ is initially performed as $\widetilde{\text{H}}\widetilde{\text{W}}\times\widetilde{\text{C}}$ @ $\widetilde{\text{C}}\times\widetilde{\text{H}}\widetilde{\text{W}}$ in an element-wise manner, and the resulting coefficient matrix with a size of $\widetilde{\text{H}}\widetilde{\text{W}}\times\widetilde{\text{H}}\widetilde{\text{W}}$ is then element-wise multiplied with $\mathbf{V}$. In this context, A1 explicitly models the Retinex theory, while A2 does so implicitly. As shown in Table, A2 achieves better results, suggesting the effect of the implicit model.
• As demonstrated in Fig. 7, the breakdown ablation and swap experiments confirm that our Retinex prior effectively guides the enhancement process in both illumination restoration and detail enhancement. These results validate that our method successfully models Retinex theory within the latent space.

W3: The reliability of Retinex decomposition

• In the implementation, the decomposition network $D(\cdot)$ consists of three Conv+LeakyReLU layers followed by a Conv+ReLU layer, utilizing the pretrained model released by URetinex [1]. After obtaining the Retinex components from the low-quality input, RLDM is employed to estimate high-quality Retinex priors from these low-quality components, which are subsequently used to guide the enhancement process.

• For fairness, we do not selectively optimize the decomposition model. However, it is evident that a more robust decomposition model yields superior Retinex components, thereby enhancing overall performance. Additionally, the effectiveness of both the prior extraction manner and the prior incorporation strategy has been validated in Tables 6 and 7.

• Beyond these evaluations, we further demonstrate in Table 7 and Fig. 7 that our method maintains strong performance even when the Retinex priors are partially compromised due to extreme environmental conditions. This comprehensive analysis and experimental evidence affirm the reliability and robustness of our method to the Retinex theory.

Q2. Why use Retinex priors rather than the RGB prior?

We propose extracting Retinex priors to guide the enhancement process through a decomposition approach, which is tailored to the IDIR task. We contend that relying solely on priors extracted from the RGB domain struggles to fully represent valuable texture details and correct illumination cues, leading to suboptimal restoration performance. To verify this, we substitute our RLDM for the LDM structure used in DiffIR. In LOL-v2-syn, we observe that the PSNR rises from 24.76 to 26.14 and the SSIM increases from 0.921 to 0.933.

[1] URetinex, CVPR 2022.

Q1: The reasons for the failure cases

The reasons for the failure cases are discussed in Section E in the supplementary material. Here we briefly explain the reasons.

In the first image of Fig. S11, our method fails to distinguish the two regions marked by the dashed box due to the ambiguous boundary of the right region and the intrinsic similarities shared between the two areas. Consequently, our method interprets the two distinct regions as a single object and attempts to merge them. This behavior contrasts with the successful separation of the lower clothing, which exhibits more apparent differences. A similar issue is observed in the second image.

This limitation is not attributable to the Retinex priors or the RGformer, as neither is specifically designed to highlight subtle differences. To address this challenge, future research could explore extracting texture priors from alternative domains, such as the frequency domain. Such priors could complement those in the RGB domain by emphasizing subtle distinctions.

W1. Lacking related works

The recommended method [1], along with two other state-of-the-art approaches [2, 3], is discussed in further detail under "Related Work".

W2. Why use cross-attention?

Our RGformer uses cross-attention to model the Retinex theory with the purpose of refining and aggregating features modulated by reflectance and illumination priors, ensuring enhancement performance. To achieve this, we originally employed two different attention mechanisms to model the Retinex theory. The first mechanism denoted as A1, involves setting $\mathbf{Q}=\mathbf{W} _ Q \mathbf{F} _ {\mathbf{R}}$, $\mathbf{K}=\mathbf{W} _ K \mathbf{F} _ {\mathbf{R}}$, and $\mathbf{V}=\mathbf{W} _ V \mathbf{F} _ {\mathbf{L}}$. The second mechanism, termed A2 (Ours), is a cross-attention. The multiplication of $\mathbf{Q}\mathbf{K}$ is initially performed as $\widetilde{\text{H}}\widetilde{\text{W}}\times\widetilde{\text{C}}$ @ $\widetilde{\text{C}}\times\widetilde{\text{H}}\widetilde{\text{W}}$ in an element-wise manner, and the resulting coefficient matrix with a size of $\widetilde{\text{H}}\widetilde{\text{W}}\times\widetilde{\text{H}}\widetilde{\text{W}}$ is then element-wise multiplied with $\mathbf{V}$. In this context, A1 explicitly models the Retinex theory, while A2 does so implicitly. As shown in Table, A2 achieves better results, suggesting the effect of the implicit model.

W3. Why extract Retinex priors and employ a diffusion model to extract them?

• We propose extracting Retinex priors to guide the enhancement process through a decomposition approach. By separating features into reflectance and illumination components under the guidance of Retinex priors, our method effectively addresses texture loss and illumination distortion, enabling the generation of enhanced images with coherent content.

• We introduce a latent diffusion model (LDM) to extract compact Retinex priors, aiming to directly estimate high-fidelity priors from low-quality input images. Using an LDM enables more accurate prior estimation compared to other structures, such as CNNs, thereby enhancing the robustness and generalizability of image enhancement in extreme degradation scenarios. To validate this, we replaced our LDM prior predictor with alternative structures of similar computational cost and found that our LDM achieves superior performance.

W4. Comparisons of different priors and different prior extraction strategies

• We propose extracting Retinex priors to guide the enhancement process through a decomposition approach, which is tailored to the IDIR task. We contend that relying solely on priors extracted from the RGB domain struggles to fully represent valuable texture details and correct illumination cues, leading to suboptimal restoration performance.

• We provide comprehensive evidence that our Retinex priors facilitate superior image enhancement compared to RGB priors across various conditions, including standard IDIR settings and extreme IDIR scenarios. In the standard setting, as presented in Table 1, we confirm that Reti-Diff with Retinex priors achieves an average performance gain of 20.6% over the state-of-the-art DiffIR method [4], which employs only RGB priors. Additionally, our ablation study (Table 7) reveals that our approach with Retinex priors (denoted as w/ $\mathbf{Z}$) consistently outperforms our method with RGB priors alone (denoted as w/o $\mathbf{Z}$). Similar trends are observed under two extreme IDIR conditions in Table 7.

• Furthermore, we demonstrate that our LDM serves as a more effective predictor than conventional network architectures, as highlighted in response to W3.

W5. Effect of pretrain

We pretrain the RLDM to ensure robust optimization of the latent diffusion model and employ a joint training strategy to enhance restoration quality further. When the pretraining phase is omitted, we observe a decline in performance, underscoring its importance in achieving optimal results.

W6. Typo errors

We have carefully revised the paper and fixed the typos.

[1] CFWD, arXiv 2024.

[2] UNIE, ECCV 2022.

[3] NTD, ACM MM 2023.

[4] DiffIR, ICCV 2023.

W1: More results in real-world datasets

• We present the visualization results of the real-world IDIR task in Fig. S5, where our method demonstrates superior performance in refining texture details and correcting inconsistent illumination, even under complex real-world degradation conditions.

• Additionally, we provide an expanded evaluation with three additional metrics, comparing our model against two state-of-the-art methods. As shown in the table, our approach consistently outperforms the alternatives, securing a leading position in the comparison.

W2: Missing citations

We have discussed the two recommended and relevant papers in the "Related Work".

W3: Computational complexity & runtime

As shown in Fig. 8, our method overcomes the inherent low-speed limitation of diffusion models by utilizing the LDM to estimate highly compact Retinex priors. This approach requires much fewer iterations, specifically set to 4, than other DM-based IDIR solutions. Table 2 further verifies the efficiency of our method, with comparisons of Parameters, MACs, and FPS demonstrating its superiority in terms of storage efficiency, computational cost, and runtime performance.

Q1: How to handle extremely degraded cases?

Our method extracts highly compact priors from low-quality inputs to guide the enhancement process, demonstrating resilience against complex degradation. To validate this, we evaluate our performance on the underwater image enhancement (UIE) task, which is inherently affected by noise, color cast, and distortion. As shown in the table, our method achieves a leading position. Furthermore, we utilize the state-of-the-art image quality assessment method, DA-CLIP [1], to identify the 15 underwater images with the lowest quality and most severe illumination issues, representing the most extreme degradation cases. These form two sub-datasets, LSUI_Extreme and UIEB_Extreme. As evidenced in the table, our method consistently outperforms others, achieving the best performance.

Q2: Strategy for tuning the noise variance

• For fairness, we adopt the default parameter optimization strategy used by DiffIR [2]. As shown in Tables 2–5 of our experiments, Reti-Diff demonstrates the ability to extract valuable priors, ensuring high-quality image restoration across four IDIR tasks.

• We acknowledge that further refining the optimization strategy could enhance the diffusion model's capacity to address noise across varying levels and facilitate the generation of higher-quality enhanced images [3, 4]. We have included added this as a future research direction and will thoroughly investigate how this strategy can contribute to developing a more effective IDIR algorithm.

Q3: Performance in extreme lighting conditions

• Our proposed Retinex prior effectively captures valuable information from the Retinex components extracted from the original degraded input. Even when these components are inadequately captured due to interference from extreme lighting conditions, the compact Retinex priors compensate for this limitation, guiding the decomposition of image features. This approach facilitates the generation of refined images with consistent content and enhances robustness in handling complex degradation scenarios.

• Our method demonstrates superior performance in backlit image enhancement, effectively addressing challenges posed by inconsistent lighting conditions. Besides, we select the top 15 images with the most inconsistent illumination by analyzing their pixel distribution histograms and thus form a dataset with extreme and complex lighting conditions, BAID_Extreme. As shown in the table, our method still achieves a leading place in this task.

[1] DA-CLIP, ICLR 2024.

[2] DiffIR, ICCV 2023.

[3] Common Diffusion Noise Schedules and Sample Steps are Flawed, WACV 2024.

[4] MuLAN, NeurIPS 2024.

W1: Why choose Retinex theory rather than other better models?

• We adopt Retinex theory due to its notable contributions to the field of Illumination Degradation Image Restoration (IDIR). According to recent estimates, over 100 papers in the last three years have leveraged Retinex theory to address IDIR problems, underscoring its prominence in this domain. Ranging from traditional methods [1,2] to advanced architectures like convolutional neural networks [3,4], transformers [5,6], and diffusion models [7-9], the Retinex prior has consistently played an essential role in addressing IDIR challenges and achieving state-of-the-art results.

• Motivation: Different from existing practices of Retinex theory, we first propose to use a latent diffusion model to directly estimate Retinex priors from the low-quality inputs and then use the priors to guide a transformer to enhance degraded images, ensuring texture enhancement and illumination restoration. For its strong generalization, this framework can even function as a plug-and-play method to improve the performance of existing methods by using Retinex priors to guide their enhancement process (See Table 8).

• We evaluate another intrinsic prior based on the Simplified Lambertian Intrinsic Model (SLIM), introduced in TOG 2024 [10], and construct a comparative method termed SLIM by directly replacing our Retinex priors with SLIM priors. Notably, the decomposition network in SLIM is highly complex, comprising 337M parameters, whereas our framework employs a simpler UNet-based decomposition network. As shown in the table, our framework utilizing Retinex priors consistently outperforms the one using SLIM priors, further validating the superiority of our proposed Reti-Diff.

W2: Lack of novelty for simply combining diffusion model and transformer

• Our approach represents a pioneering effort in introducing a latent diffusion model to address the IDIR problem. The real working mechanism of Reti-Diff involves employing a latent diffusion model (RLDM) to estimate Retinex priors from low-quality inputs, which subsequently guides the transformer (RGformer) in enhancing those inputs.

• Our proposed RLDM functions as a plug-and-play module that can be seamlessly integrated into various frameworks. As presented in Table 8, our RLDM can further improve the performance of existing methods. This result highlights the strong generalizability of our Retinex priors and underscores that our framework is not a mere combination of diffusion model and transformer but a novel and effective solution. Importantly, because RLDM estimates vector-level Retinex priors rather than directly restoring degraded images, the computational cost of our method is substantially lower than other diffusion model-based IDIR solutions (see Table 2).

• The carefully tailored design of Reti-Diff enables it to restore degraded images with enhanced details and balanced illumination, even in challenging degradation scenarios. Extensive experiments across four IDIR tasks and three downstream applications consistently highlight the superior performance and efficiency of our approach.

W3: Incorrect evaluation across different conditions

• In total, our experiments abandon the GT-mean strategy in all experiments and only compare with those methods abandoning the GT-mean for a fair comparison.

• In specific tasks, we follow the corresponding common practice, including the use of training and testing sets and the selection of the corresponding cutting-edge methods. For instance, we follow the methodology of Retformer [5] for low-light image enhancement and NU2Net [11] for underwater image enhancement. Similarly, we adopt the training approach of CLIP-LIT [12] for backlit image enhancement and CIDNet [13] for real-world illumination degradation image restoration. These practices ensure that our experiments are conducted fairly and provide reliable comparisons.

• If any experiments remain unclear, please do not hesitate to reach out. We are very happy to clarify and address any misunderstandings.

[1] A weighted variational model for simultaneous reflectance and illumination estimation. CVPR 2016

[2] Structure-revealing low-light image enhancement via robust retinex model. TIP 2018

[3] Uretinex-net. CVPR 2022

[4] CRetinex. IJCV 2024

[5] Retformer. ICCV 2023

[6] RFR. CVPR 2023

[7] Diff-retinex. ICCV 2023

[8] Retinex-diffusion. arXiv 2024

[9] DI-Retinex. arXiv 2024

[10] Colorful Diffuse Intrinsic Image Decomposition in the Wild, TOG 2024

[11] NU2Net, AAAI 2023

[12] CLIP-LIT, ICCV 2023

[13] CIDNet, arXiv 2024

We notice that Reviewer p8Wx has updated the score to 6, indicating that our rebuttal has played a positive role in resolving the reviewer's earlier concerns!

We wish to express our sincere appreciation to the reviewer for recognizing the substantial significance of our Reti-Diff's contribution.

Your acknowledgment holds great importance to us and serves as a meaningful validation of our dedicated efforts to advance this critical area of research.