Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Network

Thanks for the valuable comments. If not specifically stated, all experiments are conducted on the real-world image dehazing task with RTTS for space limitation.

W1 & W2: Typo and Misuse of symbols.

We apologize for the errors in our writing and appreciate you pointing out these mistakes in our article. We will correct the issues you mentioned in the final version to improve the readability of the paper.

W3: Lack of clarity in the formulas

We have revised Eq. (5) and Eq. (7) to clearly express the handling of information across different channels. The updated Eq. (5) is as follows:

The updated Eq. (7) is as follows:

We will update this content in the final version.

W4: More Ablation studies.

We tested our CORUN method on the O-HAZE and I-HAZE datasets, using PSNR and SSIM for evaluation. The results, presented in Table A4, illustrate the robustness of CORUN across dehazing datasets.

Q1: Plug-in-play experiment

Thank you for your suggestion. According to our experiments, Colabator exhibits plug-and-play characteristics and brings enhancements across different networks and real-world tasks. In Figure. 1 and Table. 2 of the paper, we validated the effectiveness of Colabator on deep unfolding networks and real-world image dehazing tasks. Here, we integrate our Colabator with more cutting-edge image restoration methods and fine-tune them 5000 iterations in underwater image enhancement and Real robotic laparoscopic hysterectomy desmoke tasks after pretraining.

In the UIE task, all methods pretrained on the UIEB dataset, finetuned and tested on the EUVP trainset’s low-quality part and testset, we find the performance average gain by 11.3% in SwinIR [68], 20.6% by Restormer[69], 15.4% by GRL[70], and 13.0% by AST[71].

In the Desmoke task, all methods pretrained on Desmoke-LAP with synthetic paired data, finetuned and tested on Desmoke-LAP’s real unpaired data, we find the performance average gain by 7.9% in SwinIR [68], 4.9% by Restormer[69], 7.1% by GRL[70], and 6.4% by AST[71].

The gains brought by our Colabator are significant, demonstrating that Colabator possesses plug-and-play capabilities across different networks and tasks.

Q2: Ablation study on trusted weights calculation

We apologize for not explicitly explaining the reason for using weighted summation in the text. We opted for summation rather than multiplication to calculate the Trusted weight, primarily because both the CLIP score and NR-IQAs score need to contribute their respective effects to the weight distribution. This approach also allows for the adjustment of different score ratios for different tasks to achieve more reasonable and better results. If weighted multiplication were used, it would be impossible to control the importance between different components. As shown in the table, we experimentally demonstrated that using the score summation method to calculate the Trusted weight, compared to using score multiplication, results in better final outcomes.

Q3: Ablation study on NR-IQA

Thank you for your question. In fact, we have previously tested many NRIQA methods and found that MUSIQ has the best alignment with the human visual system. MUSIQ evaluates images at multiple scales, comprehensively considering both the technical and aesthetic quality of images, making it broadly applicable to various image quality assessment tasks. We conducted ablation experiments using different NR-IQA methods, and the results show that using MUSIQ in the scenarios presented in this paper can achieve better overall performance.

Q4: Similar portion in RTTS and URHI

Thank you for your question. We used the URHI dataset following the common setting as referenced in works like PSD, so the issue does not cause unfairness in the comparative experiments. However, this concern is worth considering. Therefore, we attempted to remove images from the URHI dataset that overlap more than 60% with the RTTS dataset, ultimately retaining 3728 images for the second phase of fine-tuning CORUN. The final results are shown below. From the results, it can be seen that our method still achieves excellent performance even after using the cleaned dataset.

We found that the NIMA score actually increased, which we believe is due to the removal of some low-quality data during the data cleaning process.

[68]SwinIR, ICCVW, 2021

[69]Restormer, CVPR, 2022

[70]GRL, CVPR, 2023

[71]AST, CVPR, 2024

Thanks for the valuable comments. If not specifically stated, all experiments are conducted on the real-world image dehazing task with RTTS for space limitation.

W1: Unclear derivation of equation

For space limitation, we only show the detailed derivation of Eq. (7) and the Eq. (10) can be derived similarly.

Having gotten Eqs. (5) and (6), the solution of $\hat{\mathbf{T}}$ can be formulated according to the proximal gradient algorithm:

Then we obtain the partial derivative:

Let the partial derivative be equal to zero, we archieve the closed-form solution for $\hat{\mathbf{T}}$ in Eq. (7).

Similarly, the solution of $\hat{\mathbf{J}}$ can be formulated as

The corresponding partial derivative is

The closed-form solution for $\hat{\mathbf{J}}$ is presented in Eq. (7) when let the partial derivative be equal to zero.

W2 & Q1 & Q3: Details of using DA-CLIP and reasons for partition

Our method for using DA-CLIP as a haze density estimator involves calculating the cosine similarity between the encoded input image and the encoded input text. The specific formula is

The text we used is "hazy" from the DA-CLIP provided in the text list.

(Q1) As Density Loss, we normalize the result of this formula from 0 to 1, the higher result means higher density.

(W2) As Haze Density Evaluator for Colabator, the $norm$ in eq. (12) means $1 - normalized(\mathcal{D}(\mathbf{S}^{R}\_{\widetilde{HQ}}))$ which has the same value as (1 - density loss), the higher result means lower density.

(Q3) We assess the image both globally and locally. We use global assessment to determine whether the overall quality of the image has improved relative to the images in the existing optimal label pool, updating the pseudo-labels. We also use block-based processing to obtain independent haze density and image quality scores for each region of the image. This allows us to assign appropriate confidence weights to the pseudo-labels of each block. This approach enables our network to maximize the use of high-quality dehazed parts of the image while avoiding being misled by poorly dehazed or low-quality parts. By focusing on both global and local aspects of the image, our method achieves better performance.

(W2) The patch size N is set to 32 to balance performance and efficiency. We will add this.

W3: Ablation study on CPMM

After removing all CPMM modules, our method retained only 8 learnable parameters. As shown in the table, removing CPMM brings a significant performance decline both in the FADE metric, which measures haze removal capability, and the NIMA and BRISQUE metrics, which measure the quality of generated results. However, compared to the original hazy images, the CPMM-removed CORUN still possesses a certain haze removal capability.

In the Figure.A3, we provide a case of generated result examples. The images include, respectively, the input image, the result with the CPMM module, the result without the CPMM module, and the result without the CPMM module but with manually increased exposure. From our analysis, we can conclude that removing the CPMM module severely affects the flexibility of CORUN, causing the generated results to lose a significant amount of detail, especially in dark areas, and leading to a substantial decrease in haze removal capability.

Q2: Ablation study on strong data augmentation before student network

We conducted ablation experiments on the data augmentation strategies used during the fine-tuning phase. In the table below, we provide the quantitative results obtained after disabling strong data augmentation for the student network input. From the results, it can be seen that the strong data augmentation strategy for the student network indeed brings some performance gains, but even without strong data augmentation, Colabator still provides a considerable performance improvement to the network. Therefore, we can conclude that the performance gain during the fine-tuning phase does not entirely come from the strong data augmentation strategy.

Thanks for the valuable comments. If not stated, all experiments are conducted on the RTTS dataset.

*Experiment results are placed in the attached PDF for space limitations*

W1: Writing and formula

• The redundant comma will be removed.
• The simplified $P$ in eq.(2) differs from the $P$ in line 110. In line 111, the original atmospheric scattering model $P=J\cdot T+A\cdot (I-T)$ is divided by the atmospheric light A, resulting in $P/A=J/A\cdot T+I-T$. We then simplified $P/A$ and $J/A$ to $P$ and $J$, leading to $P=J\cdot T+I-T$. We will reorganize it.

This simplification aims to help integrate atmospheric light $A$, which is closely related to image features, with the scene for better results. Experiments in Table A1 show that this improves performance by 10.7%.

• We will correct the notion: pseudo-dehazed image ${\mathbf{P}^{R}\_{\widetilde{HQ}}}\_{i}$ and previous pseudo-label ${\mathbf{P}^{R}\_{Pse}}\_{i}$.
• In eqs. (15) and (16), we introduced two loss functions for different phases. Eq. (15) includes reconstruction loss and perceptual loss, while eq. (16) substitutes the perceptual loss with a stronger supervised contrastive perceptual loss. We will rename eq. (15) and eq. (16) to Reconstruction-Perceptual Loss and Reconstruction-Contrastive Perceptual Loss.

W2: Unfair in data augmentation and generation pipeline

To ensure a fair comparison with RIDCP, we follow its setup by using the same data pipeline (generation + augmentation) and dataset.

To fully show our results, we provide results under two extra settings: 1) We remove the augmentation pipeline, 2) We further replace RIDCP's dataset and generation pipeline with the OTS dataset (commonly used in previous methods).

As shown in Table A2, the results verify our method still achieves a leading place under the two settings compared with existing methods.

W3: Effect of Colabator and results on full-reference

• We integrate our Colabator with cutting-edge dehazing methods and test them in RTTS. In Table A3, we find evident performance gains brought by Colabator: 23.2% in C2PNet, 22.0% in FFA-Net, 31.7% in GDN.

• We further evaluate our methods on the full-reference setting with O-HAZE and I-HAZE. Because we have paired data for training, we drop Colabator and only train our CORUN following the practice of MB-TaylorFormer. Table A4 shows that our method achieves leading performance with a big margin on both datasets.

Q1: Essential advantages

Previous state-of-the-art dehazing methods, such as C2PNet[60], fail to generalize to broader real-world scenarios like the RTTS dataset when trained on synthetic data. RIDCP, with its well-designed data synthesis pipeline, generalizes better to RTTS, while is limited by depth estimation errors, causing unrealistic haze residuals.

Our CORUN is specifically designed for real-world scenarios. It integrates deep unfolding networks and proximal gradient descent algorithms to combine physical priors with network learning. Therefore, as shown in Table A5, our method achieves state-of-the-art real-world dehazing results.

Q2: Haze residues and medium/dense haze removal

Apart from the problem of the synthetic training data, i.e., the inaccurate depth estimation and the hazy residue from the ground truth, the reasons for failing to completely remove the haze lie in our semi-supervised setting with incomplete supervision. In Figs. 4, it is a common problem that also exists in other comparative methods.

However, as shown in Fig. A1, our method has a better dehazing effect compared to existing methods, which is attributed to the joint estimation capacity of CORUN and the generalizability of Colabator.

Besides, we randomly select 300 data in RTTS and invite 10 naive observers to score them based on how hazy they are. Then we select the first 100 images as dense hazy data and the 101st-200th images as medium-density hazy data according to the scores. Table A6 shows our superiority in removing medium-density and dense haze.

Q3: Loss functions overlap

The two losses, i.e., eq.15 and eq.16, are employed in different phases with different inputs and their specific parameters in eq.20 and eq.21.

In the pre-training phase (eq.20), we use eq.16 with contrastive learning for constraints, using synthetic data. During the fine-tuning phase (eq.21), we employ both eq.15 and eq.16. Here, eq.15 uses synthetic data to ensure stability in the fine-tuning process, while eq.16, which incorporates contrastive perceptual loss, uses real-world data to provide stronger constraints and align the model training with real-world conditions. As shown in Table A7, our ablation verifies that combining these two losses enhances the model's performance. We will reorganize the content.

Q4: Robustness of DA-CLIP

We evaluated the robustness of DA-CLIP by randomly selecting 300 images and assessing their density score. We invited 10 observers to rate haze density in these images. In Table A8, our analysis revealed that while DA-CLIP's results generally align with human ratings (with discrepancies under 0.2), significant differences occur in certain situations. Notable discrepancies (over 0.5) were observed in cases involving 1) images with predominant sky coverage on cloudy days, 2) color distortion or severe faults, and 3) insufficient lighting at night (examples shown in Fig. A2).

To address DA-CLIP's limitations in handling complex scenes, we designed a pseudo-label rating strategy. Along with DA-CLIP, we used NR-IQA to evaluate potential pseudo-labels. A pseudo-label is updated only if its scores from both DA-CLIP and NR-IQA surpass the latest label in our optimal label pool. Locally, we partitioned images and used DA-CLIP and MUSIQ to assign weights, encouraging the network to focus on higher-quality regions with better dehazing effects. Table A9 verifies the effect of our pseudo-label rating strategy.

Thanks for the valuable comments. If not specifically stated, all experiments are conducted on the real-world image dehazing task with RTTS for space limitation.

W1: Contribution

Our work goes beyond simply combining existing methods. We introduce the Cooperative Unfolding Network (CORUN), which iteratively optimizes transmission maps and scene information with theoretical guarantees. We also present Colabator, a semi-supervised fine-tuning framework that requires no extra computational cost and adapts well to real-world scenarios with just a few iterations.

Specifically, compared to previous methods, our innovations are as follows:

• Methods by Mondal et al.[64], Im et al.[65], and Ren et al.[66] use common networks to estimate atmospheric light and transmission maps through the atmospheric scattering model but overlook other complex scene information. Zhang et al.[27] estimate the transmission map and haze features from hazy images but ignore the atmospheric model during reconstruction, limiting generalization.

  In contrast, our method mathematically models the relationship between the image scene and transmission map, utilizing a deep unfolding network based on proximal gradient descent for progressive dehazing. This enhances interpretability and detail restoration, resulting in more realistic images. To verify this, we incorporate our joint estimation component into the three open-sourced methods and observe evident performance gain.

  Datasets	Metrics	Ren[66]	Ren+CORUN	Mondal[64]	Mondal+CORUN	Im[65]	Im+CORUN	CORUN(Ours)
  O-HAZE	PSNR↑	15.72	19.91	16.37	18.85	18.85	21.36	25.66
  	SSIM↑	0.703	0.753	0.717	0.750	0.776	0.793	0.847
  I-HAZE	PSNR↑	14.49	18.83	15.52	18.27	17.93	20.15	23.90
  	SSIM↑	0.710	0.760	0.721	0.763	0.752	0.785	0.868

• PDN[50] introduced a deep unfolding network for image dehazing but couldn't effectively use complementary information between the scene and transmission map, leading to sub-optimal results with real-world data. In contrast, our new dual proximal gradient descent cooperative unfolding network considers both atmospheric light and image information, enhancing robustness and flexibility. We integrated our optimization mode into PDN[50], resulting in improved performance.

  Datasets	Metrics	PDN[50]	PDN+CORUN	DGUN[33]	DGUN+CORUN’s model	DGUN+CORUN	CORUN(Ours)
  O-HAZE	PSNR↑	16.37	19.02	18.86	20.16	22.74	25.66
  	SSIM↑	0.717	0.763	0.756	0.788	0.805	0.847
  I-HAZE	PSNR↑	15.86	18.80	18.27	19.82	22.63	23.90
  	SSIM↑	0.712	0.757	0.732	0.763	0.791	0.868

• DGUN models the relationship between degraded and clean images but ignores the interaction between the scene and transmission map in dehazing. Additionally, DGUN[33]'s unfolding strategy overlooks complex hazy conditions in real-world scenarios. Our CORUN addresses this with CPMM modules and a proposed global coherence loss. Replacing DGUN[33]'s dehazing model with ours and integrating our unfolding framework into DGUN[33] both improve performance.

• Mean-teacher, commonly used in high-level vision tasks, commonly utilizes uncertainty to constrain pseudo-labels which isn't suitable for low-level vision tasks. Some attempts have been made to adapt it for low-level tasks using data augmentation, but these still face issues with erroneous pseudo-labels and overfitting. Colabator improves this by using CLIP and NR-IQA to weight pseudo-label regions, guiding the student network to focus on high-quality areas. It also evaluates pseudo-label quality globally to refine the optimal label pool iteratively, enhancing performance over Mean-teacher. Comparison results can be found in Table A10.

• Totally, the major difference and technical novelty in comparison to the combination of existing methods above is:

  • From a mathematical optimization perspective, we jointly model the transmission map and image scene based on the atmospheric scattering model, creating a novel optimization function that effectively handles complex real-world haze scenarios.
  • Our unfolding network, CORUN, is tailored to this optimization formula, integrating scene and haze feature commonalities. Using the proximal gradient descent algorithm, CORUN reconstructs high-quality dehazed images progressively from coarse to fine.
  • Our Colabator architecture is plug-and-play, designed for low-level vision tasks like real-world image dehazing. It evaluates pseudo-labels globally and locally, helping the student network learn from high-quality regions while minimizing errors. It also dynamically maintains an optimal label pool to ensure pseudo-labels stay globally optimal.

• Therefore, our work is not a mere assembly of existing methods but an effective framework proposed specifically based on the characteristics and challenges of real-world image dehazing.

W2: Detailed ablation study

In the paper, we presented Colabator, along with ablation experiments on the Optimal Label Pool, Trusted Weights, Iterative Mean-Teacher, and the number of layers in the deep unfolding network. We have supplemented additional ablation experiments in Global Rebuttal‘s pdf supplement, which will be included in the supplementary materials of the final version.

The results of our CORUN and Colabator on various datasets, data processing methods, and types of Image Quality Assessment (IQA) are presented in Tables A2, A4, A6, A17. Tables A3, A14 demonstrate the generalizability of Colabator across different tasks and networks. Tables A1, A12 detail the ablation studies of CORUN’s components, while Table A7 shows the ablation of loss functions. Table A8 assesses the robustness of DA-CLIP. Tables A9, A11, A13, A15, A16 demonstrate the ablations of Colabator’s components and functions.

[64]Mondal et al., CVPRW, 2018

[65]Im et al., CIKM, 2023

[66]Ren et al., ECCV, 2016

[67]Zhang et al., TCSVT, 2019

I appreciate the authors' rebuttal, which has addressed most of my concerns. One additional concern I have after reading the rebuttal is the authors have made comparisons against other methods on I-HAZE, O-HAZE, but not on RTTS, a benchmark the authors have used in the main paper. I believe this has to do with the other reviewer's recommendation. It would be great if the authors could make comparisons on RTTS as well in the final version. Also, the authors have presented a lot of new experimental results during the rebuttal, requiring a major change to the paper. I'm ok with the acceptance and hence raise the rating, given that the authors make these major changes to the final version of the paper, along with new comparisons on RTTS as well.