Self-Supervised High Dynamic Range Imaging with Multi-Exposure Images in Dynamic Scenes

We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses could address the concerns.

1. Limitation of Inputs

We agree with the reviewer's insight on method limitations. Moreover, most existing multi-exposure HDR reconstruction methods (including supervised and self-supervised ones) only focus on removing ghosting artifacts caused by misalignment between inputs, having this limitation as well. This work (SelfHDR) has taken a step toward more realistic self-supervised HDR imaging by deghosting, while our ongoing work is to further address this limitation.

Here we share two possible solutions. On the one hand, we can combine HDR reconstruction with some self-supervised image restoration methods when considering more realistic shooting conditions. For example, there may exist noise in short-exposure images and blur in long-exposure images. In order to achieve a self-supervised algorithm, we can combine HDR reconstruction with some self-supervised denoising and debluring works to process input images for removing degradations.

On the other hand, an adaptive method may need to be explored to select a more appropriate image as a base frame. For example, when a mid-exposure image suffers more severe degradations than others, the method should adaptively take short-exposure or long-exposure images as a new base frame for HDR reconstruction. Thus, it can alleviate the shooting requirements for a mid-exposure image.

We will add this in Sec. E of the appendix in the revision.

2. Pre-Alignment in Structure-Focused Network

The ultimate goal of the structure-focused network is to provide HDR supervision with good structure that is complementary to the color component. It should avoid ghosting results, rather than avoid errors caused by optical flow alignment. Moreover, different from naive merging operation (in Eqn. (5)) for generating color component, learning-based structure-focused network has the ability to handle misalignment problem, including the remaining misalignment after optical flow pre-alignment, as shown in Fig. A (l).

Below we further explain the effect of pre-alignment. On the one hand, significantly incorrect optical flow mainly appears in occlusion and overexposed areas from other frames. These areas have little valuable information, so the results mainly rely on the base frame, and pre-alignment or not has little impact on the results. Thus, it's not bad to perform pre-alignment in this case. On the other hand, when optical flow is roughly correct, pre-alignment reduces the processing burden of the structure-focused network that does not adopt explicit alignment. Thus, it naturally brings improvements.

We additionally test the structure-focused network on 74 training scenes with and without optical flow pre-alignment, respectively. First, the average results are shown in the following table. It can be seen that the pre-alignment manner has a significant improvement on average. Second, we compare the results between the two manners one by one. We find that only in 6 scenes, the results without pre-alignment are more than 0.1dB better than those with pre-alignment on PSNR-$u$. In the other 68 scenes, the pre-alignment manner always gives better or comparable results. Moreover, we also observe the characteristics of these 6 scenes. We find that their alignment is not worse than the other 68 scenes and is moderate in general. The experiments show that the above explanation is reasonable in most scenes.

We will add this in Sec. B of the appendix in the revision.

3. Paper Writing and Misc

Many thanks. In the revision, we will add pointers to the contents of the appendix. We will also briefly introduce HDR-VDP-2 and mark the best results in Tables 3$\sim$6 in bold.

We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses could address the concerns.

1. SelfHDR Sensitivity to Noise

SelfHDR does not specifically deal with the noise problem. If the noise in the multi-exposure image is obvious, the result may also contain noise. Moreover, most existing multi-exposure HDR reconstruction methods (including supervised and self-supervised ones) only focus on removing ghosting artifacts caused by misalignment between inputs, having this limitation as well. SelfHDR has taken a step toward more realistic self-supervised HDR imaging by deghosting, while our ongoing work is to further address this limitation.

Nonetheless, perhaps benefiting from the combination of two complementary supervision, our method still performs better than other self-supervised methods (e.g., FSHDR) when testing noisy images, as shown in Fig. 4(b).

2. SelfHDR Sensitivity to Misalignment

The robustness to misalignment depends on the HDR reconstruction networks we use. In Table 1, we use four HDR reconstruction networks (including CNN-based ones and Transformer-based ones) for experiments. From the results, the Transformer-based networks (i.e., HDR-Transformer and SCTNet) perform better. And the qualitative results testing on out-of-domain datasets (i.e., Sen et al. and Tursun et al. datasets) also show this trend.

In addition, benefiting from the self-supervised properties of SelfHDR, the best way to improve robustness may be to collect as much data as possible for training.

3. Combining with Other Techniques

For noisy images, we have tried to deploy a pre-trained real-world denoising network, but it leads to over-smooth results, losing many details. This may be because the noise model and level of noisy images mismatch with the data (i.e., SIDD dataset) on which the denoising network was trained.

We also agree with the reviewer's insight that HDR reconstruction can be combined with some image restoration methods, especially when considering more realistic shooting conditions. Our ongoing work is aimed at this. Instead of using pre-trained networks, we think that it may be more appropriate to introduce some self-supervised image restoration works to process input images.

For over-exposed regions, we argue that it is difficult to generate realistic details with a hallucination network such as Stable Diffusion, as it can only imagine the information. Actually, the content of over-exposed regions can be preserved well in low-exposure images. It may be more reliable to search for the corresponding information in low-exposure images.

4. Failed Cases

In addition to handling noisy images, here we give some other situations where SelfHDR may fail. First, when long-exposure images suffer from blur, SelfHDR may introduce blurry results. In future work, we can combine some self-supervised image restoration methods to process the blurry images. Second, when the scene irradiance changes drastically in shooting multi-exposure images, SelfHDR may fail, as the constructed color components may be inaccurate. And this situation is also difficult to handle for supervised methods.

Besides, SelfHDR has no specific requirements for scene type and content color. In terms of generalization ability in different scenes, SelfHDR is similar to the corresponding supervised HDR reconstruction networks. And the best way to improve generalization ability is to collect more data and retrain the network.

We will add this and give some examples in Sec. E of the appendix in the revision.

We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses could address the concerns.

1. Performance Upper Limit

The performance upper limit of our work is indeed limited by the upper limit of structure or color supervision. Actually, the performance of existing supervised methods is also limited by the quality of ground truth (GT). Moreover, although the individual structure or color component is not sufficient as supervision, combining the two components as supervision achieves comparable results to supervised methods. As a self-supervised method, the performance has been enough good.

Below we describe in detail the combination of the two complementary components. Two strategies are presented to combine their advantages and discard their disadvantages. First, we use different loss terms for the two supervisions to learn their different information. We adopt L1 loss for the color component and VGG loss for the structure component. The purpose of the former is to learn color information, while the latter is to learn texture information. Second, we design masks to exclude ghosting areas in color components, thus avoiding adverse impacts from these areas.

2. Limited by Alignment Method

The alignment approach mainly influences the color component. Nevertheless, the color component is not very demanding for the alignment approach, as its poorly aligned areas can be further supplemented by the structure component. Furthermore, the development of the current image alignment works is mature, they provide us with enough assistance to achieve comparable performance to supervised methods.

We will add this in Sec. E of the appendix in the revision.

3. Visualizations of Structure and Color Supervision

We respectively visualize a color and structure supervision in Fig. A (g) and (l) of the appendix.

4. Discussion with HDR+

Traditional methods to remove ghosting include rejecting misaligned areas, aligning input images, and using patch-based composite, as mentioned in the second paragraph of Sec. 1 (Introduction). Recently, deghosting methods based on deep learning perform more effectively than traditional ones. Thus, in this work, we adopt learning-based methods to explore self-supervised HDR reconstruction.

In particular, HDR+ takes bursts captured by the same exposure setting as input, while our method adopts multi-exposure images. Unlike multi-exposure images, bursts have limited dynamic range, which makes HDR+ only suitable for scenes with moderate dynamic range, rather than high dynamic range. In addition, HDR+ may only handle small misalignment between the input images, while our method can handle large motions.

We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses could address the concerns.

1. Discussion with Existing Self-Supervised Methods

Existing self-supervised methods (e.g., FSHDR (Prabhakar et al., 2021) and Nazarczuk et al. (Nazarczuk et al., 2022)) construct pseudo-inputs and a corresponding pseudo-target. For these methods, motion and illumination in input images are synthetic, and exhibit gaps with real-world ones.

In this work, we maintain the original inputs and only construct pseudo-targets for them. It does not destroy the real-world motion and illumination in the original inputs. Moreover, we construct two complementary supervisions to address the possible information missing problem in a single pseudo-target.

We will make it clear in the revision.

2. Dependence on Registration Method

The color component indeed relies on the registration method. Nevertheless, it is not very demanding for the registration method, as its poorly aligned areas can be further supplemented by the structure component. Furthermore, the development of the current image registration works is mature, they provide us with enough assistance to achieve comparable performance to supervised methods.

We will add this in Sec. E of the appendix in the revision.

3. More Discussion about Color Components

The color component is generated by weighting the aligned images, as shown in Eqn. (5). When some areas are aligned well, the corresponding areas in the color component are consistent with the ground-truth. When some areas are aligned poorly, the corresponding areas in the color component may have ghosting artifacts. For the former situation, we can directly take these well aligned areas as supervision of HDR reconstruction network. For the latter situation, we construct a mask in Eqn. (14) to exclude these poorly aligned areas when training HDR reconstruction network. And experiments in Table 4 show that the mask indeed plays a positive role.

In addition, we visualize a color component in Fig. A (g) of the appendix. It can be seen that its characteristics are consistent with the phenomenon we describe above.

4. Decomposition Way

Our decomposition way is a focus or emphasis on color and structure relatively, not an absolute separation. Nevertheless, it may have some similarities with Laplace decomposition. The color component is similar to the low-frequency part of Laplace decomposition, while the structure component is similar to the high-frequency part of Laplace decomposition.

5. Mask-Based Weighting

Fig. 1 is a commonly used weighting function for generating an HDR image from static multi-exposure images. In Fig. 1, the abscissa of the function figure is the medium-exposure image value, and the ordinate is the mask value. In other words, the mask value depends on the image value. As long as the images are aligned well, additional artifacts will not be introduced in the color component. Moreover, we visualize a mask in Fig. A (i) of the appendix. It can be seen the mask is highly relevant to the image value, and is not scattered and irregular.