Exposure Bracketing Is All You Need For A High-Quality Image

We thank you for your valuable comments and suggestions. We appreciate your questions and hope our responses could address your concerns.

1. Question: The use of RIFE

Response: We agree with your point that the synthetic blur still has a gap with the real-world one due to the limitations of the frame interpolation model. In this work, the proposed self-supervised real-image adaptation method is to alleviate this gap. Besides, recent interpolation models are more capable of dealing with large motion [1] and complex textures [2]. We believe this problem will also be alleviated with the advancement of interpolation models. We have added them in Appendix A.2.

[1] Jain, Siddhant and Watson, Daniel and Tabellion, Eric and Poole, Ben and Kontkanen, Janne and others. Video interpolation with diffusion models. In CVPR, 2024.

[2] Zhong, Zhihang and Krishnan, Gurunandan and Sun, Xiao and Qiao, Yu and Ma, Sizhuo and Wang, Jian. Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation. In ECCV, 2024.

2. Question: Comparison between self-supervised and supervised manners

Response: For supervised methods, large amounts of high-quality paired data are critical to model performance. However, it is very expensive or even impossible to obtain a large amount of paired data. For self-supervised methods, since ground-truth is not required, the available data is much larger. Benefiting from this, self-supervised methods may further enhance performance by expanding the data scale. And with the improvement of self-supervised algorithms, we believe that they can approach or reach the performance of supervised methods. Thus, it is necessary and valuable to explore self-supervised methods in the current era.

3. Question: Comparison with AI-ISP

Response: As far as we know, there are almost no suitable open-source AI-ISPs for comparison. Here we indirectly demonstrate our advantages by conducting an ablation study that compares the joint processing and progressive processing manners on BracketIRE task on synthetic dataset.

We mark our multi-task joint processing way as "Denoising&Deblurring&HDR". In the ablation study, we decompose the whole task with three steps: (1) first denoising, (2) then deblurring, and (3) finally HDR reconstruction. We mark the way as "Denoising + Deblurring + HDR". During training, we construct data pairs and modify our TMRNet as the specialized network for each step. The inputs of each step are all multi-exposure images concatenated together, which aim to exploit the complementarity of multi-exposure images in denoising, deblurring, and HDR reconstruction task, respectively. During inference, we sequentially cascade the networks at all steps to test. The results are shown in the following table. It can be seen that step-by-step processing is inferior to joint processing.

Actually, during step-by-step processing, the denoising model performs well. The main reason for the unsatisfactory performance is that the deblurring model has a limited effect when dealing with the severe blur in the long-exposure image. It prevents the HDR reconstruction model from working well. Specifically, in the training phase, the input multi-exposure images of "HDR" model are blur-free and noise-free. However, in the testing phase, there may still be some blur remaining in the input of "HDR" model, due to the limited capabilities of "Deblurring" models. Thus, a data gap between training and testing appears in "HDR" model, and it hurts the model performance. In contrast, joint processing can avoid this problem.

Besides, joint processing only produces one model for deploying. It can simplify the complexity of the entire imaging system and make it easier to deploy in actual scenarios. Benefiting from this, the way of joint processing is also being pursued by some mobile phone manufacturers, as far as we know.

We have added them in Appendix E.

4. Question: Would it be better to move the flowchart of the synthetic paired dataset from the supplementary material to the manuscript?

Response: Yes, we agree with that. Due to space limitations, we moved it to the supplementary material when we submitted the review version. In the camera-ready version, we will try to move it back to the main text.

I sincerely appreciate the authors' response.

(Discussion:) I believe that simulating realistic motion blur using frame interpolation can only serve as a temporary solution. Modeling continuous motion through inherently inaccurate discrete data has inevitable limitations. It might be more reasonable to generate blur kernels based on motion estimation rather than directly relying on frame interpolation. Perhaps data production schemes based on computer graphics are more promising for the future.

By the way, I find the academic community's current focus on works modifying data pipelines somewhat strange. When it comes to deploying raw image processing tasks, the limited computational resources and runtime memory of edge devices (e.g., smartphones) mean that almost no frameworks proposed in published research can be directly applied in practice. Therefore, the real question should always be: What effective information increment does this work bring to the community?—not just What novel ideas does this work include? I have encountered far too many low-level vision papers with "novel" ideas that ultimately fail to provide practical value, serving little purpose beyond helping students fulfill graduation requirements.

For a practical application-oriented study like this, I believe the most significant contribution lies in demonstrating the advantages of exposure bracketing for multi-task scenarios and offering a robust, reproducible framework. In my view, this is the strongest aspect of the paper and the main reason I am inclined to rate it highly.

Overall, my experience in the industry allows me to resonate with the authors' perspective. Therefore, I am leaning towards keeping the original rating.

4. Question: Effectiveness of TMRNet on other datasets

Response: We conduct experiments on Kalantari [3] dataset for HDR image reconstruction. We compare our TMRNet with recent HDR reconstruction methods. The following results show that TMRNet achieves the best results.

We also conduct experiments on BurstSR [4] dataset for burst image super-resolution. We compare our TMRNet with recent burst super-resolution methods. The following results show that TMRNet still achieves the best results.

We have added them in Appendix C.

[3] Deep High Dynamic Range Imaging of Dynamic Scenes. ACM TOG 2017.

[4] Deep Burst Super-Resolution. CVPR 2021.

5. Question: Effectiveness of self-supervised loss on other datasets

Response: We conduct an experiment on burst image super-resolution dataset [4], which is the only commonly used multi-image processing dataset with both synthetic and real-world data. We first pre-train our TMRNet on synthetic data, and then use our self-supervised loss to fine-tune it on real-world training dataset. The following results show that our self-supervised loss brings 0.87 dB PSNR gain on real-world testing dataset, demonstrating its effectiveness.

We have added it in Appendix C.

We thank you for your valuable comments and suggestions. We appreciate your questions and hope our responses could address your concerns.

1. Question: Problem setting and motivation

Response: We kindly remind the reviewers that our work does not simply train a model using data containing four degradations. It is problem-oriented and has clear motivations.

Our work aims to process multiple images with different exposures to generate a single high-quality image. Multi-exposure images, especially those taken in low-light environments, inevitably possess noise, blur, and low dynamic range. Meanwhile, the magnitude and parameters of these degradations vary across multi-exposure images, making the multi-exposure images complementary in removing each degradation. We propose to leverage the complementarity of multi-exposure images to integrate the four problems (i.e., denoising, deblurring, high dynamic range reconstruction, and super-resolution) into a unified framework, generating a noise-free, blur-free, high dynamic range (HDR), and high-resolution image. To the best of our knowledge, this is the first work to achieve multi-exposure image processing from this perspective. Additionally, compared with previous multi-image processing work, the degradations we consider are more comprehensive and more consistent with the actual situation.

The work itself is not directly oriented to application, but expects to utilize the complementarity of multi-exposure images on multiple tasks to solve multiple problems together. Even in terms of application, HDR imaging at night is only a single significant one. Our work also has the potential to be applied in low-light denoising and motion deblurring.

2. Question: Novelty of the proposed TMRNet

Response: Conv-LSTM and the existing temporal sequence processing networks (e.g., RBSR we compared) generally use the same module parameters when processing different sequences, since these sequences are of similar type (e.g., degradation). In our work, TMRNet suggests learning different parameters for each frame sequence, as frames in multi-exposure images suffer different degradations. In Figure 1, modules with different colors have different parameters. Compared to the baseline network (i.e., RBSR) in Figure 1 (a), TMRNet can achieve a 0.25dB PSNR improvement.

Our TMRNet is similar to [1] only in naming, while the contributions and main focus are completely different. In [1], 'temporal modulation' means to modulate the feature according to time t, thus interpolating an intermediate frame corresponding to t. However, the module parameters in [1] that process different frames are still shared. In our TMRNet, 'temporal modulation' means to learn specific parameters for each frame, i.e., the module parameters that process different frames are not fully shared. We also take [1] to experiment on our synthetic dataset, and its PSNR result is 36.14 dB, which is far behind our outcome of 39.35 dB.

In general, TMRNet is problem-oriented with clear motivation, and has novelty.

[1] Temporal Modulation Network for Controllable Space-Time Video Super-Resolution. CVPR 2021.

3. Question: Novelty of the proposed self-supervised loss

Response: Our self-supervised real-image adaptation loss contains two terms, i.e., temporally self-supervised loss and EMA regularization loss. The former plays a major role and is significantly different from [2]; the latter plays a supporting role and can be considered similar to [2]. As shown in Figure 2 (a) and Equation (8), temporally self-supervised loss uses results with more input frames to supervise results with fewer input frames. This idea exploits the temporal characteristics of multi-exposure image processing, i.e., the more input frames, the better the result. As far as we know, this idea has not appeared in related works and is novel.

[2] Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning. CVPR 2021.

Dear Reviewer QkhS,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. With the discussion period deadline approaching (in 42 hours), we kindly request your feedback if there are any remaining concerns or suggestions you would like us to address.

Thank you once again for your valuable contributions to enhancing the quality of our work.

Authors of Submission 4958

We thank you for your valuable comments and suggestions. We appreciate your questions and hope our responses could address your concerns.

1. Question: Add the results without fine-tuning TMRNet on real-world data to Table 2

Response: Thanks for the suggestion. The first line in Table J shows the results on real-world data without fine-tuning TMRNet. We have added them in Table 2 and provided a note in the revision.

2. Question: Would introducing more frames with over-exposure or large blur make the restoration process more challenging?

Response: 'More frames result in better performance' is just a conclusion for Table 3, which is the case at least within 5 frames. Moreover, in our data, the frame with the highest exposure is already highly blurry and overexposed, as shown in the lower left side of Figure 4.

More generally, we think that as the number of frames increases, the worst case is that the network does not extract useful information from the increased frame. In other words, adding frames does not lead to the worse results, but leads to the similar or better ones. For each scene in our dataset, we have compared the results when taking the varying number of frames as input. As frames increase, the results do not worsen in almost all scenes. This observation supports our view to some extent.

We have added these in Appendix G.

3. Question: Task setting, degradation setting, and contribution point statement

Response: The main difference between our works and previous works is that previous works only achieved limited tasks (see Table 1), and did not bring out the potential of multi-exposure images. For example, [1] and [2] only consider denoising and deblurring tasks, but do not involve HDR and super-resolution tasks. Our work points out the complementarity of multi-exposure images on multiple tasks (i.e., denoising, deblurring, HDR reconstruction, and SR), and leverages this to jointly address these tasks.

Moreover, most HDR reconstruction works are only verified on clean and sharp inputs, i.e., they ignore noise and blur degradations in multi-exposure images. Although recent HDR reconstruction works have considered noise, almost no work has considered blur in long-exposure images as far as we know. Our work takes degradation factors into account more realistically and comprehensively, incorporating both noise and blur degradation. Furthermore, even in [1] and [2] that focus on denoising and deblurring tasks (but do not involve HDR reconstruction), they only consider blur from camera shake, and do not consider blur from object movement. Our degradation contains both camera motion blur and object motion blur (see the first line in Figure H).

For the first contribution point statement, we have realized that the 'unify image restoration and enhancement tasks' statement may be somewhat exaggerated. We have changed it to 'We propose to utilize exposure bracketing photography to get a high-quality (i.e., noise-free, blur-free, high dynamic range, and high-resolution) image by combining image denoising, deblurring, high dynamic range reconstruction, and super-resolution tasks.', which may be more in line with the practical contribution and application of this work. The title and the related statements in the paper have also been revised.

[1] Low-light image restoration with short- and long-exposure raw pairs.

[2] Digital gimbal: End-to-end deep image stabilization with learnable exposure times.

Dear Reviewer GQoP,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. With the discussion period deadline approaching (in 42 hours), we kindly request your feedback if there are any remaining concerns or suggestions you would like us to address.

Thank you once again for your valuable contributions to enhancing the quality of our work.

Authors of Submission 4958

6. Question: Why do the authors add SR for BracketIRE+? To match the title as restoration tasks or to show the ability on low resolution images?

Response: The main purpose of our work is to explore the complementarity and usability of multi-exposure images in performing multiple tasks. According to [2], the sub-pixel shift between multiple images caused by camera shake or motion is conducive to multi-frame super-resolution. Thus, we add super-resolution task for BracketIRE+, showing the ability of multi-exposure images in super-resolution task.

In addition, in practical applications, it is possible to demand higher-resolution images while performing noise-free, blur-free, and HDR imaging. The task setting of BracketIRE+ is also to meet this actual need.

[2] Wronski, Bartlomiej, et al. "Handheld multi-frame super-resolution." ACM Transactions on Graphics (ToG) 38.4 (2019): 1-18.

7. Question: Are code and dataset publicly available?

Response: The main code, pre-trained models, and all datasets are already publicly available. Due to the double-blind policy, we did not add them to the paper. We will add this link to the camera-ready version.

We thank you for your valuable comments and suggestions. We appreciate your questions and hope our responses could address your concerns.

1. Question: Comparision with all-in-one methods

Response: All-in-one models mean that the models can process images with different degradations. There are three main differences between them and our work. First, all-in-one models generally input a single image, and output a single image. Our model inputs multi-exposure images, and outputs a single image. Second, in all-in-one models, the degradation type across input samples can be different. In our model, the degradation across input samples is basically consistent, and the degradation is different between multiple images within an input. Third, all-in-one models utilize the capabilities of the model itself to achieve multiple tasks. Our model can additionally exploit the complementarity of the input images to achieve multiple tasks.

Perceive-IR is not publicly available, and we have conducted an experiment using AdaIR. The original AdaIR model can only input one image. For a fair comparison, we concatenate multi-exposure images aligned by an optical flow network together as the input of AdaIR. Its PSNR result is 38.06 dB, which is lower than our 39.35 dB. We argue that the main reason for AdaIR’s poor performance is that it is not specifically designed for multi-image processing. In contrast, the methods compared in Table 2 are all specific multi-image processing methods.

2. Question: Computational cost

Response: We suggest inference time as the main reference for computational cost comparison, as the testing time is more important in practical applications. Our method has a similar inference time with RBSR, and a shorter time than recent state-of-the-art ones, i.e., BIPNet, Burstormer, HDR-Transformer, SCTNet, and Kim et al.

In the recent state-of-the-art methods, BIPNet, RBSR, and our TMRNet are based on the convolutional neural network (CNN), and Burstormer, HDR-Transformer, SCTNet, and Kim et al. are based on Transformer. TMRNet has similar #FLOPs to RBSR and lower #FLOPs than BIPNet, while TMRNet has higher #FLOPs than these Transformer-based methods. We think this is acceptable and understandable for two reasons. First, although Transformer-based methods have an advantage in #FLOPs, they bring higher inference time and it is more difficult for them to deploy into embedded chips for practical application. Second, the main idea of TMRNet is to assign specific parameters for each frame while sharing some parameters. We implement this idea based on a more recent CNN-based method (i.e., RBSR), and the basic modules only adopt simple residual blocks. For TMRNet, the basic modules can be easily replaced with #FLOPs-friendly modules, which has great potential for #FLOPs reduction. We plan to experiment with it and provide a lightweight TMRNet in the next version.

We will add this in the revision.

3. Question: What is the type of CMOS for those videos? Is it the same with the setting of UPI? Would the RIFE affect data distribution of the original videos?

Response: The original HDR videos [1] in the synthetic dataset are shot by the Alexa camera, a CMOS sensor based motion picture camera made by Arri. We do not use the same parameter setting with UPI, but use the parameter setting from Alexa camera to convert RGB into raw images. We will add the details in the revision and make the data synthesis code public after the paper is accepted.

Moreover, RIFE is only used to interpolate between two frames, and it does not affect the data distribution. From visual observation of interpolation results, it also supports this point.

[1] Froehlich, Jan, et al. "Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays." Digital photography X. Vol. 9023. SPIE, 2014.

4. Question: Do the authors perform packing operation for raw data?

Response: Yes, we perform a packing operation for the raw data before feeding it into the model. We will add the details in the revision.

5. Question: Why do the authors exclude 10 and 4 invalid pixels around the original input image? Do the proposed method deal with marginal areas?

Response: The surrounding 10 pixels of the original HDR video [1] are all 0 values. Thus, the surrounding 10 and 4 pixels of the synthetic input image are all 0 values for BracketIRE and BracketIRE+ task, respectively. In practice, this situation hardly occurs, so we exclude them for evaluation.

The model can deal with marginal areas. Actually, in the early version of our paper, we used all pixels for evaluation. After the suggestions from peers, we changed the evaluation way, as the current way is more in line with the actual situation.

We sincerely appreciate your feedback and thoughtful suggestions. Addressing your concerns has been an honor and a valuable opportunity to improve our work.

We have revised the paper thoroughly.

• Question 1 was added to Appendix D.
• Question 2 was added to Appendix B.
• Question 3 was added to Appendix A.1 and Appendix A.2.
• Question 4 was added to Section 5.1.
• Question 5 was added to Appendix A.4.
• The responses for Question 6 and Question 7 were marked with blue in Introduction (Lines 94-96) and Abstract (Lines 25-27) , respectively.
• Moreover, the suggested revisions from other reviewers were also added to the paper.

We hope the revised paper can sufficiently address your concerns and provide the necessary support for an improved rating. Let me know if you'd like further refinement or adjustments.