Learning Gain Map for Inverse Tone Mapping

3. Concerns on the metrics of PSNR-NGM and PSNR-mu

First, we need to clarify that PSNR-NGM is NOT a metric for evaluating HDR images. It is only used in the ablation experiments on our network, evaluating the quality of the normalized GM.

Second, we follow the previous work [7] using PSNR-L in the linear domain, but employ PSNR-PQ to evaluate visual similarity instead of PSNR-mu, which plays a similar role in compressing HDR images. The main reason is that our work is closer to ITM tasks, so applying PQ-metric is more suitable for practical applications.

Third, to further address the reviewer's concerns, we further provide the experimental results of PSNR-mu, PSNR-PQ, and PSNR-L on real-world datasets in the table below, and our method still achieves superior performance.

[7] Chen, Xiangyu, et al. "Hdrunet: Single image hdr reconstruction with denoising and dequantization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

4. Explanation of the normalization of $Q_{max}$

The $Q_{max}$ in the real-world dataset is in the range of [0,5], and the $I_{GM}$ is also in the range of [0,5]. The $I_{GM}$ is divided by the global maximum 5 for normalization, differing from the $I_{NGM}$ normalized by the respective maximum of each data. Therefore, $I_{GM}$ will be normalized to [0,1], making a uniform and smooth convergence.

5. Concerns on the upsampling of $I_{GM}$

There might be some misunderstanding here. The resolution of GM in the real world is reduced in most cases, as the downsampling is recognized by the standards [1] [2] [3] to save the bandwidth. Therefore, our method aims to learn the Groud-Turth GM, not the Interpolated GM.

Moreover, we perform ablation experiments on the resolution of GM in Table 5 in our paper, demonstrating that GMNet can achieve superior performance no matter with or without interpolation.

We appreciate the reviewer's valuable comments and suggestions, and we hope our responses could address the concerns.

1. Similarity to the transfer function

We respectfully cannot agree. GM is NOT similar to the transfer function, which typically maps the input to the output with a learned function (implemented by neural networks nowadays). Suppose $f_{\phi}(\cdot)$ is a neural network, in the HDR task, it is implemented by $L_{HDR}=f_{\phi}(L_{SDR})$, where $\phi$ denotes the learned parameters of the network. In contrast, GM is an image-like auxiliary data that records pixel-wise dynamic range information, which is independently collected along with the SDR image and is implemented by $L_{HDR}=L_{GM}\odot L_{SDR}$, differing from the transfer function in essence. The proposed GMNet is inspired by the latest HDR data format [1] [2] [3], and the novelty of exploring this new format has been uniformly recognized by the other three reviewers.

[1] ISO. "Gain map metadata for image conversion." https://www.iso.org/standard/86775.html, 2024.

[2] Adobe. "Gainmap specification." https://helpx.adobe.com/camera-raw/using/gain-map.html, 2024.

[3] Google. "Ultra-hdr image format." https://developer.android.com/media/platform/hdr-image-format, 2024.

2. Additional comparison methods, e.g., DCDR-UNet

We keep up with the latest research, but there have been few open-source methods in recent years. For example, the reviewer mentioned DCDR-UNet [4] (NOT open-sourced yet), in which the latest comparison method is also KUNet that we have already compared in the paper. Besides, we have tried to positively contact the authors for code during the research and rebuttal periods, but we did not get responses.

To address reviewers' concerns, we unofficially reproduce DCDR-UNet by ourselves and add two more baselines, EPCE-HDR [5] and ITM-LUT [6]. The experiment results on the synthetic dataset are shown below, demonstrating that our method still achieves superior performance over these latest works. Finally, we guarantee that our work will be open-sourced to promote research along this line.

[4] Kim, Joonsoo, et al. "DCDR-UNet: Deformable Convolution Based Detail Restoration via U-shape Network for Single Image HDR Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Tang, Jiaqi, et al. "High Dynamic Range Image Reconstruction via Deep Explicit Polynomial Curve Estimation." ECAI 2023. IOS Press, 2023. 2330-2337.

[6] Guo, Cheng, et al. "Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display." Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production. 2023.

We appreciate the reviewer's valuable comments and suggestions, and we hope our responses could address the concerns.

1. Additional investigation on stack-based methods

The stack-based HDR reconstruction task mentioned by the reviewer is an important research branch of SI-HDR. We investigate the works of [1] [2] [3] [4] [5] [6], and summarize the main differences from our method as follows. The GM-ITM methods are inspired by a novel HDR image format, learning auxiliary data GM that can upgrade SDR to HDR display. By contrast, the stack-based methods simulate the HDRI technology, generating the multi-exposure stack for HDR reconstruction. Besides, compared to the stack-based methods, the GM-ITM methods obtain the final HDR result by multiplication of a single SDR image and the corresponding GM, avoiding the artifacts that may occur during the multi-image fusion process. We will include these discussions in the revised paper.

[1] Wang, Lin, and Kuk-Jin Yoon. "Deep learning for hdr imaging: State-of-the-art and future trends." IEEE transactions on pattern analysis and machine intelligence 44.12 (2021): 8874-8895.

[2] Endo, Yuki, Yoshihiro Kanamori, and Jun Mitani. "Deep reverse tone mapping." ACM Trans. Graph 36.6 (2017): 1-10.

[3] Lee, Siyeong, Gwon Hwan An, and Suk-Ju Kang. "Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image." IEEE Access 6 (2018): 49913-49924.

[4] Lee, Siyeong, Gwon Hwan An, and Suk-Ju Kang. "Deep recursive hdri: Inverse tone mapping using generative adversarial networks." proceedings of the European Conference on Computer Vision (ECCV). 2018.

[5] Kim, Junghee, Siyeong Lee, and Suk-Ju Kang. "End-to-end differentiable learning to HDR image synthesis for multi-exposure images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021.

[6] Zhang, Ning, et al. "Revisiting the stack-based inverse tone mapping." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

2. Comparison experiments on efficiency

We actively contacted the authors of [6] for the code but did not get responses. Therefore, we compare the efficiency of our approach with other available methods [7] [8] [9], and the experimental results are shown below. The runtime is evaluated in NVIDIA A100 as the average of 100 trials on the real-world dataset in the resolution of $4096\times3072$.

As can be seen, benefiting from the simple form of the target GM, our method achieves the second fastest runtime. While the look-up-table-based solution ITM-LUT is faster, its performance on image quality metrics is much lower.

[7] Kim, Joonsoo, et al. "DCDR-UNet: Deformable Convolution Based Detail Restoration via U-shape Network for Single Image HDR Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[8] Tang, Jiaqi, et al. "High Dynamic Range Image Reconstruction via Deep Explicit Polynomial Curve Estimation." ECAI 2023. IOS Press, 2023. 2330-2337.

[9] Guo, Cheng, et al. "Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display." Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production. 2023.

3. Detailed structure of the proposed network

We will provide more details of our network structure in the revised version of our paper. Specifically, in the local head, we use three convolutional layers and ReLU activation functions to extract initial features, where all convolutional layers are in size 3, stride 1, and padding 1, except for the stride of the first convolutional layer is set to 2 to downsample the input. The local tail can be represented as $(Conv \circ ReLU)^2 \circ PS \circ Conv$, where $(\cdot)^n$ means cascade of $n$ modules and all convolutional layers are in size 3, stride 1, and padding 1. The rest implementation details of the other modules can be found in our code that will be open-sourced.

We select LDR images with the exposure of 0 [EV] as the inputs for both VDS and HDREye datasets. The experiments with additional HDR-VDP3 metric on VDS and HDREye datasets are shown below.

As mentioned, these datasets are customized for the Sl-HDR task with a different intent from the ITM task. The uniformly low quantitative metrics for all methods clearly demonstrate the domain gap between the two tasks, making the comparison results less informative.

4. Validation experiments on other datasets

Thank you for providing information of additional datasets. We would like to first explain that these datasets are customized for the SI-HDR task, which is different from the intent of the ITM task. The ITM task focuses more on dynamic range restoration and color gamut conversion instead of the illusion of missing details and reconstruction of unlimited irradiance. Nevertheless, as suggested, we conduct additional experiments and the results for VDS [3] and HDREye [10] are shown below.

Note that, while the domain gap between the tasks makes the quantitative metrics relatively low for all methods, the qualitative comparisons of diverse data from different sources [3] [10] [11] are meaningful and help to increase the credibility of our experiments. The results of the qualitative experiments are shown below.

[Figure A] https://picx.zhimg.com/80/v2-b62509ce89fd10e0a848910fa2465d4e.png

[Figure B] https://picx.zhimg.com/80/v2-e0543ac6ca186abb25a5b05ac38e92a3.png

Experimental results show that our method recovers local and global contrast well and achieves smooth and realistic visual results, verifying its generalizability over a wider range of data. We will add these additional results in the revised paper or supplementary document.

[10] Nemoto, Hiromi, et al. "Visual attention in LDR and HDR images." 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM). 2015.

[11] Liu, Yu-Lun, et al. "Single-image HDR reconstruction by learning to reverse the camera pipeline." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

5. Why the GM can be transmitted at a reduced resolution?

The downsampling of the GM is recognized by standards [12] [13] [14] to save the bandwidth. The resolution-sensitive information such as edges, textures, etc., is stored in the full-resolution SDR image, and the dynamic range information recorded by GM is coupled with the SDR image. Therefore, the downsampling has a small impact on the detail of the final HDR, which is acceptable for bandwidth saving.

[12] ISO. "Gain map metadata for image conversion." https://www.iso.org/standard/86775.html, 2024.

[13] Adobe. "Gainmap specification." https://helpx.adobe.com/camera-raw/using/gain-map.html, 2024.

[14] Google. "Ultra-hdr image format." https://developer.android.com/media/platform/hdr-image-format, 2024.

6. Experimental results for directly learning $Q_{max}$

The ablation results on the synthesis dataset are shown in the table below, where "-", "O", and "√" in the QM column represent not learning, directly learning, and indirectly learning, respectively. Compared to not learning $Q_{max}$, learning $Q_{max}$ directly gets better linear performance, but the imbalanced supervision between tensor and scalar causes a decline in perception metrics, and all the results have a significant decay compared to indirectly learning $Q_{max}$.

7. Open source for code and datasets

We commit to making our codes and datasets publicly available upon acceptance of the paper, enabling the researchers to validate our work and apply it to broader scenarios. We appreciate your emphasis on transparency and look forward to contributing to the community in this way.

We appreciate the reviewer's valuable comments and suggestions, and we hope our responses could address the concerns.

1. Comparison with more related methods

To further increase the comprehensiveness of the experiments, we append three more baselines for comparisons, namely DCDR-UNet [1], EPCE-HDR [2], and ITM-LUT [3]. The experiment results on the synthetic dataset are shown below, demonstrating that our method still achieves superior performance over these latest works.

[1] Kim, Joonsoo, et al. "DCDR-UNet: Deformable Convolution Based Detail Restoration via U-shape Network for Single Image HDR Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Tang, Jiaqi, et al. "High Dynamic Range Image Reconstruction via Deep Explicit Polynomial Curve Estimation." ECAI 2023. IOS Press, 2023. 2330-2337.

[3] Guo, Cheng, et al. "Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display." Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production. 2023.

2. Complexity and computation

There is no additional burden on our dual-branch network for high-resolution input, as the input of the GLE branch $I_{SDR}^{LR}$ is at a fixed $256\times256$ resolution. To evaluate the computational efficiency and real-time processing capability of our method, we conduct comparison experiments and add three more baselines [1] [2] [3]. The runtime is evaluated in NVIDIA A100 as the average of 100 trials on the real-world dataset in the resolution of $4096\times3072$.

As can be seen, benefiting from the simple form of the target GM, our method achieves the second fastest runtime. While the look-up-table-based solution ITM-LUT is faster, its performance on image quality metrics is much lower.

3. Relevance to the new data format

It needs to be clarified that our method does not rely on the new GM format. The proposed GMNet can estimate GM from the input SDR image. Then the SDR-GM pair can be directly encapsulated into the new GM format, but also can be calculated to linear HDR through the pipeline in Section 3. After that, we can convert the linear HDR into other mainstream HDR formats, such as PQ-encoded HDR image in HDR10 standard, not limited to specialized devices or formats.

4. Visual comparisons in challenging real-world scenarios

Figure 9 and Figure 10 in the Appendix demonstrate the superiority of our method in reconstructing edges and high-contrast night scenes. To enable more in-depth analysis in challenging scenarios, we perform qualitative experiments as follows.

[Figure A] https://picx.zhimg.com/80/v2-f7a1fc6a62f5bf91169047c041d40f49.png

[Figure B] https://picx.zhimg.com/80/v2-9a63b9bf1e886282ae886f41fccfad9e.png

As shown in Figure A, our method achieves superior performance in the sun region at extreme brightness. The HDCFM also achieves low errors, but with grid effect due to operator properties. Figure B shows the estimation results on the leaves with complex textures and eaves with sharp edges, and our method demonstrate superior performance on complex textures in the real-world scenario, validating the generalization and the robustness of the proposed method.

We appreciate the reviewer's valuable comments and suggestions, and we hope our responses could address the concerns.

1. Insight into challenging conditions

Figure 10 in the Appendix demonstrates the superiority of our method in reconstructing high-contrast night scenes. To enable more in-depth analysis in challenging conditions, we conducted the following qualitative experiments.

[Figure A] https://picx.zhimg.com/80/v2-f7a1fc6a62f5bf91169047c041d40f49.png

[Figure B] https://picx.zhimg.com/80/v2-9a63b9bf1e886282ae886f41fccfad9e.png

As shown in Figure A, our method achieves superior performance in the sun region at extreme brightnesses. The HDCFM also achieves low errors, but with grid effect due to operator properties. The Figure B demonstrates superior performance of our method in challenging condition with complex textures.

2. Model size trade-offs

Keeping the network structure unchanged, we adjust the number of hidden layers to control the model size. The results of the ablation experiments on the synthetic dataset are shown below, and our implementation in the paper is 64 hidden layers.

The experimental results show that reducing model parameters does not lead to significant performance degradation, especially in SSIM-L, SSIM-PQ and HDR-VDP3 that measure perception quality, verifying that our method can maintain accuracy with reduced resources.

3. More dataset information

In the Appendix of our paper, we present the diversity of the real-world dataset in Figure 7 and Figure 8. To address the reviewer's concerns, we demonstrate the diverse scenes and thumbnails of the synthetic dataset as follows.

[Figure C] https://picx.zhimg.com/80/v2-c9660565e0e70b53d24970ceafcdb691.png

[Figure D] https://picx.zhimg.com/80/v2-f0c2500d31685c44d74b83eca67fd82d.png

The SDR-GM pairs of the real-world dataset are directly derived from the mobile device. The SDR images in the synthetic dataset are degraded from HDRTV frames. Specifically, we first roll off the input HDR by EETF and transfer it to the P3 gamut after linearization, then conduct extended Reinhard tone mapping, clip it under 100 nit, and finally store it in uint8 for quantization. After getting the SDR images, the detailed formation pipeline of the GM can be found in Section A in the Appendix. For more detailed information, we will release our codes and datasets upon acceptance of the paper.

4. Real-Time performance evaluation

To evaluate the computational efficiency and real-time processing capability of our method, we conduct comparison experiments and add three more baselines [1] [2] [3]. The runtime is evaluated in NVIDIA A100 as the average of 100 trials on the real-world dataset in the resolution of $4096\times3072$.

As can be seen, benefiting from the simple form of the target GM, our method achieves the second fastest runtime. While the look-up-table-based solution ITM-LUT is faster, its performance on image quality metrics is much lower.

[1] Kim, Joonsoo, et al. "DCDR-UNet: Deformable Convolution Based Detail Restoration via U-shape Network for Single Image HDR Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Tang, Jiaqi, et al. "High Dynamic Range Image Reconstruction via Deep Explicit Polynomial Curve Estimation." ECAI 2023. IOS Press, 2023. 2330-2337.

[3] Guo, Cheng, et al. "Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display." Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production. 2023.

5. Scalability of the model

It needs to be clarified that the experiments in our paper are all performed on high-resolution images, where the resolution of the synthetic dataset is $3840\times2160$ and the resolution of the real-world dataset is $4096\times3072$. Therefore, the GMNet can scale effectively with high-resolution images in most cases.