High Dynamic Range Novel View Synthesis with Single Exposure

Reviewer ZUxK

Q1: While it may be relatively straightforward to outperform HDR-GS and HDR-NeRF when tailoring the design to a specific setting, the real challenge lies in demonstrating that the method can also surpass these baselines under their own conditions.

Great point! As suggested, we have now evaluated Mono-HDR-GS under the conventional multi-exposure setting. As shown in the results below, we show that overall, our method achieves superior performance for both HDR and LDR rendering. We will add this test in the final version.

HDR rendering results on the synthetic datasets.

LDR rendering results of observed exposure (LDR-OE) on the synthetic datasets.

LDR rendering results of novel exposure (LDR-NE) on the synthetic datasets.

Q2: The authors consider ideal conditions, where multi-view images have the same exposure. However, in real-world scenarios, this is difficult because each camera has different exposure settings.

Apologies for this misunderstanding (multi-view vs. multi-camera). Under the proposed single-exposure setting, even a single camera device with a single shutter time setup would suffice to acquire multi-view training imagery. This is more convenient than the multi-camera requirement as the reviewer mentioned. We will clarify this.

Q3: Some LDR novel view synthesis methods maybe need to discuss, which handle lightness correction in LDR domain, and may could connect with 2D inverse tone mapping methods to serve as a baseline, i.e. [5-7]. It might be more reasonable to add some 2D inverse tone methods and basic 3DGS combinations to the comparison methods. [5] Lighting up NeRF via Unsupervised Decomposition and Enhancement, ICCV 2023; [6] Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption, AAAI 2024; [7] A Bilevel Optimization Approach for Novel View Synthesis, AAAI 2024.

Thanks for the suggestions. We will discuss them though being less relevant, as they either focus on luminance correction (vs. our color space transformation) or single image cases (vs. our 3D scene modeling). They are thus not proper competitors due to addressing different problems.

Q4: Currently, the baselines are limited, and the paper lacks more depth analyze about how to ensure multi-view consistency.

To the best of our knowledge, HDR-NeRF and HDR-GS are the only two state-of-the-art methods for HDR-NVS. We are more than happy to include more if suggested. The core of our model lies in how to learn the mapping from LDR to HDR, whilst multi-view consistency is ensured by the 3D scene model (e.g., 3DGS or NeRF) adopted. As a result, our approach is open and generic to integrate with any 3D representation models.

Q5: Would it be possible to include some video comparison results in the supplement? This could make the visual effects more apparent, especially regarding 3D consistency.

Great suggestion! We will add video results.

Reviewer nprx

Q1: How to validate the two MLPs L2H-CC and H2L-CC decompose the camera imaging process. There are no supervision in the loss function to ensure this part.

Great question! It is exaclty due to no such supersion that makes the problem extremely challenging. It is the architecture of the two convertors (Fig. 3 & 4) we introduce here that imposes structural prior of camera imaging (Eq. (6)) to drive the model approaximate the underlying camera imaging process. This has been validated in ablation study (see Tab. 3) by comparing with plan MLP without such structure.

Q2: The improvements on real scenes are marginal. Why? The authors do not explain this.

On real scenes without HDR ground truth, the improvement is indeed more challenge to acquire. However, to provide more evidence, we have now quantitatively evaluated two no-reference image quality assessment (NR-IQA) metrics: NIQE [3] and CLIP-IQA [4], without the need for HDR ground truth. We report the HDR results on real-world datasets below:

This test further indicates the meaningful superiority of our method over previous alternatives. Pleases note, Tab. 2 (main paper) reports the results of LDR rendering which is not the focus of this work.

[3] Mittal A, Soundararajan R, Bovik A C. Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 2012. [4] Wang J, Chan K C K, Loy C C. Exploring clip for assessing the look and feel of images. AAAI 2023.

Q3: Code and models are not submitted. The reproducibility cannot be checked.

Our code and models will be released later.

Q4: I suggest the authors to re-organize the paper to remove the blank in lines 159-164.

Thanks, we will.

Q5: The HDR results in Fig. 5$c$ look very terrible and are totally different from those in the original paper of HDR-GS.

Great spot, but please note that our single-exposure setting is a more challenging and more practical, as comthe to the conventional multi-exppsure setting as used in the HDR-GS. This contrast illustrates exactly the challenges with our new setting (e.g., the limited luminance information cannot fulfill the Nyquist-Shannon sampling theorem requirements for dynamic range recovery). We used the official codes of HDR-GS to ensure the correction (with the same code, we can reproduce the results of multi-exposure setting). Please note, this work does not fully solve the single-exposure HDR-NVS problem but indicates a meaningful forward step and forster more advanced research in the future.

Reviewer dY8v

Q1: The proposed L2H-CC converts an LDR model to an HDR model, however, after reading the paper, I'm not completely clear how its design prevents the module from learning a trivial solution of mapping to an LDR instead of HDR mode.

Let us summarize the key features of our method: First, previous methods such as HDR-GS (Cai et al., 2024) and HDR-NeRF (Huang et al., 2022) are inferior in design to tackle this more challenging single-exposure HDR-NVS problem since directly learning a HDR scene model from single-exposure multi-view imagery is extremely challenging (see L86-93). With Mono-HDR-3D, we instead first learn a LDR scene model. More importantly, we impose the inherent camera imaging mechanism [1-2] (see the camera imaging formula Eq. (6), Sec 3.2) to facilitate HDR color estimation - enabling more robust translation from LDR to HDR by leveraging imaging physics prior knowledge (L190-216, Sec. 3.2). We will further clarify.

[1] Noise-optimal capture for high dynamic range photography[C]//CVPR, 2010: 553-560. [2] Compressed-SDR to HDR Video Reconstruction[J]//TPAMI, 2023, 46(5): 3679-3691.

Q2: How does Mono-HDR-GS use HDR loss? The paper claims to only use LDR images, there should be no HDR ground truth during optimization.

Apologies for this misunderstanding. As stated in L255-265 that, our model supports both cases, with and without HDR ground truth. Given HDR data, the HDR loss $L_\text{hdr}$ is used along with the others as shown in Eq. (9).

Q3: The ablation study of closed-loop design lacks implementation details. How does supervision work without “H2L-CC”? What supervision is used to learn “L2H-CC”?

In the closed-loop ablation study (Tab. 4), when the H2L-CC module is omitted, the HDR loss $L_\text{hdr}$ directly supervises the L2H-CC module to learn the LDR-to-HDR transformation, and $L_\text{ldr}$ enforces the input to L2H-CC as a valid LDR model. Additionally, if H2L-CC exists, an extra loss $L_\text{h2l}$ will be added to supervise the L2H-CC.

Q4: It’s better if authors can add ablation on the losses, $L_\text{ldr}$, $L_\text{hdr}$, and $L_\text{h2l}$ for a more comprehensive evaluation.

Thanks! As suggested, we have now conducted an exhaustive analysis of loss combination. The results of HDR rendering are reported below:

We highlight that:

• HDR loss $L_\text{hdr}$ is basically important as expected;
• LDR loss helps clearly by properly supervising the LDR scene model optimization;
• The closed loop loss $L_\text{h2l}$ adds further value on top.

Reviewer yk8h

Q1: I question the necessity of H2L module: what happens if you render the image in HDR, and simply convert it to LDR with existing modules or analytic method, instead of approximating it with MLP? Is this case not possible because there are no module which support backpropagation for training? If not, how does the model perform in the case of such naive HDR-to-LDR conversion method? Please elaborate.

Great point! Please note analytical HDR-to-LDR conversion needs the camera CRF which is typically unknown. To address this, we thus learn to approximate. We will further clarify.

Q2: I question the authors about the case when only LDR images are available for optimizing the scene, which the setting implied in the first paragraph of H2L-CC section at Section 3.2. In this case, how is the L2H module able to learn the mapping from LDR to HDR? It seems that this module is not generalizable and optimized per scene with NeRF or 3DGS, so I suppose it would not be able to receive and guidance signals for learning L2H in cases when no HDR images are available. I ask the authors to provide additional elaboration and experimental results for this setting.

Without HDR ground truth, any model will be less properly constrained including ours. However, compared with previous methods, our Mono-HDR-3D has advantages due to leveraging the inherent camera imaging mechasnim (imaging physics prior) along with the closed-loop design (H2L-CC followed by L2H-CC, forming a loop).

As suggested, we now quantitatively evaluate two no-reference image quality assessment (NR-IQA) metrics: NIQE [3] and CLIP-IQA [4], without the need for HDR ground truth. We report the HDR results on real-world datasets below:

This test validates the efficacy of our model design.

[3] Mittal A, Soundararajan R, Bovik A C. Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 2012. [4] Wang J, Chan K C K, Loy C C. Exploring clip for assessing the look and feel of images. AAAI 2023.

Q3: I believe additional experiment would have had to be performed to validate the performance of this method in cases when different ratio of LDR / HDR images are available, including extreme cases where only LDR or HDR images are available for 3D scene optimization.

Great suggestions and many thanks! We have now conducted this suggested experiment, with the ratio of LDR / HDR images being 1/1, 2/1, 3/1, 5/1, 0/1, 1/0, respecitively.The results are reported below:

We highlight that:

• As the amount of HDR data decreases, our model degragdes only marginally, suggesting the merit of data efficiency.
• LDR supervision is useful by providing a better quality scene model to be converted.
• HDR supervision is most critical as expected.
• Compared with HDR-GS, overall our model is superior acrss all the cases.

Q4: I find qualitative results to be somewhat lacking for me to be fully convinced in the performance of this method.

Reconstructing HDR radiance fields from single-exposure LDR inputs constitutes an ill-posed inverse problem, as the limited luminance information fails to satisfy the Nyquist-Shannon sampling theorem for dynamic range recovery. Therefore, the perceptual quality of synthesized HDR images remains fundamentally constrained by the limited luminance information in single-exposure inputs.