HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting

Response to Reviewer VwS8

Q-1: Highlight of innovations and explanations of model design and theoretical underpinnings

A-1: We propose the first 3DGS-based framework with 1.91 dB improvements and 1000x inference speed for HDR novel view synthesis (NVS). Firstly, we find that directly applying the original 3DGS to HDR NVS does not work well. In Table 3 (a), 3DGS only yields 12.35/11.83 dB on LDR-OE/LDR-NE and cannot render HDR images. To improve the performance, we make innovations in algorithm and data.

(i) Algorithm. The original 3DGS is designed for LDR imaging. If directly enforcing an HDR constraint to the output of 3DGS like Eq.(16), 3DGS only yields a suboptimal result of 30.15 dB on HDR, as shown in the following table. This is because the broader range of brightness and high-frequency details in HDR images are difficult to model by the spherical harmonics with limited order in the original Gaussian point clouds. Plus, 3DGS cannot change the light intensity of the rendered LDR images, thus yielding poor results on LDR.

To address these problems, we design the DDR Gaussian point cloud model based on the tone-mapping theory [16]. Our DDR Gaussian model in Eq.(1) contains more attributes required for HDR NVS, such as HDR color, exposure time, etc. We perform a global tone-mapping operation on all Gaussian point clouds at one time to extract more contextual color information and accelerate the inference speed. Then, we develop the parallel differentiable rasterization to render the LDR and HDR images. As shown in Table 3 (a) of the paper and the above table, our innovation in Gaussian point cloud model leads to significant improvements of 21.64, 17.36, and 8.16 dB on LDR-OE, LDR-NE, and HDR.

(ii) Data. As analyzed in Lines 174 - 181, the normalized device coordinate (NDC) system restricts the representing ability and spatial transformations of 3D Gaussians. Besides, the datasets collected by HDR-NeRF do not provide point clouds for initialization. To address these problems, we recalibrate the camera parameters and compute the initial point clouds as Eq. (14). In Table 3 (a), the data recalibration and point cloud initialization lead to 2.27/2.58 and 4.84/4.56 dB improvements on LDR-OE/LDR-NE, while alleviating the over-fitting issue of 3DGS.

Q-2: Analysis of the performance difference between HDR-GS and HDR-NeRF

A-2: We are not sure if the "performance difference" you mention refers to speed or image quality. Thus, we analyze both aspects.

Firstly, two techniques lead to the speed advantage of HDR-GS.

(i) Rendering Scheme. As analyzed in Lines 40 - 45, HDR-NeRF suffers from a time-consuming rendering scheme for LDR and HDR images. It needs to sample many 3D points and then compute their densities and colors for every single ray, severely slowing down the training and inference processes. In contrast, HDR-GS adopts the parallel differentiable rasterization for LDR and HDR rendering. As described in section 3.2, the rasterization computes different tiles divided from the 2D projection in high parallelism on GPU, thus enjoying much faster inference speed.

(ii) Tone-mapping Method. HDR-NeRF conducts tone-mapping in a time-consuming ray-tracing manner. It needs to convert HDR to LDR for all sampled points along every single ray. It can only exploit the limited color information along a single ray for the tone-mapping transformation and requires repeated computation many times. In contrast, HDR-GS performs a global tone-mapping operation that converts the HDR colors of all Gaussian point clouds at one time. This operation can extract more contextual color information for HDR-to-LDR transformation and accelerate the inference speed. The following table compares the effects of the two tone-mapping methods on HDR-GS. Our global tone-mapping method performs better and faster than the ray-tracing tone-mapping of HDR-NeRF.

Secondly, although our work is based on 3DGS, directly using 3DGS for HDR NVS yields poor image quality. To improve the quality to surpass HDR-NeRF, we make innovations. Please refer to A-1 for details.

Q-3: Why modeling the HDR color first?

A-3: This is because HDR images capture a broader range of illuminance levels retaining the details in dark and bright regions. HDR images contain information stored in LDR images. Thus, HDR-to-LDR transformation is an easy information compression process. In contrast, the transformation from LDR to HDR is an ill-posed problem because it needs to reconstruct the missing brightness and details that are not captured in LDR images. This process usually involves more complex computations. We do experiments to compare HDR-to-LDR and LDR-to-HDR transformations in the following table. HDR-to-LDR performs better.

Q-4: Condensing section 3.2

A-4: We will follow your advice to condense this section

Q-5: Questions about the parallel differentiable rasterization

A-5: In fact, parallel differentiable rasterization does not mean parallel computing for HDR and LDR in one rasterization. We perform HDR rasterization and LDR rasterization, separately. The word "parallel" here has two meanings:

(i) The HDR and LDR rasterization processes query the same attributes and spatial transformations of the same 3D Gaussians in Eq. (10) - (13), except for the color.

(ii) The tiles divided from the 2D projection are computed in parallel on GPU during the rasterization to accelerate the speed, see Line 165 - 169 and Eq. (13).

We will add an explanation in the revision

Response to Reviewer tLXM

Q-1: Analysis of logarithmic tone-mapper vs. linear tone-mapper

A-1: Thanks for providing a mathematical explanation of the logarithmic tone-mapping operation. We will add this analysis in the revision with acknowledgment.

Q-2: Why using the images under the same exposure of different views to recalibrate the camera parameters instead of using all images?

A-2: The recalibration is done by the Structure-from-Motion (SfM) algorithm. SfM needs to detect and then match feature keypoints between different views. Thus, if using all images for calibration, the feature detection and matching might be less accurate due to the change of light intensity at the same position. As a result, the calibrated camera poses may also be inaccurate. We conduct the experiment of using the camera parameters calibrated from all images on the synthetic datasets to train HDR-GS, the model achieves results of only 34.21, 36.19, and 33.75 dB on HDR, LDR-OE, and LDR-NE. Compared to Table 3 (d), these results are 2.67, 3.52, and 1.53 dB lower than the lowest results of single-exposure recalibration with $t_1$ = 0.125 $s$ on HDR, LDR-OE, and LDR-NE.

Q-3: Add some legend or annotation to the teaser figure

A-3: Thanks for reminding us. In Figure 1, we report five metrics of HDR novel view synthesis including PSNR in dB, inference speed in fps (frames per second), training time in minutes, SSIM, and LPIPS score. In particular, lower values are better for LPIPS and training time, while higher values are better for the other metrics. We will add a legend to explain the metrics.

Q-4: Which perceptual network is used for the LPIPS score?

A-4: Following the same test settings as HDR-NeRF for fair comparison, we use AlexNet [77] as the perceptual network to compute the LPIPS score. We will add more experimental details in Section 4.1.

[77] ImageNet Classification with Deep Convolutional Neural Networks. In NIPS 2012.

Q-5: Are the training sets and testing sets of real scenes completely separate?

A-5: Yes, of course. You have a good observation. We also found this mistake in the official repo of HDR-NeRF. We rewrite the data splitting code to make sure there is no overlap between training and testing sets of real scenes. Please check the submitted code, which is consistent with the description of the implementation details in our main paper.

Q-6: The training stability of our HDR-GS

A-6: We did not experience model collapse during the training process. Our models were trained once and succeeded. We conduct five repeated experiments on the synthetic datasets. The PSNR results are shown in the following table. The performance fluctuation is within 0.21, 0.16, and 0.09 dB on LDR-OE, LDR-NE, and HDR. These results suggest the robustness and training stability of our HDR-GS.

Response to Reviewer E3vH

Q-1: Questions about the training data

A-1: (i) Actually, our method only requires an LDR image with a single exposure time $\in$ {$t_1, t_3, t_5$} at each view to train in Eq.(15).

(ii) As claimed in Lines 198 - 200, we do not use HDR images for supervision $\mathcal{L}_c$ in real scenes, $\gamma = 0$ in Eq.(17). We only use HDR supervision for synthetic scenes to align the HDR output with the Blender rendering style. This is because the transformation from LDR to HDR is not unique. It is affected by camera parameters, rendering software settings, etc. In our work, synthetic HDR images are synthesized by the software Blender with specific settings and the evaluation metrics for HDR images are hard pixel alignment. Thus, it is reasonable to enforce a constraint on the rendered HDR images. To this end, HDR-NeRF uses the GT camera response function (CRF) correction coefficient $C_{0}$ to rectify the HDR output. But $C_{0}$ is a strong scene-specific prior and not available in real practice because CRF is the target to model, not a given condition. Hence, we use tone-mapped HDR images for supervision instead. For fair comparison, we conduct experiments without using HDR restrictions in the following table. HDR-GS is 14.91 dB higher than HDR-NeRF on HDR.

In Figure 6, our method renders more visually pleasant HDR images without HDR supervision than HDR-NeRF in real scenes.

Q-2: Comparison of HDR-GS and HDR-NeRF

A-2: Our HDR-GS is different from HDR-NeRF in

(i) Motivation. HDR-NeRF aims to learn the neural HDR radiance fields. It is an implicit 3D representation that lacks geometric information. In contrast, HDR-GS aims to reconstruct 3D HDR Gaussian point clouds that capture the scene geometry. It is an explicit 3D representation with better controllability and interactivity.

(ii) Technique. (1) In Lines 40 - 45, HDR-NeRF suffers from a time-consuming rendering scheme. It samples many points to compute densities and colors for each ray. In contrast, HDR-GS adopts rasterization to render different tiles in high parallelism on GPU, enjoying much faster speed. (2) The tone-mapping operation of HDR-GS is also fundamentally different from that of HDR-NeRF. HDR-NeRF conducts tone-mapping in a ray-tracing manner. It converts HDR to LDR at all sampled points for every single ray. This ray-tracing tone-mapping only extracts the color information along a single ray and further slows down the inference speed. In contrast, HDR-GS performs a global tone-mapping operation that converts HDR colors of all 3D Gaussians to LDR colors at one time. This operation captures more contextual color information for HDR-to-LDR transformation and accelerates the inference speed. The following table shows the results of using the two tone-mapping methods on HDR-GS. Our global tone-mapping is better and faster.

(iii) Data. HDR-NeRF uses the normalized device coordinate (NDC) system. But, as analyzed in Lines 174 - 181, the NDC system restricts the representing ability and transformations of 3D Gaussians. Plus, the datasets collected by HDR-NeRF do not provide point clouds for initialization. To address these problems, we recalibrate the camera parameters and compute the initial point clouds as Eq.(14). See Table 3 (a), our data recalibration and point cloud initialization lead to 2.27/2.58 and 4.84/4.56 dB improvements on LDR-OE/LDR-NE, while alleviating the over-fitting issue.

(iv) Performance. HDR-GS shows significant advantages over HDR-NeRF. HDR-GS surpasses HDR-NeRF by 1.91 dB on HDR in synthetic scenes (Table 1) and 3.84 dB on LDR in real scenes (Table 2), while enjoying 1000x inference speed. In Figure 6, HDR-GS reconstructs clearer HDR details and brightness.

Q-3: Performance of 3DGS trained with GT HDR images

A-3: We conduct experiments to compare HDR-GS with 3DGS directly trained with GT HDR images in the following table. HDR-GS outperforms 3DGS by 25.98, 22.86, and 8.16 dB on LDR-NE, LDR-OE, and HDR.

We analyze these results:

(i) The suboptimal HDR result of 3DGS stems from that the high-frequency details and broader range of brightness in HDR images are hard to capture by the spherical harmonics (SH) with limited order in original Gaussian point clouds.

(ii) The LDR results of 3DGS are poor because the original 3DGS cannot control the light intensity according to the exposure time.

(iii) Besides, training 3DGS with HDR images is hard in real scenes where HDR images are difficult to obtain. In contrast, HDR-GS only requires LDR images for supervision in practice.

Q-4: Ablation of $b$ in Eq.(8)

A-4: We follow your advice to do an ablation of $b$ in the following table.

Q-5: Can HDR-GS be applied to data where each view has a different exposure time?

A-5: Yes. HDR-GS only requires an LDR image with a single exposure time at each view to train. We do experiments with training views of different exposure times in the following table. HDR-GS performs better and faster.

Q-6: The difficulty of data acquisition in real world

A-6: The real-world data is easy to obtain. In real scenes, HDR-GS only requires a single LDR image at each view to train and no HDR images are required. The exposure time can be easily set in the camera and read from the EXIF files.

Thanks for your reply. We appreciate your recognition of our work and performance.

Actually, our HDR-GS is not simply adjusting HDR-NeRF to 3DGS. Here are some comparisons in technical details:

(i) Tone-mapping Operation.

The tone-mapping operation of HDR-NeRF converts the HDR color of the sampled 3D points to LDR color following volume rendering along every single ray. Specifically, the HDR volume rendering is formulated as

$\mathbf{I}^h(\mathbf{r}) = \sum_{i=1}^{N} T_i (1 - \exp(-\rho_i \delta_i)) \mathbf{c}^h_i$,

where $\rho_i$ is the volume density at the $i$-th sampled point. $\mathbf{c}_i^h$ is the HDR color at the $i$-th point. $\mathbf{I}^h(\mathbf{r})$ is the HDR color of the pixel where ray $\mathbf{r}$ lands on. The tone-mapping operation of HDR-NeRF converts $\mathbf{c}^h_i$ to the LDR color $\mathbf{c}^l_i$.

3DGS also adopts a similar point-based rendering in rasterization. So directly adapting HDR-NeRF to 3DGS should perform the tone-mapping operation following the HDR rasterization. Specifically, in Eq.(13), the point-based rendering in HDR rasterization is

$\mathbf{I}^{h}(p) = \sum_{j \in \mathcal{N}} \mathbf{c}_j^h \sigma_j \prod _{k=1}^{j-1}(1-\sigma_k)$.

The naive adaptation should perform tone-mapping to convert $\mathbf{c}_j^h$ to $\mathbf{c}_j^l$ along the ray landing on the pixel $p$. However, as compared in the following table, which is copied from the table in A-2 of our rebuttal for your convenience, we found this adaptation only yields suboptimal results and speed because it can only extract limited color information on a single ray for HDR-to-LDR transformation and needs to compute many times.

To address these issues, we design the Dual Dynamic Range (DDR) Gaussian point cloud model that performs a global tone-mapping operation converting the HDR colors of all Gaussian point clouds to LDR at one time to extract more contextual color information and accelerate the inference speed. As shown in the above table, our global tone-mapping leads to improvements of 1.59/1.65/2.19 dB and 48 fps on LDR-NE/LDR-OE/HDR and speed.

(ii) Data Recalibration.

As analyzed in Lines 174 - 181, HDR-NeRF adopts the normalized device coordinate (NDC) system that rescales the coordinates to the unit cube [-1, 1]$^3$ to help stabilize the training. Plus, the data collected by HDR-NeRF does not provide point clouds for the initialization of Gaussian point clouds.

The naive and straightforward adaptation is to randomly init the positions of Gaussian point clouds within the cube [-1, 1]$^3$ and use the NDC with 2D projections to optimize. However, when we try this naive adaptation, the HDR-GS only achieves poor results of 24.45, 25.31, and 23.08 dB on HDR, LDR-OE, and LDR-NE for two reasons. Firstly, the NDC restricts the representing ability and spatial transformation of Gaussian point clouds. Secondly, training the randomly initialized Gaussian point clouds with few views (only 18) leads to an overfitting issue.

To address these problems, we use the SfM algorithm to recalibrate the camera parameters and compute the initial point clouds. A naive method is to use all images to recalibrate, as mentioned in Q-2 of our response to reviewer tLXM. However, this naive recalibration achieves suboptimal results of only 34.21, 36.19, and 33.75 dB on HDR, LDR-OE, and LDR-NE because the light intensity change degrades the accuracy of feature keypoints detection and matching. Hence, we use the LDR images with the same exposure time to recalibrate in Eq.(14), leading to improvements of 4.10, 4.91, and 2.58 dB on HDR, LDR-OE, and LDR-NE.

(iii) HDR supervision.

HDR-NeRF uses the ground truth CRF correction coefficient $C_0$ for HDR supervision. The naive adaptation should also use $C_0$. Yet, we found that training with $C_0$ correction is unstable and the model easily collapses. Besides, $C_0$ is unavailable in real scenes where HDR-NeRF naively sets $C_0 = 0.5$. Yet, this inaccurate $C_0$ may cause color distortion or introduce black spots, as shown in Figure 6. Thus, we use Eq.(16) as the HDR supervision for quantitative evaluation to stabilize the training process. When HDR images are not available in practice, we do not use inaccurate $C_0$ to avoid degradation.

Besides, we resolved the data splitting issue on the real datasets, as mentioned in Q-5 of our response to reviewer tLXM.

3DGS is a great work. Yet, directly applying 3DGS or naively adapting HDR-NeRF to 3DGS does not work well. Our work, as the first attempt, proposes an effective method to explore the potential of 3DGS for HDR imaging. We are glad to share our code, model, and recalibrated data with the community.

Feel free to ask us if you have other questions. Looking forward to your reply.