Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

Thank you for recognizing the motivation, effectiveness, and thoroughness of our experiments on LE3D. We also appreciate your valuable comments! Below, we address the specific points.

1. Does CSI really matter? We have done our ablation studies on CSI in our main paper, please refer to Fig. 4 (b) in our main paper and Fig. 1 in the rebuttal PDF. Compared to LE3D, the removal of CSI leads to a lack of distant scene details and the splitting of a large number of gaussians at incorrect depth levels to fit the distant scene, thereby resulting in failure in structural optimization. Additionally, there is a severe decline in visual quality (please zoom in for the best view). This is because the sparse point cloud cannot reconstruct distant points effectively, whereas CSI can effectively increase the initial point cloud in distant areas, achieving the goal of effective distant scene reconstruction.

2. Initial SfM on Dark Scenes: We follow the exact settings of RawNeRF to obtain the initial SfM using COLMAP (except we use the PINHOLE camera model instead of OPENCV). In the RawNeRF dataset, COLMAP works well on calibrating camera poses but not very well on generating point clouds. The lack of point cloud initialization is also one motivation for our purpose of CSI initialization, which enriches the point cloud and extends its depth range. If the scene is too dark in other data, leading to poor COLMAP calibration, we brighten the scene using the DNG data to obtain corresponding JPG images, and then perform COLMAP calibration.

3. Why SH is less expressive than MLP? As this is a common problem, please refer to the third section "What limits the final representation capability of SH on linear color space" in the global rebuttal for details.

4. Depth Distortion: Our depth distortion map is an approximate implementation of equation (14) from Mip-NeRF 360 as we mentioned in the main paper L197-L198. Both of our regularizations are used to improve structure reconstruction. Please refer to Table 2 in the supplementary for a quantitative comparison of the effects of both $R_{dist}$ and $R_{nf}$, and Fig. 6, 7 in the supplementary for qualitative comparisons.

5. Overstated Performance: Thanks for pointing out this. We apologize and will adjust our wording accordingly in our next version.

6. Quantitative Results of Ablation Studies: Please refer to Table 2 in our supplementary.

We hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.

Thank you for recognizing our motivation and the structure of our paper, as well as your valuable comments! Below, we address the specific points.

1. About the related work: we will add and discuss them in our next version. We will add more discussion for sRGB images (both w/wo multi-exposure) based on novel view synthesis techniques with your referred papers.

2. Does RawGS (RawNeRF + 3DGS) perform well enough? We do not believe that RawGS (RawNeRF + 3DGS) performs well enough. The reasons are as follows: 1) Colour reconstruction failure & more floater remaining: As shown in Fig. 1 (left), Fig. 3, and Fig. 13-15 in our main paper and supplementary, these results indicate that RawGS performs worse than LE3D in terms of colour and floaters. 2) Structure reconstruction failure: As shown in Fig. 5 (c, e) and Fig. 13-15 in our main paper and supplementary, it is evident that RawGS fails to reconstruct the structure (depth map). This is disastrous for downstream tasks such as refocusing, as shown in Fig. 5 (b, d) in our main paper and Fig. 9 in supplementary. Therefore, we do not believe RawGS performs well enough, which is also the motivation for LE3D: to make 3DGS available on real-time HDR view rendering from noisy RAW images, as well as downstream editing tasks.

3. Regarding CSI and dense point cloud initialization: We have previously conducted experiments using dense point clouds (this is also what we tried before coming up with the idea of CSI), which indeed partially solved the problem of failing to reconstruct distant scenes. However, it led to the following issues: 1) Longer time cost: The time required to reconstruct dense point clouds is four to five times that of sparse point clouds (from 6min 23sec to 28min 53 sec on scene 'bikes'), thereby increasing the overall reconstruction time. 2) Redundant gaussians when converged: An excessive number of points leads some gaussians to fit noise, which negatively increases the count of gaussians. For instance, in the 'bikes' scene, the number of points increased from 4.1e5 to 5.4e5, which is approximately a 30% increase, resulting in longer rendering time and more storge cost. So a balance between sparse and dense point cloud initialization is important, and then here comes our CSI. In fact, it is hard to find out the difference between the visual result of dense initialization and CSI initialization. We can provide proof of this.

4. Regarding the speed issue between MLP and SH: We believe this balance is worthwhile because both structure and colour are important for downstream tasks, and the speed of LE3D remains real-time. More discussion on MLP and SH can be found in the third section "What limits the final representation capability of SH on linear colour space" in the global rebuttal.

5. Regarding increasing the splitting threshold and the idea before CSI: We experimented with increasing the splitting threshold as shown in Fig. 1 (c) in the rebuttal PDF, but it did not effectively solve the issue of distant scenes and actually increased reconstruction noise. The reason it cannot effectively address distant scenes is that our dataset is forward-facing, and gaussians tend to move parallel to the camera plane rather than perpendicular to it. Therefore, if there are not enough gaussians for distant scenes during initialization, additional gaussians will not appear spontaneously. We observed this phenomenon based on previous experiments with dense point clouds. Dense point clouds can reconstruct points in the distance, but this also impacts training time and the number of gaussian. Our CSI effectively balances the number and distance of points.

6. Regarding colour representation besides MLP and the co-optimization of colour and structure: Currently our choice of MLP proves sufficiently effective with its learning ability, as shown in Fig. 1 (d) and Table 2 in the rebuttal PDF. However, we agree with the reviewer that trying other strategies is interesting future work. In vanilla 3DGS (with SH), there is indeed a phenomenon where colour and structure do not optimize well together. As we mentioned in the ablation study (Fig. 4 (e) in main paper), gaussians with SH in the early stages lead to colour optimization failure, which in turn causes structure optimization failure before 15,000 iterations (during which gaussian primarily optimize structure through split/clone). This, in part, limits the final colour performance of SH. Please refer to the third section "What limits the final representation capability of SH on linear color space" in the global rebuttal for details. Besides, the per channel scaling behaving with/without the colour MLP can also be found in Fig. 1 (d) in the rebuttal PDF.

We hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.

I thank the reviewers very much for their response.

2) Thank you for stressing the refocusing results, I think these indeed show that RawGS just can't perform some tasks, despite not performing badly on NVS metrics.

3, 5) I think the author's response and Fig 1. in the rebuttal adequately show the importance of CSI. I would implore the authors to add this discussion to the paper/supplement.

4, 6) I think the authors misunderstood me slightly, I'm not saying the MLP is not necessary, I guess I just wanted to know what other alternatives the authors have tried. I think this is important since as I mentioned the MLP brings about a 1.7 times reduction in rendering speed, which is the main point of the paper.

I appreciate the figures the authors have included in the general rebuttal about the statistics of the SH-derived colors. I think they beg the question, what if an activation was used on top of the radiance prediction from the SH? RawNeRF uses an exponential activation to increase the dynamic range. I think this might serve a similar purpose here, increasing the dynamic range and also making the outputs non-negative? Does that make sense?

I think I'm replying quite late, and the authors might not have enough time to address this, apologies for this. Although I would love to see this experiment before the end of the discussion phase, I understand this might not be enough time and would then love to see it for the camera ready.

Thank you for your recognition of our writing and your valuable comments! Below, we address the specific points.

1. Request for more insights on MLP and SH: We have analyzed the instability factors of SH during training and provided more statistical data to demonstrate its limited representation capability under high contrast conditions. Please refer to the third section "What limits the final representation capability of SH on linear color space" in the global rebuttal for details.

2. Request for more ablation studies on MLP and SH: We have conducted experiments on the size of MLP and the degree of SH. Please see Table 2 in the rebuttal PDF. The experiment demonstrates that modifying the parameters of SH does not address the inherent issues, which underperforms MLP across all metrics. Moreover, adjusting the parameters of the MLP has minimal impact, given the small disparity in the metrics. Consequently, transitioning to an MLP is entirely justified.

3. Detailed results on gaussian numbers for w/wo CSI: In some scenes (especially bikes and windowlegovary), the sparse point cloud fails to reconstruct distant points accurately. During training, gaussians tend not to move perpendicular to the camera plane but rather parallel to it. This results in many foreground gaussians attempting to fit the background from different perspectives, as shown in Fig. 4 (b) in our main paper, leading to more foreground split/clone operations and thereby increasing the number of gaussians and leading to worse results in structural reconstruction. CSI is used to add sparse points for distant scenes, and due to the optimization capabilities of 3DGS, in all scenarios, CSI not only did not increase the number of points, but actually reduced them. Besides, CSI is particularly effective in the outdoor scene (-24.86% for bikes and -13.10% for windowlegovary). Details can be found in Table 1 in the rebuttal PDF.

We hope our response addresses your concerns. Here, I would like to reiterate the potential impact of LE3D: real-time rendering for HDR view synthesis is important because it can support more downstream editing tasks, allowing novel view synthesis technology to be applied to VR/AR or post-production in film and television (reframing and postprocessing). As demonstrated in our demo video, our current progress already supports various subsequent processing techniques in near real-time. These techniques bring LE3D closer to practical applications. Details for more about our motivation can be found in the first section of the global rebuttal.

Dear Reviewer PtE5,

We hope this message finds you well. As we approach the culmination of the discussion period, we would like to extend our heartfelt thanks for your insightful feedback and the time you have invested in evaluating our work.

We are keen to ensure that all your queries and concerns have been satisfactorily addressed. Should there be any remaining points that you believe require further clarification or discussion, we kindly urge you to share them with us at your earliest convenience.

It is our sincere hope that our responses thus far have successfully clarified the issues you raised. If you find that your concerns have been resolved, we would be immensely grateful if you would consider adjusting your score to reflect the improvements and the efforts we have made to refine our paper.

Thank you once again for your time and consideration.

Warm regards, The Authors

Thank you for recognizing our structural regularization and the thoroughness of our experiments! Below, we address the specific points.

1. About the typos: thanks for pointing them out! We will definitely fix them in the next version of our paper.

2. About the difference between LE3D and concurrent papers: we have discussed the differences between LE3D and other methods in the global rebuttal part, please refer to details there. The main differences between LE3D and other methods lie mainly in two aspects: a) LE3D can perform blind denoising on the data without the need for noise model calibration, which broadens its range of applications. b) LE3D places more emphasis on structural information, making it more suitable for downstream editing tasks.

3. The tradeoff in FPS/Training time: While our assertion may seem somewhat overstated, we confidently advocate for the consistent selection of LE3D, with the disclaimer that this comparison is made relative to LDR-GS, HDR-GS, and RawGS as discussed in our main paper. The main reasons are as follows: a) Visual quality: As shown in the main paper and supplementary figures, LE3D exhibits less noise compared to other methods (e.g., floaters in Fig. 1, 3) and better color shown in Fig. 13. b) Structural information: LE3D captures structural details better than other methods and much better than RawGS (Fig. 13, 14, 15), which benefits downstream tasks, as demonstrated in the defocus tasks shown in Fig. 5 and 9.

We hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.