DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos

1) Why is the opacity map modeled to be independent of car position?

Thank you for pointing out that, according to Fresnel's law, the opacity of the window should also depend on the car's position. However, in experiments, we found that most of occlusions are caused by the objects between dash camera and car window, such as mobile phone holders and stains, rather than the objects outside the car. Since these objects remains relatively still to cars, it is reasonable to design the opacity map to be independent of the car's position. Given the extremely ill-posed nature of this task, we chose to omit the dependence between window's opacity and car's position.

2) The two observations in 3.3 about the static motion prior and vary illuminations of obstructions are too strong.

We acknowledge that the assumptions guiding our design are quite strong. However, this task is highly ill-posed, as evidenced by the suboptimal results of previous methods, such as some reflection removal models trained on paired datasets. These strong assumptions have been proved to be crucial for achieving satisfactory performance. We will explore to relax these assumptions in the future research, enabling adaptation to a wider range of scenarios.

3) Details of $\boldsymbol{\mathcal{L}_{opacity}}$. This question is answered in the global response.

4) Rendering speed. This question is answered in the global rebuttal response.

5) Visualization of 3DGS. We provide a visualization of the generated 3DGS in Figure 1 of the attached PDF. Due to the one-page limit of the attached document, please refer to our supplementary materials for additional visual results.

1) Moderate novelty because the similar components in the proposed method can be found, in part, from other related work.

We appreciate if the reviewer could be more specific about the components. Then we can provide detailed explanations during the discussion period. While both 3DGS and hierarchical hash encoding are tools that have been developed and widely used in related works, the novelty of this paper lies in our effective design of a framework that utilizes physical priors to address this challenging task for the first time.

2) Correctness of equation 4. Thank you for pointing out this typo. The physically correct version of the composition equation should be:

We will correct this typo in the revised version of the paper.

3) Details of the datasets. Thank you for bringing this up. We will include these details in the revised paper.

• The curated BDD100K dataset contains 8 scenes, while the self-collected DCVR dataset includes 10 scenes. Each sequence consists of approximately 300 frames, extracted from 10-second videos at a frame rate of 30 Hz.
• Modern dash cameras provide decent quality. The resolution of the images in BDD100K is around 1296x500. For the DCVR dataset, we used a 4K wide-angle camera (purchased on Amazon, selected based on high sales volume). The original resolution of these images is 3840x2160. We performed camera calibration and undistortion using a chessboard and the OpenCV library before running the SFM algorithm. To facilitate the training process, we downsampled the images 3 times to a resolution similar to that in BDD100K, aligning with popular datasets like KITTI-360 and BDD100K.
• 4 out of 8 scenes in BDD100K contain moderate dynamic objects. No scenes in DCVR contain dynamic objects.
• As shown in Fig. 7 in the paper, our dataset includes opaque occlusions. Our dataset does not contain car hoods, which we will discuss later.
• 3 out of 8 scenes in BDD100K contain turning. 4 out of 10 scenes in DCVR contain turning. We visually depict the turning trajectories in Fig. 2 of the attached PDF.

4) How well the proposed method works for the reflective car hoods? Thank you for bringing up the topic of car hoods. While car hoods are indeed prone to strong reflections, they typically occupy only around 10% of the lower parts of the images, which do not contain much information about the street scenes. Additionally, car hoods are consistently located in the same position in each video. We addressed this issue by manually cropping out the areas containing the car hoods during the dataset curation process. We believe that this can still be done on a large scale, as segmenting car hoods is not a particularly difficult task.

5) Limitations. Limitations are discussed in the global response.

Thanks for your answers to my questions. I am glad to know more details about the evaluated data. These evaluated data are sufficiently diverse to demonstrate the improvements brought by the proposed method.

I am happy with the authors' response and will remain or possibly raise my score later.

One additional concern I had is similar to the Reviewer si9q's W1, the proposed method is practically limited to the car-mounted dash cam, but it doesn't address the obstruction challenge from the general scenes. This is still a key limiting point.

1) Details of $\boldsymbol{\mathcal{L}_{opacity}}$. The formulation of opacity loss is shown in the global response. The opacity map is learned in a self-supervised way without relying on additional labels.

2) Why only compare with general Novel View Synthesis methods?

As the first work addressing the novel task of NVS for dash cam videos, there are no prior works specifically designed for this task. As a result, we compare ours with several SOTA methods. NeRF-W is designed for NVS using images captured with different cameras and at different times but still cannot handle obstructions like reflections on the windshield. We evaluate NeRF-W on the dash cam videos, and as shown in the table below, our method significantly outperforms NeRF-W in both synthesis quality and rendering speed.

Table 2. Comparison with NeRF-W. Tested on BDD100K

Furthermore, as illustrated in Figure 4 in the attached pdf, although NeRF-W is designed to handle illumination variance and transient objects, it fails to separate obstructions from the images. None of the obstructions are accurately represented in the transient image. This suboptimal result occurs because the obstructions move with the camera rather than being transient parts of the images. The reflections and transmissions are intertwined in a complex manner that NeRF-W's design cannot effectively address, unlike our method.

3) Novelty of G3E. We use the standard geometry consistency filter from PatchMatchNet to improve novel view synthesis. Some works also leverage MVS priors to help NVS task. However, simply applying PatchMatchNet to original input images does not yield optimal results. In our method, we first run DC-Gaussian without multi-view stereo (MVS) initialization. We then use the trained model to synthesize $\hat{I}_t$, as explained in Equation 7. This strategy effectively suppresses obstructions in the original images and enhances overall performance. In Fig 8, we compare the results using MVS initialization (a w/ G3E) and that using SfM initialization (b w/o G3E), we will make it more clear in revised paper.

4) Limitations. The limitations are discussed in the global response.

We value your feedback and are eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.

As the deadline approaches, I am eagerly anticipating your response.

1) Concern about the practical value of using dash cam videos for Novel View Synthesis

Dash cam videos have unique values for autonomous driving. Dash cam videos deeply reflect the diversity and complexity of real-world traffic scenarios. They are used to provide large-scale, diverse driving video datasets in a crowd-sourced manner [1]. Dash cam videos also offer important data sources about multi-agent driving behaviors [2] and evaluating the robustness of algorithms under visual hazards [6]. BDD100K [1], a large scale diverse driving videos dataset built with dash cam videos, has been cited more than 2000 times. It has been widely used in many important vision tasks, such as Object Detection, Semantic Instance Segmentation [8], and Multiple Object Tracking and Segmentation [7], etc. This fact shows the huge interests of vision community in dash camera studies.

Although most dash cam videos are monocular and prone to obstructions, we believe studying how to leverage the important data source is quite important to many vision tasks, such as novel view synthesis and 3D reconstruction. Moreover, our work is an attempt to tackle obstruction removal from dash cam videos, making it possible to achieve high-quality NVS from dash cam videos. It will provide huge 3D information for autonomous driving.

[1] Yu F, Chen H, Wang X, et al. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. CVPR, 2020.

[2] Chandra R, Wang X, Mahajan M, et al. Meteor: A dense, heterogeneous, and unstructured traffic dataset with rare behaviors. ICRA, 2023.

[3] Grand View Research. Dashboard camera market size, share & trends analysis report by technology (basic, advanced, smart), by product, by video quality, by application, by distribution channel, by region, and segment forecasts, 2024 - 2030, 2023. Accessed: 2024-05-16.

[4] Martin-Brualla, Ricardo, et al. "Nerf in the wild: Neural radiance fields for unconstrained photo collections." CVPR, 2021.

[5] Gao, Chen, et al. "Dynamic view synthesis from dynamic monocular video." ICCV, 2021.

[6] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402–416, 2018.

[7] Luiten, Jonathon, et al. "Hota: A higher order metric for evaluating multi-object tracking." International journal of computer vision 129 (2021): 548-578.

[8] Yan, Bin, et al. "Universal instance perception as object discovery and retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

2) Rendering speed. This question is answered in the global rebuttal response.