D$^2$GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstruction

Dear Reviewer GUm1,

We sincerely thank Reviewer GUm1 for the insightful review and valuable feedback. In this rebuttal, we provide new experiments using a monocular metric depth estimator to demonstrate our framework's robustness, alongside detailed clarifications on the motivation for our key design choices.

1. W1 & W4: What is the motivation of the current depth initialization method? And what about alternative monocular metric depth estimation methods?

We thank you for the important questions regarding our motivation for choosing depthsplat-based MVS as our initialization method.

From our perspective, existing depth estimation methods still produce imperfect results, making them unsuitable for direct use as ground truth in high-quality reconstruction tasks. As we argued in L51-55, monocular depth estimation methods typically generate relative depth with an unknown scale, while stereo-based methods struggle with dynamic scenes. Therefore, we aim to design a framework to take the imperfect, metric depth maps as a starting point and jointly optimize both the scene geometry and the depth, which is also our core contribution. We choose metric depth specifically because it is essential for initializing point clouds at the correct scale and for enabling strong, direct, and multi-view consistent geometric supervision during training.

Second, with the rapid development of depth foundation models, recent works such as UniDepth and Metric3D as you suggested, have achieved impressive performance in metric depth estimation. We adopt a DepthSplat-like MVS approach mainly because the camera poses are known in our setting. However, we emphasize that our framework is orthogonal to the choice of initial depth source and can be generalized to any method that provides a reasonable metric depth initialization, whether it originates from MVS or monocular metric depth estimation.

Here, we provide experiments using the state-of-the-art monocular metric depth estimator, **UniDepthV2 $1$ **, to initialize our pipeline instead of MVS. The results on the Waymo dataset are as follows:

Tab. A: Supplementary comparison to Tab. 1 with quantitative results of different initialization and training methods. These results are based on 8 sequences from the Waymo dataset due to the time limitations of the rebuttal phase. We will provide the full results in the revised manuscript.

As shown in Tab. A, the results demonstrate that our framework can effectively leverage and refine the initial depth from a monocular method. This confirms that our proposed optimization pipeline can be generalized to other specific choice of initializer. However, the performance improvement of ours + UniDepthV2 is less significant than that of ours + DS, compared to the OmniRe counterpart. We infer that this is because UniDepthV2 provides more accurate depth estimation, leaving less room for further improvement by our method. We will add these experiments and this discussion to our revised manuscript.

$1$ Piccinelli, Luigi, et al. "Unidepthv2: Universal monocular metric depth estimation made simpler." arXiv preprint arXiv:2502.20110 (2025).

2. W2: What is the difference between our Progressive Pruning method and ADC in vanilla 3DGS?

Thank you for this question, which allows us to clarify the critical role of our Progressive Pruning component and its fundamental differences from the native pruning within 3DGS.

• Distinct Goals:
  1. The goal of ADC in vanilla 3DGS is to refine rendering quality during the optimization by splitting Gaussians in high-gradient areas and removing those with near-zero opacity. In contrast, our Progressive Pruning component is a pre-processing step designed to (1) ensure computational feasibility by efficient point pruning and (2) establish a geometrically accurate and compact starting point for the main training.
• Why Progressive Pruning is Necessary:

1. Computational Feasibility: Initializing from dense depth maps generates an enormous number of points. Attempting to load this point cloud into GPU memory for standard training pipeline may cause a GPU OOM error at the very first iteration. The experiment results in Table B show that loading 15 millions of points leads to the model maintaining around 6 million points during training that cannot be effectively pruned, indicating significant point redundancy and a slowdown in training speed.
2. Geometric Distillation: More importantly, Progressive Pruning is a scheduled, multi-stage pre-training process that acts as a Geometric Distillation mechanism. First, this process operates on the massive point cloud by completely disabling densification operations like splitting and cloning. The learning objective is focused exclusively on refining the geometry. Within this simplified framework, a Gaussian's opacity is optimized to reflect its consistency with the geometric constraints. Second, the progressive filtering is guided by a final target point count, not a fixed threshold. As mentioned by L344 in our Supplementary Material, in each stage, we rank all Gaussians by their learned opacity and prune those that contribute least to a coherent geometric structure. This adaptive, ranking-based selection ensures that a compact and representative set of Gaussians is preserved at a specific amount.

Table B: Analysis of computational cost, comparing our Progressive Pruning method with a baseline using a manageable number of points to demonstrate the efficiency gains.

3. W3 & Q1: How does this paper handle dynamic objects?

Thank you for pointing out the lack of clarity regarding our handling of dynamic objects. We will add a dedicated section in the final manuscript to elaborate on these technical details. In response to your specific questions:

• Dynamic Object Modeling: Our approach to modeling and decomposing dynamic objects follows our baseline, OmniRe, by utilizing a Scene Graph. Each dynamic object is represented as an independent node within this graph, possessing its own set of Gaussian models (e.g., dynamic or rigid representations).
• Depth Estimation for Dynamic Objects: We acknowledge that initial depth estimates derived from MVS will inevitably contain errors for moving objects, a common challenge in depth estimation. First, since our method uses given camera poses from Waymo, dynamic objects do not affect pose estimation. Depth errors are therefore localized to these dynamic areas, with minimal impact on the static background. Furthermore, during the iterative optimization of the Gaussians and the Depth Enhancer, the depth of dynamic objects is further refined. Specifically, this refinement is driven by both the photometric loss from Gaussian splatting and the accurate geometric supervision provided by our diffusion-based Depth Enhancer. This joint optimization loop is equally effective for both static and dynamic scene components, allowing it to effectively correct initial errors and achieve accurate depth prediction

We believe a comprehensive visualization of the depth refinement process would best illustrate the optimization of depth for dynamic objects. As requested by Reviewer e8Wz, we will include such visualizations and a supplementary explanation in the final manuscript to demonstrate the rationale and effectiveness of our method.

4. Q2: How does this paper handle the sky?

Thank you for this observation. We follow the standard practice used in OmniRe to model the sky. Specifically, we use a pre-processed sky-mask for calculating opacity loss, encouraging Gaussians in the sky region to become fully transparent. The appearance of the sky itself is handled by a separate model (e.g., a cubemap representation), which is decoupled from the 3D scene Gaussians. We will clarify this in the implementation details of our final manuscript.

Dear Reviewer e8Wz,

We sincerely thank Reviewer e8Wz for the insightful review and valuable suggestions. In this rebuttal, we clarify our motivations and provide comprehensive ablation studies to quantitatively analyze each component's impact.

1. W1 & Q1: Could you explain why dense depth estimation is used for initialization instead of SfM point clouds?

Thank you for this important question. We acknowledge that SfM is a common initialization strategy for many image-only GS methods. Our decision to use dense depth maps from a MVS approach instead of SfM was based on the following considerations for our specific problem setting:

1. Availability of High-Quality Camera Poses: In our primary dataset (Waymo), high-quality, pre-calibrated camera poses are already provided. A key function of SfM is to solve for these poses from images, which is not only redundant in our case but also computationally intensive, especially for large-scale urban scenes with lots of images.
2. Requirement for Metric Depth: Our method is designed to leverage and refine dense geometric priors with a correct metric scale. Traditional SfM pipelines typically produce point clouds that have an arbitrary scale, which would require an additional, non-trivial step to align to a metric scale. Using an MVS-based method with known camera poses provides a more direct and efficient path to the dense, metric depth maps that serve as a strong geometric starting point for our framework.
3. Orthogonality to Our Method: our framework is orthogonal to the choice of initial depth source and can be generalized to any method that provides a reasonable metric depth initialization.

In future work, we plan to extend our method to a camera-pose-free setting. In that scenario, integrating a pose estimation module, such as a learning-based method or a traditional SfM approach, would be a necessary and valuable first step, followed by joint optimization of the poses during reconstruction. We will clarify this motivation in the revised manuscript.

2. W2 & Q2: The independent ablation for two different loss functions of Road Node.

Thank you for this valuable suggestion. We agree that a more detailed ablation is needed to demonstrate the individual contributions of the two loss functions for the road node. We have performed this additional ablation study as follows.

Tab. A: Ablation study of the two loss functions for the road node. These results are based on 8 sequences from the Waymo dataset due to the time limitations of the rebuttal phase. We will provide the full results in the revised manuscript.

As shown in Tab. A, both the loss $L_{plane}$ and the loss $L_{shape}$ contribute positively to a more accurate geometry (evaluated with depth results). The best performance is achieved when both are used together. We will include these quantitative results and a corresponding qualitative analysis in the final version of the manuscript.

3. W2 & Q3: The ablation study only reports depth-related metrics. Could you report the impact of each component on novel view synthesis (NVS) performance?

Thank you for this excellent suggestion. We completely agree that novel view synthesis (NVS) metrics are crucial for evaluating scene reconstruction quality. Our original ablation study emphasized geometric metrics because the ill-posed nature of sparse-view reconstruction can be misleading, and models can easily overfit to training views, yielding high PSNR scores that don't reflect true geometric accuracy. We now have updated our main ablation study to include PSNR, SSIM, and LPIPS, and will add the following table to the manuscript.

Tab. B: Image-level performance of all ablation study results in our original manuscript.

As shown in the Tab. B, each of our proposed components contributes to the final performance. When combined with Table 3 in the main manuscript, the results show that, in most cases, poorer depth quality leads to inferior NVS performance. Notably, the 'Periodic Update' variant shows a significant drop in NVS performance compared to our full model, indicating it has overfitted to the training views' photometry with the out-dated depth information. This result underscores the importance of our proposed iterative refinement scheme for achieving both geometric accuracy and high-quality novel view synthesis.

4. Q4: Could you provide visualizations of the estimated depth and rendered depth at different stages to demonstrate the effect of online updates?

This is an excellent suggestion. Visualizing the depth evolution process provides a clear and intuitive demonstration of our method's core mechanism. Due to the rebuttal format restrictions, we cannot include new figures here. However, the method figure in our main manuscript (Figure 2) already provides a snapshot of a single update step, showing the rendered depth from the Gaussians (D.E. input) and the refined depth map (D.E. output). This figure illustrates how our method corrects geometric inconsistencies like floating artifacts in a single step. We will add a new, dedicated figure in the final manuscript to show this process over multiple training stages. This visualization will clearly demonstrate how the depth maps are progressively refined from an initial noisy state to a coherent and accurate representation, powerfully illustrating the online update process.

Dear Reviewer Cnm4,

We sincerely thank Reviewer Cnm4 for the thorough review and constructive feedback. In this rebuttal, we provide new experimental results on the nuScenes dataset as you suggested, comprehensive ablation study results, and detailed clarifications on our method for handling dynamic objects.

1. W1. Validating the proposed method on nuScenes would strengthen the paper’s contribution and make the results more convincing.

We completely agree with your assessment. Adding a new dataset is essential for validating the generalization of our method. Testing our method in scenarios with greater LiDAR noise is indeed an excellent way to highlight its effectiveness. However, while such data is common in industry, it is rarely available in public datasets. We thank you for suggesting the nuScenes dataset and for noting its less accurate LiDAR calibration. Accordingly, we have extended our method to nuScenes and compared it against OmniRe-based approaches. The experimental results are summarized below.

Tab. A: Quantitative comparison of our method against OmniRe-based approaches on the nuScenes dataset. We select 8 sequences from the nuScenes dataset (two of which are rainy scenarios) due to the time limitations of the rebuttal phase. We will provide the full results in the revised manuscript.

As the results show in Tab. A, our method achieves better performance in both image reconstruction and novel view synthesis compared to the LiDAR-free baseline. Moreover, our method remains comparable to the LiDAR-based OmniRe in reconstruction and novel view synthesis. We attribute this to the fact that while the LiDAR data in nuScenes is considered less accurate than Waymo's, it is a mature and widely-used dataset, and its errors are still relatively minor compared to those from image-based depth estimation methods. Due to the NeurIPS rebuttal limitations, we are unfortunately unable to provide visualizations at this time. We will add the experiments on nuScenes, along with corresponding qualitative and quantitative results to the final version of our manuscript to fully demonstrate our method's performance across different datasets and more challenging scenarios.

2. Q1. How does this paper handle dynamic objects?

Thank you for pointing out the lack of clarity regarding our handling of dynamic objects. We will add a dedicated section in the final manuscript to elaborate on these technical details. In response to your specific questions:

• Dynamic Object Modeling: Our approach to modeling and decomposing dynamic objects follows our baseline, OmniRe, by utilizing a Scene Graph. Each dynamic object is represented as an independent node within this graph, possessing its own set of Gaussian models (e.g., dynamic or rigid representations).
• Depth Estimation for Dynamic Objects: We acknowledge that initial depth estimates derived from MVS will inevitably contain errors for moving objects, a common challenge in depth estimation. First, since our method uses given camera poses from Waymo, dynamic objects do not affect pose estimation. Depth errors are therefore localized to these dynamic areas, with minimal impact on the static background. Furthermore, during the iterative optimization of the Gaussians and the Depth Enhancer, the depth of dynamic objects is further refined. Specifically, this refinement is driven by both the photometric loss from Gaussian splatting and the accurate geometric supervision provided by our diffusion-based Depth Enhancer. This joint optimization loop is equally effective for both static and dynamic scene components, allowing it to effectively correct initial errors and achieve accurate depth prediction

We believe a comprehensive visualization of the depth refinement process would best illustrate the optimization of depth for dynamic objects. As requested by Reviewer e8Wz, we will include such visualizations and a supplementary explanation in the final manuscript to demonstrate the rationale and effectiveness of our method.

3. Q2. Why purely compare the estimated depth directly against ground-truth LiDAR, as shown in Table 2 and Table 3?

This is a very insightful and reasonable point. We agree that using LiDAR as ground truth for depth evaluation can introduce errors, especially in real-world applications where LiDAR data may be faulty. However, for the Waymo dataset, we believe use as a reference is justified for two reasons:

1. The LiDAR data in the Waymo dataset is meticulously calibrated and is widely recognized in the academic community as a high-quality dataset.
2. The potential errors in the Waymo LiDAR data are minor compared to the errors produced by image-based depth estimation algorithms. For instance, in most regions, LiDAR typically produces centimeter-level errors.

Therefore, adopting Waymo's LiDAR as an "upper-bound” to evaluate the relative performance of different LiDAR-free methods is meaningful. Although the Waymo dataset cannot clearly reveal the impact of LiDAR calibration errors on the accuracy of reconstruction, it can still prove the effectiveness of our method in LiDAR-free setting, i.e. our method achieves performance comparable to LiDAR-based approaches and significantly outperforms other LiDAR-free methods.

4. Q3. Adding an ablation table showing image recon/synthesis metrics of removing each proposed component?

Thank you for this excellent suggestion. Regarding the results of a DepthSplat baseline without our Depth Enhancer, they are presented in Tab. 1 and Tab. 2 of the original manuscript. Additionally, we fully agree that providing image-level metrics in our ablation study is important for evaluating scene reconstruction and novel view synthesis quality. Our original ablation study emphasized geometric metrics because the ill-posed nature of sparse-view reconstruction can be misleading, and models can easily overfit to training views, yielding high PSNR scores that don't reflect true geometric accuracy. We now have updated our main ablation study to include PSNR, SSIM, and LPIPS, and will add the following table to our manuscript.

Tab. B: Image-level performance of all ablation study results in our original manuscript.

As shown in Tab. B, the results indicate our full method achieves the best performance compared to variants where other components are removed, underscoring the effectiveness of our proposed modules. When combined with Table 3 in the main manuscript, the results show that, in most cases, poorer depth quality leads to inferior NVS performance. Notably, the 'Periodic Update' variant shows a significant drop in NVS performance compared to our full model, indicating it has overfitted to the training views' photometry with the out-dated depth information. This result underscores the importance of our proposed iterative refinement scheme for achieving both geometric accuracy and high-quality novel view synthesis.

Dear Reviewer eDMK,

We sincerely thank Reviewer eDMK for the insightful review and valuable feedback. In this rebuttal, we provide new experimental results on the nuScenes dataset, a detailed computational analysis of our pruning method, and a discussion on addressing the camera pose dependency.

1. Q1. Performance of your method on other datasets, especially with noisier data or more difficult weather conditions.

Thank you for this question. We agree that evaluating on a new dataset is necessary to verify the generalization of our method, and testing on a dataset with noisier LiDAR data would better highlight our method's effectiveness. Regarding a "noisier dataset", another reviewer (Cnm4) suggested we test on the nuScenes dataset. We believe this also aligns perfectly with your suggestion. Consequently, we have extended our method to nuScenes and compared it with OmniRe-based approaches. The results are presented in the table below.

Tab. A: Quantitative comparison of our method against OmniRe-based approaches on the nuScenes dataset. We select 8 sequences from the nuScenes dataset (two of which are rainy scenarios) due to the time limitations of the rebuttal phase. We will provide the full results in the revised manuscript.

As the results show in Tab. A, our method achieves better performance in both image reconstruction and novel view synthesis compared to the LiDAR-free baseline. Moreover, our method remains comparable to the LiDAR-based OmniRe in reconstruction and novel view synthesis. We attribute this to the fact that while the LiDAR data in nuScenes is not as high-quality as that in Waymo's, it remains a mature and widely-used dataset, and the discrepancy between its point clouds and the ture depth is still relatively minor compared to the errors in depth estimation. Due to the NeurIPS rebuttal limitations, we are unfortunately unable to provide visualizations at this time. We will add the experiments on nuScenes, along with corresponding qualitative and quantitative results to the final version of our manuscript to fully demonstrate our method's performance across different datasets and more challenging scenarios.

2. Q2. Detailed computational complexity regarding our Progressive Pruning method.

Thank you for this question and for recognizing the effectiveness of our Progressive Pruning component. The fundamental reason we proposed this strategy is that initializing from a dense depth map would involve an enormous number of back-projected points (approx. $cams×timestep×H×W$). Using these points directly for Gaussian training would be computationally redundant. Furthermore, an excess of low-accuracy points would lead to artifacts during subsequent training. Therefore, Progressive Pruning is not only a method for optimizing computation but also for providing a higher-quality initial point cloud to improve reconstruction results. Regarding your specific question on computational efficiency, we have profiled the resource usage with different numbers of retained points.

Table B: Analysis of computational cost, comparing our Progressive Pruning method with a baseline using a manageable number of points to demonstrate the efficiency gains.

As shown in Tab.B, when the initial input point cloud contains as many as $1.5×10^7$ points, the gaussian model will maintain around $6×10^6$ points that cannot be effectively pruned. In contrast, our Progressive Pruning method prevents a significant waste of computational resources and avoids efficiency bottlenecks.

3. Q3. Do you think still requiring accurate camera pose knowledge might be an issue? Do you have ideas to mitigate the impact of pose estimation on the reconstruction result?

Thank you for raising this point; we consider it to be very important. We acknowledge that accurate camera calibration is also a practical challenge in the autonomous driving scenarios. However, our method relies on accurate camera poses, which is a common assumption in the field of street view reconstruction and also a prior knowledge in existing datasets. Therefore, in this work, we mainly aim to address problems related to the LiDAR data modality, such as 1) the absence of LiDAR data on vision-only autonomous vehicles, 2) projection errors from different mounting views, and 3) the inaccuracy of LiDAR calibration.

Regarding how to mitigate the need for accurate camera poses, we have the following thoughts for future work. In the absence of precise poses, we could integrate pose estimation algorithms, such as learning-based methods (e.g., VGGT) or traditional SfM methods (e.g., COLMAP), to obtain initial camera poses. Subsequently, a joint optimization module for camera poses could be used during reconstruction, enabling simultaneous refinement of both camera poses and the reconstructed scene. We will add this discussion to the final version of our manuscript to more comprehensively explain the practical implications of our work and future directions.

4. W1. The code is not made available.

Finally, regarding your comment on code availability, we commit to making our code publicly available upon acceptance of the paper.

Thank you again for your thorough review and constructive comments. They have been invaluable in helping us improve the quality and completeness of our manuscript.