GSDF: 3DGS Meets SDF for Improved Neural Rendering and Reconstruction

The paper introduces GSDF, a new approach for both novel view synthesis and surface reconstruction that relies on a dual-branch architecture combining 3D Gaussian Splatting (3DGS) with neural Signed Distance Functions (SDF).

Details:

The paper aims to improve the quality of both the rendering compared to vanilla 3DGS and the reconstructed surface compared to neural SDF approaches like NeuS. To this end, it uses two branches:

1. 3DGS Branch: Inspired by Scaffold-GS, this branch outputs RGB images with splatting-based, rasterized rendering.
2. Neural SDF Branch: Inspired by Instant-NSR, a custom implementation of NeuS using hash grids (similar to Instant-NGP) to accelerate optimization. This branch outputs RGB images using volumetric rendering.

The method establishes mutual guidance and joint supervision between the two representations, relying on three main ideas:

1. Efficient Sampling: Using depth maps rasterized with Gaussian splats to guide the volumetric rendering in the SDF branch, allowing for much more efficient sampling along rays. In practice, the sampling range for a ray $r$ going through a pixel $p$ is computed using the SDF value of the point $x$ resulting from the backprojection of the 3DGS depth map at pixel $p$.
2. Guided Densification and Pruning: Using the SDF branch to guide the densification and pruning processes in the 3DGS representation. The underlying idea is straightforward: Densification should be stronger near the surfaces, and Gaussians far from the surfaces should be pruned. In practice, the paper proposes to compute a criterion depending on both the gradient of Gaussian primitives (similarly to Scaffold-GS) and the SDF value of the center of a Gaussian. The higher the gradient and the closer the Gaussian is to the surface, the more likely it is to be densified. In the meantime, Gaussians located far from the surface are pruned.
3. Geometric Alignment: Aligning geometric properties (depth and normal maps) of both branches to ensure the consistency of the two branches as well as better geometric accuracy of the 3DGS representation, as Gaussians generally tend to cheat on the geometry to allow for better rendering.

The proposed approach limits floaters and ensures better alignment of 3D Gaussian primitives with the underlying surface. The overall geometric quality and details of the 3DGS representation are improved, reaching higher performance than concurrent state-of-the-art methods. Moreover, the paper explains that the 3DGS branch allows for accelerated convergence (and better performance) of the SDF branch thanks to much more efficient sampling along rays.

This paper proposes to jointly optimize 3D Gaussian Splatting (3DGS) and SDF (like NeuS). During GS optimization, the Gaussians are aligned to the zero-level set (and normals) of SDF. During the NeuS-like optimization, Gaussians are used to limit the range of ray sampling, resulting in efficient optimization. Experiments show that the proposed method achieves better reconstruction and rendering compared with SDF-based methods (e.g., NeuS) and GS-based methods, including recent SuGaR and 2D-GS, which explicitly align Gaussians to surfaces.

This paper tackles the challenge of representing 3D scenes from multiview images by introducing a novel dual-branch architecture named GSDF, which combines 3D Gaussian Splatting and neural Signed Distance Fields. This architecture enhances both rendering and reconstruction through mutual guidance and joint supervision. By aligning Gaussian primitives with potential surfaces and speeding up SDF convergence, the method achieves finer geometric reconstructions and minimizes rendering artifacts. Demonstrating robustness and accuracy, the approach is effective in both synthetic and real-world scenarios.

This paper introduces GSDF that utilizes a joint optimization of the GS-branch and SDF-branch to constrain the inherent geometric issues of the 3DGS. Furthermore, it proposes three mutual guidances to ensure satisfactory outcomes in both rendering and reconstruction. The extensive experiments on datasets such as DTU, Tanks and Temples (T&T), and Mip-NeRF 360 demonstrate that the method can achieve high rendering quality while obtaining better geometric results.

We thank all reviewers for their valuable feedback. We are encouraged that reviewers find

• our two-branch design is novel and effective in boosting reconstruction and rendering quality simultaneously;
• our analysis and experiments are useful and comprehensive.

We will release our code for reproduction and future research. Here we address some common concerns.

1. Function of each branch

Note that, our framework features a two-branch design, where each branch excels in its respective task while benefiting from mutual supervision: the SDF-branch specializes in accurate geometry reconstruction, while the GS-branch focuses on high-quality image rendering.

We emphasize that achieving good qualities in both rendering and reconstruction is extremely hard and no recent method can attain this goal to the best of our knowledge. As we described in the paper (line 24-26), a representation alone struggles to achieve good quality in both reconstruction and rendering. Through thorough analysis, we noticed that a naive integration (like guidance merely by loss) hardly balances the learning priority during training, thus, it can only boost the quality of either rendering or reconstruction at the cost of sacrificing the other.

Instead, we dug deeper into the architecture characteristics of the two branches and propose a tight guidance from model architecture, e.g., using the predicted depth from GS-branch to guide the ray sampling of SDF-branch (significantly guide the convergence of SDF branch) and use the SDF field to guide the densification of the GS-branch (geometry-based density control rather than heuristic strategy). The effectiveness is confirmed through comprehensive evaluations, showing improved reconstruction and rendering quality.

2. Time consumption

Inference time remains unaffected as each branch can be used individually.

Regarding training time:

• In each iteration, the training overhead of our two-branch design is higher than single-branch designs (Tab A). However, GSDF achieves better reconstruction quality comparing to the SDF-branch alone when trained for the same amount of time/iterations, as illustrated in Fig A.
• The improvement in both the rendering and reconstruction quality are nonnegligible. The effectiveness of our proposed framework comes from the combination of local and global optimization. By introducing mutual optimization, the training time will inevitably increase. However, without much overhead on training efficiency, we can achieve improvement in both rendering and reconstruction, whilst our design does not have an impact on the inference stage, guaranteeing efficiency in downstream applications.
• The core contribution is our design of the two-branch (geometry + rendering) framework, where each branch can be updated to a more efficient version in the future. We demonstrated the generalizability of our framework by switching the GS-branch from Scaffold-GS to 3DGS, which still exhibited superior rendering and reconstruction quality (Fig D, Tab B)

3. Comparison with Concurrent work

The concurrent work NeuSG aims for an improved reconstruction, which augments the SDF branch by a SDF loss to encourage the SDF value of Gaussian points and MVS points to be 0, and a normal loss to encourage normal consistency between SDF and Gaussian. Extra regularization loss (including flattening the gaussian shape) is introduced to make gaussian more friendly to geometry, regardless of the potential sacrifice in rendering quality.

GSDF has critical differences comparing to NeuSG:

• Instead of only focusing on reconstruction, GSDF aims to improve both geometry and rendering, whose effectiveness has been verified by extensive experiments.
• Beyond loss-based guidance, we investigate the combination inside the model architecture and propose a tightly-coupled two-branch design including depth-guided sampling and SDF-guided densification.
• Moreover, GSDF does not require an extra MVS process to provide accurate geometry, making ours a more versatile method requiring only coarse initialization.

The ratings for this paper include two accepts (LpsC and H3YA), and two borderline accepts (96yp and yTkf). All reviewers acknowledge the effectiveness of the method presented in this paper. The combination of 3DGS and SDF mutually enhances each other, improving both reconstruction and rendering quality. Reviewers H3YA and LpsC have concerns about the paper's contribution and originality. All reviewers raised some technical concerns. The rebuttal addressed most of these concerns, so the AC recommends acceptance.