3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction

We thank the reviewer for recognizing the improved depth and planar mesh quality outputs of our method compared to baselines.

Confus[ing] application areas. Our method is motivated by the goal of jointly improving photometric and geometric accuracy in indoor scenes, which often feature large, weakly textured planar surfaces. Methods such as 3DGS and 2DGS, which rely solely on photometric optimization without explicit geometric priors, struggle to recover accurate geometry in such regions, particularly under specular or low-texture conditions. For example, see Figure 3 in DN-Splatter, which concludes that The Gaussian based methods Splatfacto, SuGaR, and 2DGS are trained on only photometric losses and thus severely struggle to capture the scene geometry in low texture environments.

Q: Performance drop in novel view synthesis (NVS). As shown in Figures 4 and 5, our method achieves slightly lower RGB rendering quality than 3DGS-MCMC but outperforms 3DGS, while significantly improving depth accuracy. Depth serves as a strong indicator of geometric quality.

This leads to a key question: does poor geometry affect NVS performance? The answer is yes. Inaccurate geometry can degrade rendering from out-of-distribution viewpoints noticeably (those not well-covered by training views). We show this in our supplementary video. This issue is well-documented in the literature [3, 4], which highlights how weak geometry can lead to rendering artifacts under more aggressive viewpoint shifts from training views, than is usually present in NVS test sets of common datasets.

Q: Only reconstructs planar areas for surface reconstruction. While we explicitly apply geometric priors only to planar regions, we can still extract full-scene meshes using the same strategy as 2DGS. As shown in the Incomplete Mesh Extraction Evaluation section (see Response to Reviewer bJrA), overall our method produces more accurate meshes than all baselines, demonstrating the benefit of incorporating planar priors. Figure 6 presents planar-only mesh results to allow for a fair comparison with prior plane-based reconstruction methods, which typically report metrics solely for planar regions.

Unfair comparison to 2DGS. The reviewer raises an important point: incorporating planar priors does significantly improve geometric accuracy, especially in depth estimation. This is indeed a core contribution of our method: accurately detecting and fitting 3D planes, which vanilla 2DGS lacks and thus struggles in planar regions.

Importantly, our hybrid design is not simply combining 2D and 3D Gaussians everywhere. We use 2D Gaussians exclusively in the detected 3D planar regions where geometry is known, and reserve 3D Gaussians for non-planar areas without priors, where their higher expressiveness helps photometric reconstruction. This separation is deliberate: 2D Gaussians are compact and accurate when structure is known, while 3D Gaussians handle unconstrained regions. Further, while we implemented our solution on top of 3DGS (i.e. non-planar splats are 3D gaussians), there is no reason for which what we proposed could be implemented on top of a 2DGS backbone (i.e. non-planar splats would be 2D gaussians).

As for regularization-based methods like StructNeRF, these are not directly applicable, as with the advent of 3DGS we moved NVS training workloads away from ray-based implementations and towards tile rasterizers. Further, while StructNeRF imposes the planar consistency loss within any superpixel, our technique only activates the planar optimization if a likely planar structure has been detected by RANSAC. While a direct comparison is not possible due to the ray vs. tile implementation, considering the annotations as “candidates” allows for better handling of false positives. We will clarify this in the paper.

Generalization to different camera models. Our method is not uniquely dependent on PlaneRecNet. PlaneRecNet is used as a more automatic prompt generation alternative for SAMv2 compared to manual prompting. SAMv2 performs the actual segmentation. While PlaneRecNet is limited by its training on ScanNetV2 to one camera model, our generalization comes from the robustness of SAMv2. In fact, the results on ETH3D in this rebuttal use manual prompts via a user-click UI to SAMv2, showing strong performance even without PlaneRecNet.

Answer to Q3 (Method details). Each 2D Gaussian is bound exactly to a detected plane, as defined in Eqs. (1) and (2). In line 182, we compute two distances: the residual distance from a 3D Gaussian to the plane (z-axis), and the in-plane (xy) distance to the nearest 2D Gaussian. This helps decide whether to reassign nearby 3D Gaussians without growing the plane indefinitely. We agree that terms like “freeform” and “planar” could be confusing and will revise the subsection title accordingly.

Answer to Q4 (Method details). Yes, M corresponds to several planar masks for a particular 3D plane from multiple views. The 2D masks are propagated from one view or multiple user clicks using the SAMv2 video model.

Bibliography

• [3] Warburg, Frederik, et al. "Nerfbusters: Removing ghostly artifacts from casually captured nerfs." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
• [4] Goli, Lily, et al. "Bayes' rays: Uncertainty quantification for neural radiance fields." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thank you for recognizing our contribution in designing the hybrid 2D/3D representation and its ability to produce compelling results on the ScanNet dataset. We will improve the image quality and figure annotations in the camera-ready version.

Evaluation on outdoor scenes. We focus on indoor scenes, where large, weakly textured planar surfaces pose challenges for 2DGS and 3DGS methods, making ScanNet a suitable benchmark. However, we agree that demonstrating generalization to other datasets including outdoor settings is valuable. As suggested by Reviewer bJrA, we evaluate our method on ETH3D dataset, which includes two outdoor scenes containing planar structures such as roads and building facades. These results show our method remains effective beyond indoor environments. Please refer to our response to Reviewer bJrA under Insufficient Experimental Validation (i.e. our new experiments) for details.

Reliance on 2D segmentation. We recognize the reliance of our method on 2D segmentation as a limitation. However, thanks to recent progress in segmentation models (e.g., SAMv2), this requirement is becoming increasingly robust. These models keep the human in the loop through sparse but valuable point prompts, which come at low cost to the user and contribute to more reliable outputs.

Metrics for Mip-NeRF 360? Please note that the metrics are already reported in the paper within the teaser.

Thank you for recognizing our novel hybrid representation and robust 3D plane fitting. We address your concerns below.

Outdoor scenes. Our method is primarily motivated by scenes containing large, weakly textured planar surfaces, a common failure case for both 2DGS and 3DGS. Such surfaces are especially prevalent in indoor environments, which is why our main evaluation focuses on indoor datasets like ScanNet++. To demonstrate generalization to outdoor settings, we also evaluate two outdoor scenes from ETH3D, which contain roads and building facades with planar geometry. Additionally, our teaser shows the results on Mip-NeRF 360 outdoor scene with rendering quality metrics. These results show that our method remains effective beyond purely indoor environments. Please refer to our response to Reviewer bJrA under Insufficient Experimental Validation (i.e. our new experiments) for details.

Marginal improvement in visual quality. We agree that the RGB renderings in Figures 1 and 4 show only modest visual differences a-la glance. However, these comparisons are best understood alongside their rendered depth maps, which reveal the underlying geometry.

In both figures, methods without our planar prior tend to explain away specular highlights (e.g., the shiny table in Figure 1 and Figure 4) and low-texture regions (walls and floors in Figure 4) using volumetric effects instead of modeling the simple planar geometry paired with a more complex specular appearance or a flat texture.

This incorrect geometry can noticeably degrade rendering quality when viewed from slightly out-of-distribution angles, beyond the well-covered test views, as shown in our supplementary video. As noted in Line 230 of the paper, our approach strikes a better balance between geometric accuracy and photometric fidelity: we preserve correct planar structure due to our geometric prior while still capturing complex appearance effects such as specularity.

We thank the reviewer for their point about figure resolutions, we will include figures at full resolution in the camera-ready version (note that we have full resolution results in the supplementary).

Mesh accuracy metrics. Our method achieves higher accuracy because the predicted points lie closer to the ground truth surface. However, it has lower completeness since some planar regions (e.g., undetected or unlabeled planes) are missing. In contrast, the Airplanes method covers more surface area, but with less precision. Chamfer Distance reflects both of these aspects, so our conservative predictions which favors correctness over coverage leads to higher Chamfer distance on iPhone scenes.

We thank you for acknowledging the novelty of our hybrid 2D+3D Gaussian representation. Below, we clarify the motivation behind our design and provide additional evidence on how our method achieves improved geometric accuracy.

Limited justification for hybrid approach. Our primary motivation is to achieve both photometric and geometric accuracy in indoor scenes, which often contain large, weakly textured planar surfaces. Optimizing purely for photometric loss and without geometric priors, as done in 3DGS and 2DGS, often fails to capture correct geometry in these areas, particularly with specular or uniform textures like walls (e.g. see DN-Splatter [1] within which the caption of Figure 3 clearly states that “The Gaussian based methods Splatfacto, SuGaR, and 2DGS are trained on only photometric losses and thus severely struggle to capture the scene geometry in low texture environments").

Q: Why not use depth (e.g. DN-Splatter) or normal supervision? Note that DN-Splatter assumes depth input from a sensor, not only rendering the problem of 3D reconstruction significantly easier, but limiting applicability with data acquired by a limited class of devices. More specifically, only Apple has continued shipping LiDAR sensors in consumer smartphones, with their top-of-the-line “Pro Max” trim carrying them. Samsung, back in 2019–2020 experimented with rear-facing time‑of‑flight (ToF) sensors on some Galaxy S10 5G, S20, and S20+ models, but these were not Apple‑style LiDAR and Samsung later discontinued them. We argue that investigating techniques that do not rely/require depth input is critical, as it allows a broader consumer base to reconstruct 3D structures with photometric supervision alone.

Monocular depth predictors are a possible alternative, but they can only produce view-inconsistent results, limiting their effectiveness for supervision. For example, the recent Video Depth Anything [2] states how depth consistency in video still presents a challenge, and non-sequential multi-view data (a less structured generalization of video input) is not even considered.

In contrast, our method adds a simple but strong planar prior, requiring minimal user input and leveraging RANSAC to robustly detect and propagate 3D plane assignments. For planar regions, where the geometry is already well-defined, this offers a lightweight alternative tailored to the dominant structures in the indoor scenes.

Q: Why not use only 2DGS? While 2DGS offers improved geometry over 3DGS, it does so at a significant photometric cost. For example, as shown in Figure 4, 2DGS drops PSNR by ~1.7dB, while our hybrid method limits the drop to ~0.2dB, while simultaneously achieving superior depth quality. 3D Gaussians better capture photometric detail and complex geometry, while 2D Gaussians are ideal for modeling planar regions. Our hybrid design combines these strengths, yielding better geometry on flat surfaces without significantly compromising overall image quality. The remaining slight drop in performance wrt. 3DGS (0.2dB) is caused by the limited expressive power of spherical harmonics, as once one imposes a “surface” (vs. volume) reconstruction, complex view-dependent effects cannot any longer be represented by volumetric effects.

Insufficient experimental validation (i.e. our new experiments). Our experimental evaluation focuses on both photometric accuracy (rendering color error) and geometric accuracy (rendered depth error), which are direct outputs of general-purpose 3D reconstruction pipelines. These metrics are reported on ScanNet++, a dataset well aligned with our goal of accurate indoor scene reconstruction. Tangentially, we include mesh accuracy comparisons to prior plane-based 3D reconstruction methods, which generate meshes for the planar parts of a scene. However, we acknowledge the value of extracted mesh accuracy comparisons with non-planar baselines as additional evidence, and thank the reviewer for the suggestion.

Among the suggested datasets, only ETH3D features structured planar geometry, while Tanks and Temples and DTU are primarily “object-centric” and less relevant to our use case. DTU consists of small objects that do not contain a sufficient number of planes to make our method beneficial. Tanks and Temples scenes contain not enough scenes of interest as it is mostly focused on complex non-planar geometry.

Due to the limited time and compute budget, we provide additional mesh accuracy results on a selection of scenes from ETH3D (using the same mesh extraction strategy as 2DGS). The outdoor scenes were selected to contain parts of the roads and buildings to show the generalization of the method; they are also sparse (<40 train images) and unbounded. The selected indoor scene is similar to the most challenging scene from ScanNet++ we evaluated on in the paper to show generalization on indoor scenes. Note that we cannot evaluate DN-Splatter on ETH3D, as in this dataset the provided depth maps are used to create the ground truth mesh.

Our hybrid method yields competitive geometry results in these scenes, with higher photometric accuracy, further validating the benefit of incorporating planar priors. We leave the fusing of the detected planar geometry into the mesh for future work, but this would simply further improve the mesh quality results.

Electro (outdoor)

Terrace (outdoor)

Delivery area (indoor)

• Electro the most challenging as it contains sparse unbounded views of the cloudy sky, where Gaussian Opacity Fields struggles to extract a correct mesh, while 2DGS mesh extraction uses depth truncation, so it filters out the sky.
• Terrace the geometry is dominated by a long building in which the default depth truncation parameters for 2DGS and our method cut off the geometry, while Gaussian Opacity Fields mesh extraction extracts that, yet not precisely, while the planar geometry in our method is more precise as evidenced by F1-score.
• Delivery area indoor scene where our method outperforms the benchmarks that use photometric supervision.

Incomplete mesh extraction evaluation. Below, we follow reviewer’s suggestion and further provide quantitative results on the extracted meshes of ScanNet++ compared to 2DGS, 3DGS, Gaussian Opacity Fields and DN-Splatter. We provide the results for 11 scenes for the methods from the paper and additional results on the one scene for Gaussian Opacity Fields and DN-Splatter due to the limited time and available compute budget. We achieve competitive performance over all baselines, trading off geometric accuracy and rendering quality. Note that DN-Splatter uses sensor depth as input, which drastically improves the quality of mesh, however, at the cost of render quality.

ScanNet++ (11 scenes)

ScanNet++ (0a7cc12c0e)

Q: What RANSAC did the authors use to get the plane parameters? We used a custom pytorch implementation of the RANSAC algorithm as described in classical computer vision educational material.

Bibliography.

• [1] Turkulainen, Matias, et al. "Dn-splatter: Depth and normal priors for gaussian splatting and meshing." 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025.
• [2] Chen, Sili, et al. "Video depth anything: Consistent depth estimation for super-long videos." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

I thank the authors for the new experiments. These are the trends that I have seen as well, with 3DGS often outperforming or being on par with methods designed for improving geometry. I am happy with the results and willing to improve my rating.