AtlasGS: Atlanta-world Guided Surface Reconstruction with Implicit Structured Gaussians

We thank Reviewer p8vj for the valuable review comments. We are encouraged that you found our work to have higher reconstruction quality. And we have to apologize for the typos in our paper, and we will fix them in the final version. We address your concerns in detail below.

---

The limited scope of the proposed model.

W1: The prime concern of the work is the limited scope of the proposed model. As stated in the paper, since this method is based on the Atlanta world model assumption, it cannot be applied to more complex scenes with complicated objects. When experimenting with a dataset where the Atlanta assumption is valid, it is trivial to observe that the proposed method outperforms existing methods. However, the scenarios to which this assumption is applied are quite limited, and the performance of AtlasGS may be inferior to existing models in real-world scenarios. The author should demonstrate that the model can achieve comparable performance to more general scenes with complex objects, thereby validating the contribution of the work.

Our method specifically targets the challenging problem of surface reconstruction in indoor and urban scenes, environments prevalent in daily life and often poorly addressed by existing literature. General-purpose methods, such as 2D Gaussian Splatting (2DGS), frequently falter in these settings due to the common presence of low-texture regions. The integration of an Atlanta-world model provides a crucial global geometric constraint, ensuring consistent surface reconstruction in these difficult scenarios.

However, it's important to clarify that this focus does not restrict our method's applicability. To demonstrate its performance in general scenes, we conducted a comparative analysis on the Tanks and Temples dataset, a standard benchmark for general-purpose reconstruction. Our reported F-score results indicate that our method achieves comparable performance to established techniques, thereby addressing concerns regarding its generalization capabilities.

Memory analysis.

W2: In implicit structured GS, is a Gaussian predicted for every voxel in a predefined voxel grid? If so, what is the total number of voxels and the number of GSs estimated for each voxel? If a fixed number of GSs is created for every voxel instead of estimating valid anchors (Scaffold-GS), it will generate many redundant Gaussians and become memory-inefficient. This can be seen by comparing the memory cost difference between AtlasGS and other models.

In implicit structured Gaussian, a fixed number $K$ of Gaussians is predicted for each voxel in the voxel grid; however, the voxel grid is not predefined. We initialize the voxel grid with a sparse point cloud, then we generate new voxels with gradient-based densification following Scaffold-GS, and prune the voxel if the opacities of all the Gaussians are lower than a threshold. In our experiments, we set $K$ to 10.

When compared to Scaffold-GS, our method consumes slightly more memory during training, which is a deliberate trade-off. Scaffold-GS reduces memory by estimating only a subset of valid anchors, but this strategy can inadvertently mask visible Gaussians, leading to depth inconsistencies across views and a slight degradation in performance. But compared with 2DGS, our method is relatively memory-efficient, because 2DGS grows redundant Gaussians in a single voxel, while ours only allows limited Gaussians.

Biased depth caused by alpha blending.

W3: How are the values of depth and normal calculated?

In our framework, we use mean depth and mean normal, which are obtained from alpha blending as mentioned in 2DGS. A well-known artifact in volume rendering, used by both NeRF and Gaussian Splatting, is biased depth. Because the rendered depth for a pixel is a weighted average of all samples along a ray, it can be influenced by points in front of or behind the true surface. To alleviate biased depth, our framework combines two techniques. First, we apply the distortion loss from 2DGS to promote a compact opacity distribution. Second, our 3D Global Planar Regularization aligns floor and ceiling Gaussians with a single surface by explicitly regularizing their positions.

Hyperparameter analysis.

Q1: How the performance changes depending on the value of each hyperparameter?

To analyze the impact of key hyperparameters, we conducted a series of ablation studies on the ScanNet (scene0050) dataset. We focused on the weights for the depth prior $\lambda_{depth}$，normal prior $\lambda_{normal}$, distortion loss $\lambda_{dist}$, and our regularization loss $\lambda_{reg}$. When changing one value, the other one remains the same. The results, presented in the tables below, demonstrate that our chosen hyperparameters are well-justified and that each component contributes positively to the final reconstruction quality.

The distortion loss encourages compact ray sampling along the viewing direction. Increasing its weight from 1.0 to 10.0 results in a noticeable improvement in all metrics. This suggests that a stronger distortion penalty leads to a more coherent and less noisy surface representation. The performance saturates when increasing the weight further to 100.

Our proposed regularization term also plays a vital role. With a minimal weight of 0.001, the performance is suboptimal. As we increase the weight, performance improves, peaking at $\lambda_{reg}$=0.1 with the best F-score of 81.02. However, if the weight becomes too large, the performance begins to decline. This is because an excessively large loss of weight can pull the structural Gaussians too strongly towards the initial planes, potentially causing the entire structure to converge to an incorrect plane.

As the following tables show, incorporating and appropriately weighting geometric priors is crucial for high-quality surface reconstruction. These priors improve the results by directly affecting the Gaussian positions and by helping to train the plane indicators, which provide better structural regularization for the scene.

We thank Reviewer QZGu for the valuable review comments. We are encouraged that you found our work both simple and effective, good in both qualitative and quantitative results. We address your concerns in detail below.

---

Geometry priors are not the cause of the problem presented by Figure 1.

W1: So Did the problem present by Figure 1 stem at these prior? Actually, I only realize the discontinue issue in ScanNet part in Figure 1.

The broken and distorted surface of 2DGS in Figure 1 doesn't result from priors but from the discrete nature of Gaussian Splatting in low-texture regions. To validate this, we perform an ablation study by removing geometric priors from the 2DGS framework. The results are shown below.

As shown in the table above, Acc and Comp are lower compared to the case without geometry priors.

Rendering efficiency and online decoding

Q4: In Line 253, you mentioned that the scheme need to decode gaussians online? Why need online operation? Isn't gaussians fixed after it has been trained? Further, how about the total GPU memory usage during inference in indoor/urban scenes and how about the additional GPU memory increased by the online decoding operation?

W3: The rendering efficiency of the scheme seems affected a lot by the newly proposed component.

Decoding online means that at each training step, we need to decode all Gaussians from the voxel grid with an MLP. After training, the Gaussians are fixed, and we can preextract the view-independent attributes, which will reduce the memory footprint and improve the rendering performance.

The total memory required for inferencing is approximately 1.06 GB for scene0050_00 and 2.24 GB for the matrixcity block6 with the online decoding operation. By pre-extracting the Gaussians, memory usage is reduced by approximately 300 MiB for the indoor scene and 700 MiB for the urban scene.

While preparing this rebuttal, we re-validated our experiments and discovered a CUDA-synchronization error with considering I/O in our original rendering speed evaluation. We sincerely apologize for this oversight and provide the corrected results below.

The following table compares the rendering performance (in FPS) of our method against other approaches on both indoor and urban scenes. We report speeds using pre-extracted features and speeds in brackets that include online feature decoding.

On both indoor and urban scenes, the online decoding operation leads to a significant performance drop; however, with pre-extraction, the rendering performance is comparable to our methods.

Rendering, semantic segmentation quality, and training time, rendering speed for urban scenes.

W2: In the experiment, this paper only present the geometry metrics, but how about the semantics metrics after reconstruction like PSNR/SSIM/LPIPS and how about the qualitative results of the semantics reconstruction? Further, can you provide the training time and FPS in Table 3 in urban scenes like Table 2?

We report the rendering quality and semantic reconstruction quality in Section A.5 of the supplementary material. And here're the results excerpted from the supplementary material.

The following table shows the rendering quality and semantic quality, respectively. And IoUw, IoUf, IoUc represent the intersection of the union of the wall, floor, and ceiling, respectively.

We use block6 in MatrixCity to test the performance and training time. The training time is 37 minutes, and the rendering speed could reach 191 FPS with per-extraction.

Differences or strengths between our work and [2].

Q1: What isthe difference or strength between your work and [2]?

Sorry, there's no reference about [2]. We suppose that you may refer to CityGaussian, which is an urban scene reconstruction method. These urban scene reconstruction methods mainly focus on the problem that is how to reduce memory footprint and maintain the rendering efficiency, and some partitioning or compression methods have been developed. And in our method, we mainly focus on the surface reconstruction and don't take these problems into consideration.

The reconstruction quality of VGGT.

Q2: Can the latest work VGGT[3] handle the problem of “Traditional multi-view stereo methods struggle on these textureless surfaces due to the lack of distinctive” in Line 22?

While VGGT shows powerful reconstruction ability and short inference time, it can provide the desired reconstruction quality on these textureless regions, in which the geometry is simple and easy to estimate. We perform the experiments on ScanNet and Replica with 50 images at a resolution of 518 due to limited GPU memory. Before we evaluate the geometry reconstruction quality, we first use the input pose and ground truth pose to align the point cloud from VGGT, and then use ICP to refine the transformation matrix. The reconstruction quality is shown as follows. As a result, VGGT shows comparable results and higher speed with feed feed-forward model.

The “plane indicators” mentioned many times are not the same as “plane normal”.

Q3: Was the “plane indicators” mentioned many times the same as “plane normal”?

Plane indicators are not plane normal but the plane equation. As mentioned in L165, we define explicit plane indicators with a gravity direction and two distance offsets to represent the floor and ceiling.

More results on the autonomous driving scenario.

Q5: Can the proposed scheme used to reconstruct the static scenes in autonomous driving scenario like nuscenes and waymo?

We perform our experiments on 3 static sequences in the Waymo Open Dataset and compare our method with 2DGS, and evaluate the reconstruction results with the GT lidars. We report the Chamfer Distance in meters and F-score with a 0.5m threshold. As shown in the table below, our method shows better accuracy of surface reconstruction than 2DGS.

The proposed scheme will fail when the segmentation model predicts the wrong label.

Q6: In which case the proposed scheme will fail?

Our method uses a segmentation model to distinguish structural elements, such as walls, ceilings. If the segmentation model predicts the wrong label, such as walls being recognized as ceilings, the plane indicators will be biased by wrong-labeled structures, and the surface reconstruction quality of these structural elements may be degraded.

I am truly sorry for our oversight. Due to a mistake on our part, you were not included in the "Readers" list, but we selected "Reviewers Submitted" into the "Readers" list, and as a result, our previous response may not be visible to you.

Thanks for your reply, and we apologize for not explaining the issue clearly.

I am still confused about the issue of “global inconsistency for monocular geometric priors”. Since Figure 1 does not clarify this issue, it seems that the paper does not formally address it either, neither in a visual nor a mathematical way.

Our geometry priors are obtained from monocular estimation models, not from multi-view stereo (MVS). Since each prior is predicted from a single image, the normal vectors for a given surface are frequently inconsistent when transformed into the global coordinate system. This lack of global consistency is a common challenge for methods that rely on monocular geometry.

Figure 1 also demonstrates this issue of global inconsistency with examples from the ScanNet dataset, such as the noisy surfaces on the wall above the sofa (below the yellow box) and the floor beneath the chair. To address this, we adopt an Atlanta world assumption, which allows us to optimize these textureless areas using the Atlanta guided planar regularization, both in 3D and 2D. This technique results in noticeably smoother and flatter surfaces compared to those produced by 2DGS.

In the rebuttal, you mentioned that the problem in Figure 1 “doesn't result from priors but from the discrete nature of Gaussian splatting in low texture regions.” To validate this, you conducted an experiment removing priors from 2DGS, but this experiment does not seem to be related to the issue you are trying to validate. It seems that what you are validating is just whether the performance of 2DGS without priors worsens.

Our experiments show that regardless of whether geometry priors are used, the surfaces reconstructed by 2DGS visually exhibit similar issues, namely fragmented and discrete surfaces. Since we cannot include figures in the rebuttal, we have only provided quantitative results here. The corresponding visual comparisons will be added in the final version of the paper.

And previous methods, such as MonoSDF, have demonstrated that monocular geometry priors can mitigate the issues associated with textureless regions, despite existing global inconsistencies. However, when it comes to 2DGS with geometry priors, the improvements observed are limited compared to those without geometry priors, which is mainly due to the discrete nature of the approach.

Base on the second point, in the rebuttal experiment, you removed priors from 2DGS. Which priors were removed? I also asked in my weakness whether 2DGS uses priors, but it seems that this was not clarified in the rebuttal.

Sorry for not explaining in more detail. The prior used in 2DGS is the same as ours, as stated in the Eq. (11) and Eq. (12) in the main paper. We use monocular depth from DepthAnythingv2[1] and monocular normal from StableNormal[2], both for our method and 2DGS.

[1] Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., & Zhao, H. (2024). Depth anything v2. Advances in Neural Information Processing Systems, 37, 21875-21911.

[2] Ye, C., Qiu, L., Gu, X., Zuo, Q., Wu, Y., Dong, Z., ... & Han, X. (2024). Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43(6), 1-18.

We thank Reviewer pVvf for the encouraging feedback and insightful review comments. We are glad that you found our work to be simple, straightforward, and of high surface quality. We address your concerns in detail below.

---

Scenes without Atlanta assumption.

W1: Atlanta World assumption is, from my point of view, would break in many cases, e.g., indoor space at the roof floor where the ceiling is not parallel with the ground, a cinema or theater where the ground steps down, a modern design art museum with complex non-plane structured ceiling and even walls which is not plane at all. In that case, the 3D Global Planar Regularization may not stand. A mechanism for dealing with such a situation is not explored in this paper.

We agree that the Atlanta-world assumption does not hold in all scenarios, and we acknowledge this as a limitation of our current work. However, our framework includes a mechanism that provides a degree of robustness for scenes that only partially adhere to this assumption.

Our semantic lifting process explicitly includes an "others" category alongside "wall", "floor", and "ceiling". Any surface that is not confidently identified as one of the primary structural elements falls into this "others" category. These Gaussians are subsequently excluded from the planar regularization constraints. This effectively acts as a gating mechanism, ensuring that the strong geometric prior is primarily applied where strong semantic evidence exists.

As for more general scenes without such an assumption, our method still shows comparable results. The following table shows the reconstruction quality on the Tanks and Temples compared with 2DGS.

Surface extraction methods.

Q1: I failed to find details about how the predicted 3D mesh is generated from a trained 3D Gaussian Field, whether all Gaussian Splatting-based methods mentioned in the experiments are processed with the same meshing method, or not?

All GS-based methods, except for GSRec, use the same meshing method. For GSRec, we follow the meshing method from its original paper. Specifically, GSRec performs Poisson surface reconstruction, and others (including our method) do TSDF-fusion on rendered depth maps using the Open3D package.

Geometric accuracy of non-planar surfaces.

Q2: The experiments are conducted by evaluating the global geometric reconstruction accuracy. I would like to see the geometric accuracy of non-planar surfaces using AtlantaGS, like running the same evaluation but removing walls, ceiling, and ground floor.

We evaluate the accuracy of non-planar surfaces on the ScanNet dataset and use ground-truth semantic masks to exclude planar elements like floors and walls.

The table below provides a quantitative comparison against 2DGS and MonoSDF on ScanNet. Each cell shows the mean/median performance, with arrows indicating the desired direction for each metric ($\downarrow$ lower is better, $\uparrow$ higher is better).

As shown above, our method consistently and significantly outperforms both baselines across all five metrics, for both mean and median scores. This underscores our method not only benefits the planar surface in the Atlanta world assumption, but also the non-planar surface.

Dear Reviewer pVvf,

We hope this message finds you well. As the discussion period is drawing to a close, we would like to kindly check whether there are any remaining questions or suggestions we could further clarify. We're happy to engage further and provide any additional details if needed. Thank you again for taking the time to review our submission and for your thoughtful comments.

Best regards, authors

We thank Reviewer 3sjd for the encouraging feedback and insightful review comments. We are glad that you found our work to be promising and that it has high-quality reconstruction results. We address your concerns in detail below.

---

Reconstruction quality without such Atlanta assumption.

W1: I am actually pretty curious regarding whether the proposed model could maintain geometric accuracy over scenes without such assumptions, and which of the new components shown in this paper could correspond to the potential improvement on non-Atlanta scenes? (the volume grid?)

For scenes without such an assumption, monocular geometry priors cannot provide globally consistent supervision on textureless regions, which may lead to performance degradation as shown in the following table. However, our implicit structured Gaussian will provide local coherent geometry within a voxel and provide better geometry reconstruction quality, which potentially improves the surface reconstruction on non-Atlanta scenes.

For more general scenes without such assumption, we perform the experiments on Tanks and Temples, and Waymo, and compare our implicit structured Gaussian with 2DGS. Our method is still comparable with the previous method as following table. We report the F-score on the Tanks and Temple datasets and the Chamfer Distance on Waymo.

The relations between what the model has improved and what is proposed.

W2: Are those branches of works can also cope with surface continuity over textureless areas or providing rich geometric textures? Is the volume grid presented in the model that contributes most to the geometric details while the MLP contributed to the continuity (plus semantic guidance)? The relations between what the model has improved and what is proposed would be better elaborated.

GSRec and GSDF, two related methods, suffer from issues in textureless regions. GSRec's expensive sampling process, based on sparse Gaussian distributions in these areas, produces noisy surfaces and increases training time and memory. GSDF also has high computational demands, taking about 2 hours and 40 GiB of GPU memory to train its dual Gaussian and NeRF models. Furthermore, its reliance on monocular geometry priors for regularization in textureless regions can lead to globally inconsistent and error-prone results.

In contrast, our implicit-structured Gaussian framework tackles these challenges at the representation level. By using a sparse feature grid and implicit functions to organize Gaussian primitives, we avoid the need for costly external regularizers like SDFs. Our approach is further enhanced by Atlanta world assumption guided planar regularization, ensuring geometric consistency across large, textureless surfaces without the high computational overhead and quality trade-offs of previous methods.

Geometry details and continuity.

W2: Is the volume grid presented in the model that contributes most to the geometric details while the MLP contributed to the continuity (plus semantic guidance)?

Our framework utilizes a hybrid approach to accurately model scene geometry. A sparse feature grid captures the coarse geometric structure, while explicit Gaussians model the fine details. During training, gradients guide the densification and movement of these Gaussians, allowing them to accurately represent the scene's geometry. The MLP decoder provides local continuity. It acts as a continuous function that predicts the attributes of all local Gaussians from the features stored in the grid. Because a single, shared MLP generates these attributes for a neighborhood, optimizing one Gaussian inherently influences its neighbors through the shared features and the MLP itself. This ensures the resulting Gaussians are locally coherent and smooth, overcoming the discrete nature of Gaussian splatting.