RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians

Q1: ... the proposed method does not appear effective relative to the increasing number of Gaussians.

In fact, RetinaGS is equivalent to 3DGS in terms of training and rendering. In Fig. 4, we compare RetinaGS with 3DGS, and as can be seen, for the same number of splats, the performance of RetinaGS always closely matches that of 3DGS. Additionally, increasing the number of splats shows a significant improvement in PSNR for RetinaGS. Especially for the street scene dataset MatrixCity-M, we can observe that increasing the splats by a factor of 10 improved the PSNR from 27.8 to 30.8. (Note that PSNR is a logarithmic metric.)

Q2: It appears that RetinaGS depends on the accuracy of the MVS algorithm.

In fact, our primary purpose of using MVS in this paper is to obtain a sufficient number of Gaussian splats. We could rely solely on the original densification strategy for training, which would also converge to reasonable results. However, the original densification strategy does not allow Gaussian splats to scale to large sizes, an issue we have thoroughly discussed in Section 4.2 of the paper. In other words, we do not rely on the accuracy of MVS, but rather on the point cloud density that MVS provides. This is also evident in Fig. 6, where the black dots represent the results obtained from 3DGS native parameters, and RetinaGS shows only a slight improvement in metrics for the same number of splats. This demonstrates that the improvement from MVS accuracy is limited, and the real advantage of RetinaGS lies in its ability to scale the number of splats.

Q3: I recommend a comparison with Hierarchical3DGS, as both methods present chunk-based Gaussian Splatting approaches.

Thank you for pointing this out. We will include the mentioned works in the Related Work section and provide a detailed discussion. HierarchicalGS introduces the concept of Level of Detail (LOD) to 3DGS, significantly reducing the number of splats involved in rendering for each view and thereby improving rendering speed. There are two key distinctions to highlight here:

1. Focus on scaling directions: HierarchicalGS focuses on “scaling in” by reducing the computational load of a single sub-model. In contrast, RetinaGS focuses on “scaling out” by enabling parallel training of more sub-models, thereby expanding the system's scale. In large-scale models, such as LLMs, scale-in and scale-out strategies are not mutually exclusive. In fact, they should often be employed together to maximize system scalability. We appreciate the reviewer for raising this perspective.
2. Memory and computational costs: In HierarchicalGS, all splats are still maintained as leaf nodes of the hierarchical tree, and additional parent nodes must also be maintained. As a result, GPU memory requirements do not decrease but rather increase, potentially requiring more computational nodes to accommodate the same model scale. This highlights the necessity of introducing a scale-out solution like RetinaGS.

Q4: Missing a comparison of rendering speeds.

In Table 2, we list the total time for rendering and backward passes under the same setup (with the same 3DGS model and the same number of inference passes). Notably, RetinaGS benefits significantly from distributed rendering. For instance, with 5.74 million primitives, 3DGS renders at 80 FPS on one GPU. In comparison, RetinaGS achieves the same rendering results at 80 FPS on a single GPU, 127 FPS with two GPUs, and 177 FPS with four GPUs.

Q5: Fig. 5 is difficult to understand.

Thank you very much for your feedback. We will carefully improve this figure and update it in the revised manuscript.

Q6: Does k represent the k-th index?

In Eq. 3–5, k represents the index of the sub-model or sub-space.

Q7: During rendering, is it necessary to explore all subspaces along the ray path? On average, how many subspaces does each ray hit?

Yes, based on our current implementation. We conducted statistical analysis across different datasets, and the results are presented in Fig. 2 of the main text. For aerial view datasets, rays typically intersect with only one block, while for indoor datasets (e.g., ScanNet++), they intersect with an average of 2.5 blocks.

I appreciate the author's comment. I will keep my score as I still have some concerns. Additional comments are as follows.

Q3: As in the comment of reviewer oADR, Hierarchical3DGS requires consistent performance improvement to show the advantages of RetianGS, as there is a comparison with the real world dataset. I agree that this comparison experiment should be added.

Q4: I understand that the rendering speed increases as the number of GPUs increases, but more comparison with other methodologies is needed. For example, in the case of Hierarchical3DGS, I know that the FPS can go up to 159 depending on the optimization level. I think that one example is not enough for comparisons.

Q1: Limited Scale of Experimented Scenes

MatrixCity-ALL is currently the largest known dataset, containing 140k images, whereas other public datasets typically only have a few thousand images, making the scale difference around 100 times. At present, few works have tackled this dataset directly. Although MatrixCity is synthetic, we still use COLMAP to obtain initial camera poses, which has similar pose estimation errors that might occur in real-world data. Furthermore, collecting large-scale real-world scenes is very costly. We evaluate parallelization strategies for large-scale 3DGS models using the synthetic MatrixCity in this work but also look forward to new contributions of large-scale real-world datasets in the community.

Q2: Lack of Comparison with Other Methods (eg. VastGaussian or Hierarchical 3D Gaussians)

We compare RetinaGS with VastGaussian and HierarchicalGS on the Mega-NeRF rubble. RetinaGS achieves a PSNR of 27.42, outperforming VastGaussian, which reports a PSNR of 26.92, and HierarchicalGS, which reports a PSNR of 24.64.

Q3: Lack of Quantitative Results in Exploration Study

This section discusses three parts in total. The quantitative results for "Initialization and Densification" are presented in Fig. 6, where we compare our strategy with the original 3DGS strategy using both curve and scatter plots. The quantitative metrics and visualization results for "Validity of Distributed Rendering" are shown in Fig. 7. Finally, the quantitative results for "KD-Tree Partition" are provided in Table 2.

Q4: Limited Novelty

We highlight the discussion on novelty and contribution in "Novelty and Value" section of the "Official Comment," which is visible to all reviewers. Please refer to the relevant section for more details.

Q5: Insufficient Scenes in Mega-NeRF Experiments

This is a common practice in the community. CityGaussian also selected these same three scenes in their main paper and provided evaluation results for Mega-NeRF, Switch-NeRF, GP-NeRF, 3DGS, and other methods on these scenes. By using the same metrics, we ensure a fair comparison with other approaches.

Q1: Missing related work（e.g. Nerf-XL）

Thank you for your suggestion. We will include the mentioned works in the Related Work section and discuss them accordingly. Nerf-XL primarily addresses distributed training for NeRF and adopts a partial rendering strategy. However, it is important to emphasize that applying partial rendering in 3DGS is non-trivial and significantly more complex than in NeRF-based methods.

• First, the rendering primitive in 3DGS, splats, has volume, while the primitive in NeRF, sampling points, has no volume. Splats at the boundaries of sub-models pose challenges for sorting in alpha blending and for the partitioning of model parameters for parallelism. A detailed discussion of this is included in the "Novelty and Value" section of the "Official Comment" visible to all reviewers.
• Second, in NeRF-based methods, a uniform subdivision of the scene ensures load balancing. In 3DGS, however, splat distributions are often uneven, necessitating more sophisticated strategies to balance the computational workload among sub-models. We employ a KD-tree for this purpose and have demonstrated its effectiveness in our experiments.

Q2: Practical usage of KD-tree

In fact, we partition the KD-tree based on the number of splats, which ensures that the number of splats in each block is approximately equal. As a result, it can adapt to situations where the point distribution is uneven. Table 2 demonstrates that this strategy performs much better than a uniform partitioning approach (based on spatial volume).

Q3: MVS points for initialization

Thanks for raising this point. In the Mip-NeRF360 Garden dataset, if the original densification strategy in 3DGS is used, the model still converges with no significant impact on PSNR (27.33 vs. 27.41). However, the original densification strategy in our experiments can not achieve the desired large number of primitives, a limitation we have thoroughly discussed in Section 4.2 of the paper. In other words, the precision provided by MVS is not strictly necessary; a faster but less precise MVS solution (e.g., DUSt3R) could be used to provide a dense initialization, and then the adaptive capabilities of 3DGS could drive convergence. In the MatrixCity-Aerial dataset, we randomly selected 10% of the views for MVS initialization, keeping all other conditions the same. The model converged normally, and the PSNR showed only a small decrease compared to using MVS initialization with all views (27.70 vs. 27.65).

Q4: This work is developed upon 3DGS, can it be compatible with derivatives of 3DGS, e.g., 2DGS or ScaffoldGS?

Yes. Thanks for this great suggestion. The directives of 3DGS, e.g. 2DGS and ScaffoldGS, use volumetric primitives and alpha blending for rendering The proposed strategy can apply to these types of models and enable distributed training of them.

Q1: Lacks discussion of more existing distributed methods…

Thank you for pointing this out. We will include the mentioned works in the Related Work section and provide a detailed discussion. Specifically, you highlighted the connections between our work and DOGS and City-on-Web. Below, we discuss these works and their differences from ours:

• City-on-Web primarily addresses distributed training for NeRF and adopts a partial rendering strategy. However, it is important to emphasize that applying partial rendering in 3DGS is non-trivial and significantly more complex than in NeRF-based methods. First, the rendering primitive in 3DGS, splats, has volume, while the primitive in NeRF, sampling points, has no volume. Splats at the boundaries of sub-models pose challenges for sorting in alpha blending and for the partitioning of model parameters for parallelism. A detailed discussion of this is included in the "Novelty and Value" section of the "Official Comment" visible to all reviewers. Second, NeRF-based methods use uniform scene subdivision for load balancing, but 3DGS's uneven splat distribution requires more advanced strategies. We address this with a KD-tree, proving its effectiveness in our experiments.

• DOGS and VastGaussian/CityGaussian use a similar strategy of dividing training data and splats into predefined blocks, independently training 3DGS sub-models for each block. While this differs from our method (see Fig. 2), it approximates native 3DGS and may cause boundary artifacts. DOGS mitigates this with overlapping regions and boundary synchronization but assumes each ray/view belongs to one sub-model—a valid approximation for bird's-eye datasets. For general datasets, where rays intersect multiple sub-models, significant non-adjacent information must be cached, increasing parameter demands and reducing scalability as DOGS' model parallelism devolves into data parallelism.

Q2: The proposed method requires a gradient computation step and a gradient synchronization step...

The gradients that need to be computed can be divided into loss gradients and sub-model gradients. As shown in Equation (5), the central manager only computes and synchronizes the loss gradients with respect to the partial color and opacity, which a 2D map with the same resolution as the image can represent. This means that the required communication bandwidth and computation are negligible. According to our statistics, the time overhead for this step accounts for less than 1% of the total time. Additionally, a small portion of the splats is shared between workers in our implementation, requiring gradient synchronization concerning these primitives. This synchronization takes less than 5% of the total step time in our experiments. So, overall, we believe the overhead introduced by distributed training is minimal in our method.

Q3: Running MVS on large-scale datasets is time-consuming…

We want to emphasize that the primary purpose of using MVS in this paper is to obtain a sufficient number of Gaussian splats. The original densification strategy in 3DGS can be also used when the model still converges with no significant impact on PSNR. However, the original densification strategy in our experiments can not achieve the desired large number of primitives, a limitation that we have thoroughly discussed in Section 4.2 of the paper. A densification strategy that enables GS points to scale to large sizes is another important topic, but beyond the scope of this paper.

Q4: … has only marginal improvement compared to CityGaussian ...

We highlight the discussion in "Improvement is not Marginal" section of the "Official Comment," which is visible to all reviewers. Please refer to the relevant section for more details.

Q5: ...the author should provide their results with sparse/dense point clouds.

As shown in Fig. 4, when the number of splats is the same, the PSNR of 3DGS and RetinaGS, which only differ in initialization and densification, are almost identical, indicating that RetinaGS does not rely on MVS's potential higher initialization accuracy to achieve the same accuracy as 3DGS. We use MVS solely to obtain a sufficiently dense set of splats, as the native densification strategy cannot easily increase the number of GS points as we desire.

Q6: I wonder would RetinaGS would be slower/faster than CityGaussian/VastGaussian/DOGS?

Our approach is faster. As shown in the discussion above, communication overhead accounts for only about 5% of the total step time. The true bottleneck in distributed 3DGS training lies in the load balancing strategy. Other methods overlook load balancing, resulting in uneven splat distribution and significant computational disparities across sub-models, which cause idle waiting times. Table 2 demonstrates that with our KD-Tree load balancing strategy, we reduced training time from 32.4s to 18.9s when using 4-node parallelism, achieving a speedup of 71.4%.

Thanks the authors for their rebuttal. While most of my concerns are resolved, I prefer to keep my score currently and will consider to raise my score with further discussion.

As the authors claimed they use MVS point clouds for initialization to achieve a desired number of Gaussian primitives on the MatrixCity street view scenes, I just realized that most of the experiments of Hierarchical-GS (Kerbl, el,al, SIGRRAPH 2024) are done on street view scenes. Moreover, Hierarchical-GS also adopts a tree structure to enable the distributed chunk training of 3DGS, which is similar to the KD-tree proposed in this work. Therefore, I would like the authors:

1. to further highlight the similarities and differences of the method with Hierarchical-GS.
2. compare with Hierarchical-GS on the street view scenes used by Hierarchical-GS under the same initialization settings (I believe Hierarchical-GS uses sparse points from COLMAP for initialization on the street view scenes).

I would raise my score if the proposed method can achieve comparable results to Hierarchical-GS.