DiffGS: Functional Gaussian Splatting Diffusion

It is with great appreciation that we acknowledge the insightful feedback from Reviewer TmLe. We have addressed the questions and comments with careful consideration, and we encourage continued conversation to ensure the robustness of our findings.

Q1: Motivation, Literature, and Experimental Evaluation

Variational Autoencoders (VAEs) are well-suited for capturing complex distributions in a lower-dimensional latent space, which encodes diverse data into a compact latent representation, allowing for more efficient and structured generation of 3D models. In comparison to Neural Radiance Fields (NeRF), 3D Gaussian Splatting presents multiple benefits, especially regarding rendering efficiency and superior visual fidelity. Addressing the challenge of 3DGS generation is both timely and essential for the progression of the 3D modeling domain. The revised abstract will explicitly state the novelty and motivation behind using Gaussian splatting for novel 3D asset generation, emphasizing its significance in overcoming the limitations of discreteness and structural complexity.

We carried out further experiments to compare our approach with the state-of-the-art baseline methods, such as EG3D (CVPR 2022), SSDNeRF (ICCV 2023), Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM (ECCV 2024), and DreamGaussian (ICLR 2024). As shown in Figure A, Figure B and Figure D of the rebuttal PDF, our method shows more refined geometry and detailed textures. Besides, we compare text consistencies with the other SOTA methods in Table A of the rebuttal PDF. Our method demonstrates a competitive alignment between the text descriptions and the resulting 3D models. Besides, we provide a comparison of model parameter counts and generation times in Table B of the rebuttal PDF. DiffGS has significantly fewer parameters than LGM, resulting in reduced memory usage and computational overhead. Moreover, our approach achieves faster generation times than other state-of-the-art 3D generative models, demonstrating its efficiency in creating high-quality 3D content.

We extend our sincere thanks to Reviewer cAhv for their comprehensive review and valuable suggestions. In response to the concerns raised, we have offered detailed explanations in the subsequent section, and we are eager to engage in further dialogue.

Q1: Incomplete Shapes in 3DGS

It is indeed possible that some models exhibit holes and incomplete shapes, especially in regions with high-frequency details. This issue arises from challenges in capturing fine details during the Gaussian extraction process. One straightforward approach to improve this is to increase the number of sampled Gaussians. By doing so, we can achieve a higher resolution representation, capturing more details in the high-frequency regions. Similarly, increasing the depth of the octree can enhance the model's ability to resolve fine details. We aim to balance quality and computational efficiency. While increasing the number of Gaussians and octree depth improves detail capture, it also escalates computational costs. We focus on a balance between satisfactory results and inference efficiency,, and find it acceptable to occasionally generate shapes with minor artifacts.

Q2: Inference Efficiency and Optimization Impact

During Gaussian extraction, DiffGS performs optimization over the Gaussian positions to enhance the quality and accuracy of the generated 3D Gaussian Splatting geometries. This optimization step, while essential for ensuring high-quality outputs, does introduce additional computational overhead. We conducted supplementary experiments to analyze the impact of the number of Gaussian points on optimization time. The results of this ablation study are presented in Table C of the rebuttal PDF. In our experiments, we selected a configuration of 350K Gaussian points. This choice balances quality and computational efficiency, providing a robust representation of the model without excessively increasing processing time. The optimization time for this configuration is approximately 2.5 seconds. We also compared our model's generation time with other state-of-the-art 3D generative methods. The results are shown in Table B of the rebuttal PDF, where we highlight that our method demonstrates significant efficiency compared to the SOTA baselines DiffTF, ShapE, SSDNeRF and DreamGaussian, despite the added optimization step. DiffGS also offers competitive generation time with GET3D and LGM.

Q3: Comparison with Splatter Image Method

For a comprehensive comparison with Splatter Image, we evaluated both methods under the single-view generation task. We conducted supplementary experiments to compare the visual quality of reconstructions generated by DiffGS and Splatter Image. The results are presented in Figure A of the rebuttal PDF. The visual comparison indicates that DiffGS produces higher fidelity reconstructions with better handling of fine details and fewer artifacts compared to Splatter Image.

Q4: Number of Parameters and Baseline Comparisons

We have included the number of parameters for DiffGS and other baseline methods in Table B of the rebuttal PDF. As shown in the table, DiffGS has a significantly smaller number of parameters compared to most baseline methods. The smaller parameter count of DiffGS translates into reduced memory usage and potentially faster inference times, which can be advantageous in environments where computational resources are limited. Despite having fewer parameters, DiffGS demonstrate superior performance, demonstrating that its architecture efficiently captures essential features for high-quality 3D reconstruction.

We are truly grateful for the insightful review provided by Reviewer 6WmW. Your thorough analysis and constructive feedback have significantly enhanced the quality of our work.

Q1: Visual Quality and Evaluation Protocol

We justify that DiffGS achieves SOTA performance in terms of both numerical and visual qualities. In response to the reviewer's comments, we conducted additional experiments comparing our method to the SOTA baseline methods including EG3D(CVPR 2022), SSDNeRF(ICCV 2023), Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM(ECCV 2024) and DreamGaussian(ICLR 2024). We are unable to compare our method with DiffRF since it is not open-sourced. However, we have compared our method against other stronger baselines such as DiffTF and SSDNeRF, which have been shown to outperform DiffRF in various benchmarks. The visual comparison results are presented in Figure A, Figure B and Figure D of the rebuttal PDF. The results show that our method outperforms the SOTA baseline in capturing complex details while preserving geometric consistency. Additionally, we evaluate text consistencies against other state-of-the-art methods in Table A of the rebuttal PDF. Our approach shows strong alignment between textual descriptions and the generated 3D models. Furthermore, Table B highlights a comparison of model parameter counts and generation times. DiffGS employs far fewer parameters than LGM, leading to lower memory consumption and reduced computational demands. Additionally, our method achieves quicker generation times than other top 3D generative models, underscoring its efficiency in producing high-quality 3D content.

For our evaluation, we selected a diverse set of 50K images as the reference set. This selection was designed to reflect a wide range of scenarios within the dataset. We used the testing set as the reference set for rendering. This approach is aligned with practices like those used in SSDNeRF, ensuring that our evaluations are based on unseen data and provide a realistic assessment of model performance. By choosing the testing set as our reference, we prioritize demonstrating the model's ability to generalize and generate high-fidelity outputs in unseen scenarios. This decision aligns with the goals of ensuring robust evaluation and fair comparison across models.

Q2: Optimization of Loss in Equation 6

We compute the loss directly against real Gaussians, which serves as the ground truth for optimization. This approach ensures that the model aligns with actual data distributions effectively.

Each property of the Gaussians, including position, color, and shape, is represented as a continuous field in the spatial domain. This allows for precise querying and supervision with the real Gaussians.

We are truly grateful for the comprehensive feedback and time that reviewer tnRu dedicated to evaluating our work. Below, we provide responses to each of your questions. We look forward to your further comments on our responses.

Q1: Benefits of Structural Triplanes in 3D Gaussian Splatting

The triplane structure utilizes 2D planes, which allows for the integration of standard 2D neural networks. This integration provides several benefits, including access to well-established 2D processing techniques and efficient computational frameworks. Triplanes serve as a bridge to utilize the generalization capabilities of diffusion models in 3D generation. By predicting features in triplanes first, we can more effectively capture and model Gaussian distributions. Directly modeling 3D with triplanes may not leverage the full potential of these distributions, especially when complex geometries are involved. Also, triplanes allow for better handling of spatial information, facilitating the reconstruction of detailed and accurate 3D representations. We conducted an ablation study where the triplane structure was replaced with a Multi-Layer Perceptron (MLP) to evaluate its effectiveness in modeling Gaussians. The results, visualized in Figure C of the rebuttal PDF, indicate that using an MLP fails to capture the intricacies of Gaussian modeling adequately. The triplane approach demonstrates superior performance in terms of detail accuracy and geometric fidelity.

We further provide the efficiency comparison with other SOTA baselines in 3D generation, as shown in Table B of the rebuttal PDF. DiffGS signicantly outperforms other baselines in terms of generation efficiency. Although generation speed is not our sole advantage, our method effectively balances speed and quality, underscoring the strengths of the triplane approach in delivering high-fidelity results efficiently.

Q2: Advantage of Gaussian Probabilities Over Direct Point Cloud Prediction

Predicting Gaussian probabilities allows us to extract Gaussians at arbitrary numbers. This flexibility is a significant advantage over directly predicting point clouds, which are typically fixed in number and less adaptable to varying levels of detail. Moreover, direct generation of unstructured Gaussians poses challenges due to their non-structured nature. By disentangling GauPF, GauCF, and GauTF, we create a framework that systematically handles Gaussian attributes, allowing for a structured representation that addresses these challenges.

During rebuttal, we directly compare our method with "Triplane Meets Gaussian Splatting" (CVPR 2024). The visual comparison results are presented in Figure A of the rebuttal PDF, illustrating the superior visual quality achieved by our approach. Our method consistently produced more accurate and detailed reconstructions compared to the baseline. By disentangling the Gaussian attributes, our approach allows for more efficient modeling of 3D structures, ensuring that both geometric and color details are preserved with high accuracy.

Q3: Comparison with Other Methods

We conducted comparative experiments focusing on two critical tasks: text-to-3D generation and single-view 3D generation. These tasks are central to demonstrating the flexibility and robustness of our approach in different application scenarios. The visual results of our comparisons are presented in Figure A and Figure B of the rebuttal PDF. These figures illustrate the superior visual quality and fidelity achieved by our method compared to the SOTA baseline methods including Shap·E, SplatterImage (CVPR 2024), TriplaneGaussian (CVPR 2024), LGM(ECCV 2024) and DreamGaussian(ICLR 2024). Our approach consistently produced more detailed and accurate generations, effectively capturing intricate textures and geometries that are often challenging for the compared optimization-based and multi-view based methods. We also evaluated the models using the CLIP score, a metric that measures the semantic alignment between the generated 3D models and the input conditions. The results are presented in Table A of the rebuttal PDF. According to Table A, our method achieved higher CLIP scores than both DreamGaussian and LGM. This indicates that our approach more faithfully adheres to the condition inputs.

Q4: Mathematical Notation and Clarifications

We appreciate the reviewer's careful examination on the mathematics.

The use of $j$ was indeed incorrect and should be replaced with $i$ to maintain consistency in indexing. Each Gaussian $g_i$ should be consistently indexed with $i$, not $j$.

We will correct any typographical errors in Equation (3), ensuring that it accurately conveys the intended mathematical relationship.

$\psi_{pf}$: This function represents a probability-related component within our model, implying that larger values of $\psi_{pf}$ correspond to higher probabilities. And We will revise Equation (9) to correctly reflect the maximization intention.

Dear reviewer tnRu,

Thank you for your response and the helpful comments. Following your suggestions, we conduct an additional ablation study under the same experimental settings outlined in Figure C of the rebuttal PDF. Specifically, we directly model the Gaussian attributes (e.g. color, density) using triplane and evaluate its performance in terms of PSNR, SSIM, LPIPS metrics, and inference time. The results are presented in Table E below. As shown, while the approach that directly utilizes triplane for Gaussian modeling without feature predicting and decoding slightly improves the inference speed, it leads to significant degradation on the quality of 3D Gaussians. In contrast, our proposed method demonstrates superior performance in terms of PSNR, SSIM, and LPIPS, compared to this alternative design. The results demonstrate the effectiveness of our triplane framework designs and the key insight to disentangle Gaussian splatting through three novel functions.

Table E: Ablation studies on the framework designs of the triplane.

Best regards,

Authors

Dear Reviewer tnRu,

Thanks for your response and the positive assessment on our rebuttal.

We appriciate your suggestions on the potential comparison with the use of a NeRF decoder (e.g. EG3D, SSDNeRF). In response, we conduct additional experiments where we replace our Gaussian functions with the SSDNeRF decoder to predict attributes of colors and density, maintaining all other settings constant. The results of this experiment are represented in Table F below. As shown, the SSDNeRF decoder results in a noticeable decline in PSNR, SSIM, and LPIPS metrics. The results demonstrate that our proposed Gaussian functions contribute significantly to the 3D rendering quality, showcasing the effectiveness of our functional decomposition of Gaussian representation and designs on the triplane framework.

Table F: Ablation studies on the design of the decoder.

We further demonstrate that Gaussian splatting provides a more efficient solution for high-fidelity 3D rendering compared to volume-based methods like NeRF. By adopting a point-based representation for optimization and rendering, it reduces computational demands and accelerates the convergence of the training process. Most importantly, 3D Gaussian splatting significantly enhances rendering speed, making it particularly well-suited for real-time applications such as virtual reality and interactive 3D simulations. In practice, the rendering time for each frame using 3D Gaussian splatting can be less than 0.01 seconds, compared to several seconds for NeRF.

We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further.

Best regards,

Authors

We are very grateful to the reviewer 3Kid for the thoughtful feedback and time invested in evaluating our work. We address each question below.

Q1: Dataset Size and Scalability

We justify that data fitting is a common step for most generative models. For example, DiffRF requires voxelized radiance field fitting during data processing, which may take hours for each shape. NDF [1] and DiffTF require SDF-based and NeRF-based triplane fitting for data preparation. In comparison, our method only takes about 3 minutes to fit the Gaussian representation of an object using a single NVIDIA RTX 3090 GPU. This efficiency demonstrates the capability of our approach to handle 3D data with minimal computational overhead. During the rebuttal period, we faced time constraints that limited our ability to process larger datasets such as Objaverse. Our current focus was on demonstrating the model's capabilities within the constraints of available resources and time. The inherent efficiency of our method makes it well-suited for scaling up to larger datasets like Objaverse in the future.

Q2: Equation (3) and the Choice of $\tau$

In the paper, we have provided a comprehensive explanation of the role of $\tau$ and why its selection is critical, as demonstrated in our ablation study (Table B). By controlling the truncation of the GauPF, $\tau$ helps concentrate the learning process near the surface of objects. The truncation strategy prevents the learning from being influenced by distant regions in space, which are less relevant for accurate surface representation. Besides, in our ablation study, we explored both exponential and linear mappings for distance-to-probability transformations. While both methods are commonly used, our results showed that linear mapping performances better than exponential mapping in our framework. The linear mapping approach with a truncation strategy resulted in more stable and accurate training outcomes, highlighting the significance of selecting an appropriate distance-to-probability mapping technique for the task. And the ablation study identifies a suitable $\tau$ that balances sensitivity and specificity, ensuring efficient learning of the object's surface while maintaining overall model stability.

Q3: Evaluation of 3DGS Representation

During rebuttal, we further evaluate our 3DGS representation by randomly selected two airplanes from the dataset and modeled each using GauPF, GauCF, and GauTF. The results are shown in Figure C of the rebuttal PDF, which provide visual comparisons of the original and reconstructed objects, showcasing the accuracy and effectiveness of our approach in capturing intricate details. In Figure C, we also report PSNR scores, indicating a strong ability to reconstruct the original 3DGS with high fidelity. The integration of GauPF, GauCF, and GauTF ensures that both geometric and visual details are preserved during reconstruction. And Octree-Guided sampling allows for efficient handling of complex geometries, reducing computational overhead by focusing resources on critical areas. This makes our method scalable and practical for complex objects.

Q4: Suitability of 3DGS for 3D Generation

Compared to NeRF, 3DGS offers several advantages, particularly in terms of rendering efficiency and high visual fidelity. The task of solving 3DGS generation is not only timely but crucial for advancing the field of 3D modeling. As 3D content creation continues to grow in demand, having efficient and robust methods like 3DGS becomes increasingly important for various applications, including gaming, virtual reality, and digital content creation. One of the primary contributions of our work is addressing the challenges posed by the discrete and unstructual nature of 3D Gaussian Splatting. Our approach provides three continuous Gaussian Splatting functions to effectively embed 3DGS into a generative model, allowing for efficient and accurate 3D generation. As mentioned in our previous response, fitting the Gaussian representation for each shape is quite efficient, taking only 3 minutes on a single NVIDIA RTX 3090 GPU.

Q5: Comparison with GS-Based Reconstruction Models

Upon review, we conducted supplementary experiments to compare the visual quality of our method with several SOTA 3D generative models, including LGM and other GS-based approaches. The results of these visual comparisons are showcased in Figure A and Figure B of the rebuttal PDF. The visual results clearly demonstrate the strengths of our method in capturing intricate details and consistency with conditions. Our approach benefits from an efficient representation of Gaussians that enhances both processing speed and visual output quality, making it well-suited for high-fidelity content creation. In Table A of the manuscript, we provide a comparison of the CLIP Score between our method and the other SOTA 3D generative methods including Shap·E, LGM(ECCV 2024) and DreamGaussian(ICLR 2024). Our method demonstrates superior CLIP Scores, indicating that DiffGS generates 3D content that is more faithful to the input conditions. We have also included a comparison of model parameter count and generation time in Table B. DiffGS has a considerably smaller number of parameters compared to LGM, leading to decreased memory consumption and computational overhead. Additionally, our approach enables quicker generation time than other SOTA 3D generative models, showcasing its ability to efficiently create high-quality 3D content.

Q6: Correction of Optimization Target in Equation (9)

We appreciate the reviewer's careful examination of Equation (9) and for pointing out the error in the optimization target. The revised manuscript will reflect this correction.

[1] Shue, J.R., Chan, E.R., Po, R., Ankner, Z., Wu, J., Wetzstein, G.: 3d neural field generation using triplane diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20875–20886 (2023)

We sincerely thank Reviewer p5pi for the thorough review and valuable feedback. We have addressed each point in our response below.

Q1: Comparison with other recent 3D generative models

We refer the reviewer to the "Global-Q1: Evaluation Against Other SOTA 3D Generative Models" section of the global response for a comparison with other methods.

Q2: Detailed Structure of Proposed Models

Below, we address each point raised by the reviewer.

1. Latent dimension z:

The latent vector z in Figure 2 of the paper is designed to capture the essential features of the Gaussian Splatting representation. We set the latent dimension to 256 to balance expressiveness and computational efficiency. The choice of 256 dimensions is empirically determined to balance encoding diverse 3D shapes with manageable model size.

2. Gaussian Splatting Encoder

The Gaussian Splatting encoder transforms the input Gaussians into the latent space. We utilize a PointNet[1]-based architecture as the implementation of the encoder, which efficiently processes gaussian data . In detail, We adjust the dimensions of the input in PointNet within the SDF-VAE encoder to accept a K-dimensional 3DGS as input.

3. Design of GauPF/GauCF/GauTF

GauPF, GauCF, and GauTF share a similar architecture consisting of three MLP-based blocks. The input latent vector has a dimension of 131, comprising the latent dimension sampled from triplane and the XYZ coordinates. The feature channels for the first two blocks are [131,512,512,512,512] and [643,512,512,512,512] with a skip connection preceding the second block. To leverage the properties of Gaussians, the last block of GauTF is designed uniquely. To ensure the generated scale remains within a certain threshold, a truncation operation is applied to the scale attribute. Besides, in order to achieve uniformity in the form of the rotation quadruple, we normalize it so that its last dimension has a unit length. Finally, we use the sigmoid activation function to restrict opacity values to the range of 0 to 1.

4. Use of DALLE-2’s Latent Diffusion Model (LDM)

The core operation of LDM is performing diffusion on a one-dimensional latent space. This diffusion process is general and applicable to various data types, including 3D representations like triplanes. Although LDMs are frequently used for 2D image processing, their ability to handle latent spaces makes them suitable for broader applications. We begin by encoding the triplane data into a latent space suitable for latent diffusion. This encoding process captures the essential geometric and visual information needed for accurate 3D reconstruction. The LDM operates on this encoded latent space, applying diffusion processes.

5. Triplane Decoder Architecture

The triplane decoder reconstructs the 3D shape from the latent space using transposed convolution layers, interleaved with batch normalization and activation functions. This structure is designed to progressively upscale the 2D feature maps while maintaining high fidelity and detail.

6. Parameter Counts

The complete DiffGS model consists of approximately 127.4 million parameters. In Table B, we compare the parameter count of our method with other SOTA 3D generation methods. The results demonstrate the efficiency of DiffGS, which uses significantly fewer parameters than DiffTF (929.9M), ShapE (759.5M), and LGM (429.8M).

Q3: Addressing Post-Optimization Steps

We refer the reviewer to "Global-Q2: Addressing Post-Optimization Steps" for results and analyses.

Q4: Trade-Offs Between Computational Cost and Quality Enhancement

We refer the reviewer to ”Global-Q3: Balancing Computational Cost and Quality Enhancement“ for a detailed discussion on balancing computational costs with quality enhancement.

Q5: Inclusion of Text-Fidelity Metrics

We conducted experiments to evaluate the text fidelity of our text-to-Gaussian generation using CLIP-scores in Table A of the rebuttal PDF. Compared to LGM and other baseline models, our method shows a consistent improvement in CLIP-scores, confirming the superior text fidelity of our approach.

Q6: Decoding Triplane Representation into Gaussian Splatting

The triplane representation serves as an effective intermediate step that simplifies the complex task of directly regressing explicit 3D Gaussian attributes. Triplanes offer a structured approach to capture spatial information without being hindered by the non-structural nature of Gaussian splatting. The Gaussian splatting is inherently more efficient for training and rendering compared to volumetric approaches like NeRF. It utilizes point-based representations, which require less computational overhead and facilitate quicker convergence during training. The use of Gaussian splatting enables faster rendering speeds, making our approach suitable for real-time applications such as virtual reality and interactive 3D simulations.

Q7: Justification for Training a Domain-Dedicated 3D Generator

SDS-based methods require extensive optimization and are computationally intensive, particularly when distilling scores from 2D to 3D. They are also highly sensitive to hyperparameter settings like learning rates and noise schedules. Additionally, these models often produce the Janus problem in 3D shapes. We make a comparison with SDS-based Gaussian generative model DreamGaussian in Figures A and B, as well as Table B. The results demonstrate that our dedicated 3D generator DiffGS performs faster and produces more accurate, high-fidelity geometric structures than SDS-based methods.

Q8: Typographical Error

We thank the reviewer for identifying the typographical error in line 109, which will be corrected in the revised manuscript.

[1] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.

Dear Reviewers,

As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification.

Thanks,

The Authors