Atlas Gaussians Diffusion for 3D Generation

Thank you for your insightful and valuable feedback. Below are our clarifications for your concerns. We value your input and are keen to discuss any further questions or comments you may have following our response.

W1. Latent Diffusion Novelty: I believe the variational diffusion model is not a novelty in terms of technical development. Besides, the authors directly utilized the EDM algorithm as their latent diffusion.

We do not claim that using EDM or designing a new latent diffusion model is our contribution. This is why Section 3.3 is intentionally brief, comprising only about 10 lines. As stated in L345: "Since the LDM network design is not our main contribution."

Our primary contribution lies on the VAE side rather than the LDM. To apply latent diffusion to 3DGS, the critical prerequisite is designing a VAE that can efficiently map low-dimensional latents to 3DGS. This requirement is independent of the diffusion framework used (e.g., DDPM, EDM, or flow matching) or the LDM network architecture employed (e.g., those inspired by 3DShape2VecSet or DiT). To address this prerequisite, we propose:

• Atlas Gaussians representation (Contribution 1, L102–103).
• A decoder to generate Atlas Gaussians (Contribution 2, L104–105).

These contributions directly address the critical prerequisite by providing a VAE capable of efficiently mapping low-dimensional latents to 3DGS, which is the core focus of our work.

To our best knowledge, at the time of submission, no prior work had applied the VAE + LDM paradigm to generate 3DGS. Therefore, we claim integrating 3DGS into the VAE + LDM paradigm as our third contribution (L106–107), while explicitly not claiming LDM itself as our novelty.

Our contributions have also been acknowledged by other reviewers. For example:

• Reviewer 3ymM mentions: "Addressing the current challenges of VAEs in extracting efficient latent representations for generative diffusion models is important."
• Reviewer uWac mentions: "The paper nicely integrates 3DGS with diffusion generation scheme".
• Reviewer pZYd mentions: "Sufficient experimental results support the contributions claimed."

What makes the integration of the latent diffusion model a unique/novel contribution other than working with a different data type such as 3D Gaussians?

The novelty lies in addressing the challenges posed by the high dimensionality of 3DGS, especially when a very large number of Gaussian points is required (e.g., 100K). Directly applying a diffusion model to such a large number of Gaussian points, instead of a compressed latent space, is highly challenging due to optimization difficulties and slower sampling.

To tackle these challenges, our approach follows a common practice in generating high-dimensional data: compressing the data into a low-dimensional latent space before applying latent diffusion. This integration is crucial for enabling the efficient and scalable generation of 3DGS.

Do the authors plan to open source the model?

Yes, we have included the code in the supplementary material and commit to open-sourcing the model.

Can UV-based sampling potentially generate artifacts? (e.g. inconsistency or distortions)

Artifacts could potentially arise due to inconsistencies across patches, as mentioned in L319: "Note that patches may overlap, similar to AtlasNet." However, during rendering, we only consider the union of all 3D Gaussian points decoded from the patches. As long as this union globally matches the ground truth images, local inconsistencies across patches do not affect the overall representation. Empirically, we observe that our native 3D representation generalizes quite well.

How well does the proposed framework handle noisy or sparse data?

For multi-view inputs, our method can handle noisy camera poses because the poses of the input images are not fixed. We train our VAE directly on renderings from the G-buffer Objaverse (L361), where the elevation of the input image views ranges from 5° to 30° (excluding bottom views). This variability demonstrates the robustness of our approach to pose noise.

For point cloud inputs, we conducted an ablation study using the ShapeNet Chair dataset and evaluated the PSNR of the VAE. As shown in the table below, we added Gaussian random noise to the input point cloud coordinates with a distribution of $\mathcal{N}(0, \sigma^2)$. When the noise scale is small ($\sigma=0.001$), the performance remains almost unchanged. However, as the noise scale increases ($\sigma = 0.01$), the performance decreases more significantly. Additionally, we performed experiments using sparse input point clouds with only 1024 points. The results demonstrate that our framework is capable of handling sparser data.

Thank you for your insightful and valuable feedback. Below are our clarifications for your concerns.

Comparison on inference times with the increase of # of 3DGS.

We provide the computational times (measured on a V100 GPU) with an increasing number of 3DGS (#3DGS) in the table below. As #3DGS increases, the computational overhead primarily comes from decoding more 3DGS in the decoder and rendering them. During inference, the denoising time in latent diffusion remains constant, while the decoder and rendering time increases as #3DGS increases. However, compared to the denoising time, the computational overhead from using more 3DGS is negligible.

How could the current approach address gap for even finer details of the shapes as Gaussian representation struggles in fine details and often results in blurry effect. Is it only a matter of the # of Gaussians or patches?

Increasing the number of 3DGS could contribute to capturing finer details. However, it is not solely a matter of the number of 3DGS. We find that optimization, specifically how the network is trained, also plays a crucial role. For example, aligning the locations of 3DGS to the surface (as described in L320) provides a better initialization and facilitates the learning of finer details. This is why we adopt a two-stage training strategy (L969), following (Xu et al., 2024a) and (Zou et al., 2023).

That said, efficiently aligning a large number of 3DGS (e.g., 100k) to the surface remains a challenging problem. As a workaround, we currently sample a subset of points and compute CD/EMD on this subset ( $\mathcal{L}_\mu(S)$ ). With increased computational resources and more efficient loss functions for better initializing the 3DGS locations, achieving finer details could become more feasible.

The author mentioned "We take inspiration from the literature that 3D-based neural networks learn better 3D features for recognition and outperform multi-view based networks that leverage massive 2D labeled data." without any citation. There are arguments on both sides about overall performance with both approaches. This sentence seems to be subjective.

Thank you for pointing this out. Our statement is inspired by the development of 3D recognition tasks in the deep learning era, and we acknowledge that there are ongoing debates regarding the performance of different approaches. Taking 3D shape classification as an example, early methods used multi-view representations as input for 3D shape classification tasks [1]. Subsequently, 3D-based neural networks were introduced and demonstrated better performance, as seen in methods like PointNet [2] for point clouds. A chronological overview and comparison of multi-view-based networks versus 3D-based networks can be found in Figure 2 of the survey [3]. For other 3D recognition tasks, such as 3D object detection and 3D semantic segmentation, detailed comparisons are provided in Figures 7 and 10 of the same survey [3]. We will include these references in the final version to make the statement more balanced and objective.

[1] Multi-view Convolutional Neural Networks for 3D Shape Recognition

[2] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

[3] Deep Learning for 3D Point Clouds: A Survey

What does $d$ represent in L189 and L135 (Equation 4)?

$d$ is the feature dimension of $\mathbf{f}\_{ij}$ and $\mathbf{h}\_{ij}$, where $j=\{1, 2, 3, 4\}$. For example, $\mathbf{f}\_{i1}\in \mathbb{R}^d$, $\mathbf{f}\_{i2}\in \mathbb{R}^d$, $\mathbf{f}\_{i3}\in \mathbb{R}^d$, $\mathbf{f}\_{i4}\in \mathbb{R}^d$. Thus, $\mathbf{f}\_{i}=(\mathbf{f}\_{i1}, \mathbf{f}\_{i2}, \mathbf{f}\_{i3}, \mathbf{f}\_{i4})\in \mathbb{R}^{4\times d}$ . This definition is also consistent with Equation (4).

What does $n$ represent in L256 and L268?

$n$ represents the size of the set in the latent set representation. As mentioned in L257, the encoder ouputs $\mathbf{z}_0$, which is a latent set representation first proposed by 3DShape2VecSet. Intuitively, a vanilla VAE uses a single latent vector of dimension $d_0$ to define the latent space. In contrast, the latent set representation defines the latent space using a set of latent vectors, where each vector is of dimension $d_0$, and the size of the set is $n$. Therefore, the latent set $\mathbf{z}_0\in\mathbb{R}^{n\times d_0}$ . We will clarify this further in the final version.

What is the exact relationship between the shape information $\mathcal{S}$ (L263) and the point cloud $\mathcal{P}$ as well as the sparse-view RGB images $\mathcal{I}$?

The shape information $\mathcal{S}$ includes both the point cloud $\mathcal{P}$ and the sparse-view RGB images $\mathcal{I}$. In terms of the mathematical formula, you are correct in the previous comments: $z' = \text{CrossAttn}(\bar{z}, \mathcal{P}), z'' = \text{CrossAttn}(z', \mathcal{I}), z_0 = \text{MLP}(z'')$. We will clarify this further in the final version.

How is the learnable query $\mathbf{y}$ of the VAE decoder (L277, L284) initialized?

The learnable query $\mathbf{y}$ is initialized using a Gaussian distribution, implemented as: torch.nn.Parameter(torch.randn(n, d) * 0.02).

Thank you for your insightful and valuable feedback. While three other reviewers have rated our presentation as either "excellent" or "good,", we acknowledge that there is always room for improvement. We are grateful for your constructive suggestions and are committed to addressing your concerns to enhance the clarity and presentation of our paper. Below, we provide clarifications and responses to your comments.

I suspect the term “atlas” is intended in the same sense as in AtlasNet (L125~128).

Yes. The term "atlas" has the same meaning as in AtlasNet. As a concept in mathematics, an atlas is a collection of charts that collectively cover a manifold. We will add the explanation in the final version.

Figure 3 lacks detail and does not clearly relate to the main text, particularly Section 3.2, Stage 1: VAE. For instance, (1) ... (2) ...

(1) Is the process $z' = \text{CrossAttn}(\bar{z}, \mathcal{P}), z'' = \text{CrossAttn}(z', \mathcal{I}), z_0 = \text{MLP}(z'')$ correct?

Yes, that is correct. We will clarify this point and add the explanation in the final version.

(2) My understanding is that $z_0$  of Figure 3 represents the end of the VAE encoder, while subsequent steps are part of the decoder.

Yes, that is correct. As mentioned in the paper:

• L256: "Encoder. The encoder takes shape information as input and outputs a latent set $z_0$"

• L272: "Decoder. The decoder recovers patch features ... from the latent code $z_0$"

• L227: " The latent $z_0$ is used for latent diffusion."

  We will clarify this further in the final version of the paper.

The conditioning ( $\mathcal{C}$ of Section 3.3) mechanism lacks detail

The conditioning $\mathcal{C}$ is represented as a vector:

• For unconditional generation, $\mathcal{C}$ is set as a learnable parameter, implemented as torch.nn.Embedding(1, d).
• For text-conditioned generation (conditional generation), $\mathcal{C}$ is set as the CLIP embedding of the input text prompts, which is also a vector.

In Equation (12) of the main paper, $\mathcal{C}$ serves as the key/value in the cross-attention mechanism. We will provide additional details in the final version to make this clearer.

An ablation study should be conducted to analyze the impact of the loss hyperparameters ($\lambda_{r}$ and $\lambda_{KL}$).

Regarding $\lambda_r$, as mentioned in the supplementary material (L972), doubling or halving $\lambda_r$ results in almost identical loss curves. For simplicity, we set $\lambda_r=1$.

To analyze the effect of $\lambda_{KL}$, we conducted an ablation study using the ShapeNet Plane dataset and evaluated the PSNR of the VAE. As shown in the table, decreasing $\lambda_{KL}$ leads to improved performance. However, in the VAE + LDM paradigm, the KL loss plays an important role in preventing arbitrarily high-variance latent spaces. Balancing these factors, we set $\lambda_{KL}=1e^{-4}$ for simplicity.

L060 "However, these approaches often focus solely on modeling geometry without considering the appearance attributes." Can you give a more detailed explanation?

In many previous works, the focus has been on generating representations of the geometric surface, such as 3D point clouds or iso-surfaces. These methods do not generate the colors or textures of the 3D objects, meaning the appearance attributes are not considered. For example, LION generates only 3D points, while 3DShape2VecSet generates occupancy fields. These representations inherently lack color information.

Thank you for your insightful and valuable feedback. Below are our clarifications for your concerns.

The proposed approach includes both the Atlas Gaussian representation with disentangled geometry and appearance learning mechanism, as well as a transformer-based encoder. It is unclear which component contributes more significantly to the overall performance.

To ensure I understand your comment correctly, let me rephrase it: suppose we have a hypothetical baseline VAE capable of generating 3DGS. Adding our transformer-based encoder to this baseline would result in a performance gain of $\Delta_1$​, while adding our Atlas Gaussian representation would result in a performance gain of $\Delta_2$. Are you asking which of these gains is larger, i.e., which component contributes more significantly to the overall performance?"

Unfortunately, there is no existing baseline VAE using 3DGS at the time of our submission. Without such a baseline as a reference, it is challenging to quantitatively compare the individual contributions of these components in isolation.

Based on our best interpretation of your comment, this question may reflect a concern about whether all the new components in our VAE are necessary for achieving the reported performance. We believe both components are indeed crucial, though we currently lack direct quantitative measures for $\Delta_1$​ and $\Delta_2$​. However, we have evidence supporting the necessity and effectiveness of each component individually, i.e. both $\Delta_1 > 0$ and $\Delta_2 > 0$:

• Transformer-Based Encoder: Our design follows the approach in 3DShape2VecSet, leveraging a latent set representation. The usefulness of this design has been demonstrated through the ablation studies in the original 3DShape2VecSet paper. We build on their proven techniques to enhance our model.

• Atlas Gaussian Representation: We provide an ablation study in Table 4 in the main paper to validate its effectiveness. The study compares the full Atlas Gaussian design with simpler alternatives (e.g., linear weights, no disentanglement, no global feature, etc.), showing that the full design outperforms these naive baselines. This supports our claim that the representation is a necessary and effective improvement.

It is unclear how geometry and appearance features are disentangled in the latent space.

The geometry and appearance features refer to $\mathbf{f}_i$ and $\mathbf{h}_i$, respectively, as defined in Lines 188–189 of the paper. These features are not defined in the latent space. The disentanglement of geometry and texture features is achieved through the use of separate branches to learn them (L105, L516).

As mentioned in L520, both geometry and appearance features are generated from the shared latent $\mathbf{z}_0$. This means the latent space is not explicitly divided into two subspaces, where one corresponds to geometry and the other to appearance. Instead, the disentanglement is performed during feature generation. We will clarify this point further in the final version.

The process for learning geometric features remains vague.

As shown in Equation (2) of the main paper, the geometry features $\mathbf{f}_i$ are used to predict the Gaussian location $\mu$. These Gaussian location $\mu$ are supervised through Equation (9), which regularizes $\mu$ to align with the GT surface. During end-to-end training, the geometry features $\mathbf{f}_i$ will also receive gradients, enabling their learning.