GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:

Q: Parameter distribution of 3D Gaussians used for diffusion process.

A: Although the parameters of 3D Gaussians have very different distributions, we observe that after applying data normalization, they mainly obey the normal distribution, i.e., $\mathcal{N}(\mathbf{0}, \boldsymbol{I})$. Consequently, the data distribution does not present any significant divergence from that employed in the conventional diffusion process.

Q: How to guarantee the predicted properties are valid?

A: We employ quaternion representations for rotations and normalize the predicted rotations to procure a valid unit quaternion following original Gaussian Splatting. We clamp the prediction of opacity $\alpha$ to $[0, 1)$ and clamp the minimum value of predicted scaling $\mathbf{s}$ to $0$ to ensure validity. No additional operations are applied to the positional and color feature predictions. Our findings suggest that these processes effectively validate the predicted Gaussians, yielding satisfactorily rendered images. We will incorporate these specifics into the revised manuscript.

Q: The distribution of the offsets from voxel centers.

A: Thank you for your insightful suggestion. Please see Figure 3 in the attached PDF in the top-level comment. We visualize the offset distribution of 1K randomly selected GaussianCubes from each experimental dataset in Figure 3. We observe that most distributions exhibit a bell curve, similar to a normal distribution. However, the Digital Avatar dataset presents a more uniform distribution with multiple peaks. We believe these distributions offer valuable insights into how well the fitted 3D Gaussians align with voxel grid centers. Bell-shaped distributions akin to a normal distribution, such as in the ShapeNet Car and Chair datasets, suggest a strong initial alignment and lower complexity. On the other hand, broader distributions (e.g., the Digital Avatar dataset) indicate a higher level of detail (for instance, hair) and a greater need for adjustments during organization.

Q: The positions of Gaussians are replaced by offsets of voxel centers should be explained earlier.

A: Thanks for your valuable comments. We will explain this operation earlier in the revised paper.

Q: Illustration of Fig. 3 of the main paper.

A: We apologize for any confusion caused by the optimal transport illustration in Figure 3 of the main paper, where we provided a 2D toy example of two pixels. Actually, each pixel on the grid is valid rather than just two pixel corners on the diagonal. We appreciate your insightful suggestion and will revise this figure accordingly in the updated version.

Q: Missing related works of [1] and [2].

A: Thanks for pointing out this. We will discuss these works in our revision.

Q: Missing comparison with [1] on ShapeNet.

A: We have extended our analysis to include a comparison with SSDNeRF[1] on ShapeNet Car. By evaluating the official model checkpoint of SSDNeRF, we obtained the FID and KID scores between 50K generated renderings and 50K ground-truth renderings, aligning with the data in Table 3 of the main paper. The quantitative evaluation indicates that our method outperforms SSDNeRF in terms of both the FID and KID scores. The visual comparison in Figure 4 of the attached PDF in the top-level comment demonstrates that our model is able to generate high-fidelity results, surpassing SSDNeRF on the level of detail.

Q: Why do we use $x\_0$-parameterization instead of $\epsilon$ or $v$?

A: Previous work [3] demonstrates that $x\_0$, $\epsilon$ and $v$ predictions only result in varied loss weights in diffusion training. We investigate both of them during earlier exploration and find $x\_0$-parameterization achieves the best performance. Therefore, we employ $x\_0$-parameterization in all subsequent experiments.

Q: Potential for latent diffusion?

A: That's an insightful observation. Existing 2D latent diffusion models successfully achieve scale-up and exhibit amazing generative results. As such, 3D latent diffusion may potentially enhance the scalability of our model. However, this would necessitate the additional training of a Variational Autoencoder (VAE) that needs to be carefully designed to maintain the quality of reconstruction. We plan to explore this aspect further in our future research.

Q: Performance of DiffTF.

A: We infer the official checkpoints of DiffTF and report the FID score between 50K generated and 50K ground-truth renderings rather than 50K generated and all ground-truth renderings used in the original paper. All reported FID and KID in Table 3 of the main paper are evaluated between 50K generated and 50K GT.

[1] Chen H, Gu J, Chen A, et al. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 2416-2425.

[2] Schröppel P, Wewer C, Lenssen J E, et al. Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8785-8794.

[3] Salimans T, Ho J. Progressive distillation for fast sampling of diffusion models[J]. arXiv preprint arXiv:2202.00512, 2022.

I thank the authors for their clarifications regarding my concerns and providing additional results. The rebuttal addresses the mentioned weaknesses and questions. After carefully considering all reviews and answers by the authors, I would like to increase the rating to 7: Accept (see edited score in review).

The following points prevent me from giving an even higher rating:

• Limited contribution: Application of optimal transport for voxelization
• Scalability concerns:
  • high computational demand for training,
  • rather low resolution of 32x32x32,
  • and therefore the limitation to objects only

Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:

Q: Time and memory analysis of our method.

A: Thanks for your suggestions. As detailed in the supplementary material (Lines 501-503), the proposed densification-constrained fitting requires approximately 2.67 minutes on a single V100 GPU for each object over 30K iterations. The OT-based structuralization takes around 2 minutes per object on an AMD EPYC 7763v CPU. Fitting a single object consumes around 1GB of GPU memory. OT is run on CPU and costs around 5.4GB memory. For the diffusion training, we deploy 16 Tesla V100 GPUs for the ShapeNet Car, ShapeNet Chair, OmniObject3D, and Synthetic Avatar datasets, whereas 32 Tesla V100 GPUs are used for training on the Objaverse dataset. It takes about one week to train our model on ShapeNet Car, ShapeNet Chair, and OmniObject3D, and approximately two weeks for the Synthetic Avatar and Objaverse datasets. We provide the detailed inference configuration of DPM-solver and speed under various inference timesteps in the table below. Please note that all the inference times of the diffusion models in Table 5 of the main paper are reported using 100 steps to ensure a fair comparison.

Q: Novelty analysis of generation results on ShapeNet.

A: In our experiments, we perform nearest neighbor search of our generated samples on ShapeNet dataset, as depicted in Figure 12 of the main paper. The results demonstrate that our model is capable of generating novel geometry and textures rather than simply memorizing the training data.

Q: Is the digital avatar creation model conditioned on image?

A: Yes. The digital avatar creation model is conditioned on a single portrait image.

Q: Is it possible to train an unconditional digital avatar creation model?

A: Certainly. We can train an unconditional model on a digital avatar dataset using the same methodology as with the ShapeNet datasets. In fact, our image-conditioned digital avatar creation diffusion model utilizes classifier-free guidance, allowing us to conduct unconditional inference simply by dropping the image conditions. Please refer to Figure 2 of the attached PDF in the top-level comment for our unconditionally generated digital avatars.

Dear Reviewer,

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.

Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:

Q: Given that many 3D generative models are based on images or videos due to the abundance of 2D data, how data scalability can be achieved for our approach?

A: We acknowledge the reviewer's point that data holds significant importance in scaling up models. Despite that some 3D generation works are based on images and videos (e.g., MVDream, SV3D), a substantial amount of 3D data is still necessary for training in order to generate 3D consistent results. Therefore, we believe large-scale, high-quality 3D data is important for both our approach and the 2D-based methods. Recent trends indicate an active research interest in the development of expansive 3D datasets, such as Objaverse-XL[1], which boasts over 10 million 3D objects. The availability of such vast volumes of 3D data paves the way for us to scale up our model. Nevertheless, our approach is not limited to using 3D data only. We are also able to harness 2D image or video priors to further refine our generated results (e.g. leveraging Score Distillation Sampling with a pretrained 2D diffusion model).

Q: Expressiveness and generalization capability of our model to fit many datasets jointly.

A: Regarding data fitting, the explicit nature of our representation eliminates the need for the shared implicit feature decoder and enables us to achieve high-quality fitting across all datasets. The fitting experiments in Table 1 and Table 2 of the attached PDF of the top-level comment demonstrate the expressiveness and generalization capability when jointly fitting different datasets (fitting results of GaussianCube are done before paper submission). Regarding modeling the distribution of GaussianCube using diffusion models, Objaverse serves as an exemplary case. This large-scale dataset with intricate data distribution encompasses most categories in ShapeNet. Given that our model yields high-quality generation results on Objaverse, we believe our method is effective and generalizable to handle complex datasets jointly.

Q: How to scale the representation for complicated topology and intricate details?

A: GaussianCube is a highly expressive representation as we have demonstrated. A modest number of Gaussians (e.g., $32^3=32,768$) can effectively capture complex objects with high fidelity (e.g., the tire tread and hair of avatars in Figure 4 and Figure 7 of the main paper). Furthermore, we evaluate our method on the more challenging NeRF Synthetic dataset[2]. Our GaussianCube can approach the fitting quality of original GS using only $32,768$ Gaussians. For more complicated topology like Mic, our method achieves on-par quality by enlarging the corresponding voxel size and number of Gaussians (e.g., to $48^3=110,592$) without extra elaborate design to our pipeline. Please see the attached PDF in our top-level comment for visual comparison.

Q: Training time and compute requirement for diffusion model training.

A: All of our diffusion model is trained using Nvidia Tesla V100 32G GPUs. It takes about one week to train our model on ShapeNet Car, ShapeNet Chair, and OmniObject3D, and approximately two weeks for the Synthetic Avatar datasets using 16 GPUs. For Objaverse dataset, the training duration extends to around two weeks with 32 GPUs. We will add the training time and compute information in the revised paper.

Q: Necessity of Figure 3.

A: Thanks for your suggestion. We will consider revising it in the revision.

Q: Failure cases of optimal transport.

A: We utilize the Jonker-Volgenant algorithm to resolve the optimal transport problem, which invariably yields the optimal solution. Consequently, we don't identify any failure cases of optimal transport in our model. If possible, we would appreciate further elaboration from the reviewer on what constitutes a failure case in the context of optimal transport. This would facilitate a more productive discussion during the reviewer-author dialogue period.

Q: Have we considered a learnable scattering of Gaussians to a cubic grid?

A: That is indeed a compelling proposition. One potential advantage of a learnable scattering mechanism is the elimination of the time cost of OT. However, integrating it into the current pipeline presents several challenges. For example, the scattering process would need to be meticulously designed to ensure it is both differentiable and bijective. Additionally, jointly training the scattering with the diffusion model could potentially lead to instability issues. We will leave it to future research and thank the reviewer for the suggestion.

Q: Discussion of large-scale scene generation.

A: Scene generation is an interesting topic. While the experiments conducted with GaussianCube primarily focus on objects, the generality of our pipeline suggests the potential applicability to scene generation as well. Nonetheless, scene generation comes with its own set of challenges. For one, handling unbounded scenes would require specialized design (e.g., contracting unbounded space into a ball in MipNeRF-360), given that GaussianCube necessitates a voxel grid within the bounding box. Additionally, the scarcity of large-scale scene data poses significant challenges for generative modeling. We plan to delve into these aspects in our future research.

[1] Deitke M, Liu R, Wallingford M, et al. Objaverse-xl: A universe of 10m+ 3d objects[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Mildenhall B, Srinivasan P P, Tancik M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.

Dear Reviewer,

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.

Thank you for your valuable comments and suggestions. We address the reviewer's concerns below:

Q: Parameters of Triplane.

A: In Table 2 of the main paper, we assigned the size of the Triplane as $3\times256\times256\times32$. We set the size of Voxels to $32\times32\times32\times14$ with the intent to draw a comparison with GaussianCube of a similar representation size. We additionally compare with Triplane of size $3\times128\times128\times10$, which has similar representation size with GaussianCube and Voxels of $32\times32\times32\times14$. Furthermore, we include results from Voxels of size $128\times128\times128\times32$, which yield much higher fitting quality than the $32\times32\times32\times14$ counterpart and exhibit a comparable fitting quality to both GaussianCube and Triplane. The additional results are included in the attached PDF of the top-level comment. As indicated in Table 1 of the attached PDF, the Triplane is considerably more efficient than the Voxels, offering superior fitting quality than Voxels with larger parameters. However, our GaussianCube still surpasses Triplane and Voxels in terms of fitting quality using the fewest parameters.

Q: When encoding a large dataset, do triplane or voxels (with shared implicit decoder) still have more parameters than GaussianCube?

A: Yes. Due to the GaussianCube's explicit nature, it eliminates the requirement for a shared implicit decoder across various objects, leading to no significant difference in fitting quality when fitting larger datasets. Our representation size remains consistent at $32\times32\times32\times14$ across all datasets. We additionally conducted an experiment involving the fitting of the Objaverse dataset, which is not only larger but also exhibits a much more diverse distribution compared to ShapeNet. The experiment can be found in the attached PDF of the top-level comment, where we report the fitting quality of 100 randomly selected objects. As demonstrated in Table 2 of the attached PDF, our GaussianCube still achieves superior fitting quality while utilizing the minimum number of parameters compared with Voxels and Triplane.

Q: Is Equation (4) used to train the diffusion model?

A: Yes, Equation (4) is used for diffusion training.

Q: How to obtain $I\_{\text{pred}}$?

A: Our model is parameterized to predict the noise-free input $\mathbf{y}\_0$. For each training step, let $\hat{\mathbf{y}\_0} = \hat{\mathbf{y}\_\theta}\left(\alpha_t \mathbf{y}\_0+\sigma_t \mathbf{\epsilon}, t,  \mathbf{c}\_{\text{cls}}\right)$ be the prediction of our model. We can directly rasterize $\hat{\mathbf{y}\_0}$ to obtain $I\_{\text{pred}}$ without multiple denoising steps during diffusion training.

Q: How does the expectation depend on $t$ in Equation (4)?

A: Given the model's prediction $\hat{\mathbf{y}\_0}$ and the camera parameters $***E***$ used for rasteraization, we obtain $I\_{\text{pred}}$ by

$ \begin{array}{ll} I\_{\text{pred}} &= \text{Rasterize}(\hat{\mathbf{y}\_0}, ***E***) \\\\ &= \text{Rasterize}(\hat{\mathbf{y}\_\theta}\left(\alpha\_t \mathbf{y}\_0+\sigma\_t \mathbf{\epsilon}, t, \mathbf{c}\_{\text{cls}}\right), ***E***). \end{array} $

Therefore, Equation (4) can be written as

$ \begin{array}{ll} \mathcal{L}\_{\text {image }} &= \mathbb{E}\_{t, I\_{\text{pred }}}\left(\sum\_l\left\\|\Psi^l\left(I\_{\text{pred}}\right)-\Psi^l\left(I\_{\text{gt}}\right)\right\\|\_2^2\right) +\mathbb{E}\_{t, I\_{\text {pred}}}\left(\left\\|I\_{\text {pred}}-I\_{\text {gt }}\right\\|\_2\right), \\\\ &= \mathbb{E}\_{t, \mathbf{y}\_0, \mathbf{\epsilon}}\left(\sum\_l\left\\|\Psi^l\left(\text{Rasterize}(\hat{\mathbf{y}\_\theta}\left(\alpha\_t \mathbf{y}\_0+\sigma\_t \mathbf{\epsilon}, t, \mathbf{c}\_{\text{cls}}\right), ***E***)\right)-\Psi^l\left(I\_{\text {gt}}\right)\right\\|\_2^2\right) \\\\ &+\mathbb{E}\_{t, \mathbf{y}\_0, \mathbf{\epsilon}}\left(\left\\|\text{Rasterize}(\hat{\mathbf{y}\_\theta}\left(\alpha\_t \mathbf{y}\_0+\sigma\_t \mathbf{\epsilon}, t, \mathbf{c}\_{\text{cls}}\right), ***E***)-I\_{\text {gt }}\right\\|\_2\right), \end{array} $

which illustrates that the expectation of Equation (4) depends on $t$. We thank the reviewers for this question and will further revise Equation (4) in the revision to improve its clarity.

Q: Is our method able to generate meaningful results on a small dataset containing objects of different orientations?

A: That is an interesting question. Our method does not require the objects to be axis-aligned and has demonstrated strong capability to model complex distribution (e.g., Objaverse). Consequently, we believe our model is capable of providing meaningful results with a small dataset comprising objects of varying orientations. However, it is still challenging to fully represent the complete data manifolds using only a small amount of data samples. If the dataset size is excessively small, the model may be exposed to the risk of having limited extrapolation capabilities.

Dear Reviewer,

We have tried our best to address your questions as detailed in our top-level comment and the rebuttal above. Once again, we extend our gratitude for your time and effort in reviewing our manuscript, and we stand ready to provide any additional information that could be helpful.