DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Does the timestep annealing help in text-to-3D too?

Thanks for the advice! Since image-to-3D has a strong reference view image prior, we don't observe obvious instability during training.

For text-to-3D, we have updated new text-to-3D results using MVDream [1] as the guidance model in the appendix. An ablation study shows that timestep annealing is still helpful in generating a more reasonable shape with the same amount of training iterations. However, we do observe that different input prompts may require different number of training iterations to converge, so a linear annealing may not be the optimal way.

Q2: In the refinement stage, what happens if the image-to-image diffusion changes the object boundary? How to prevent the color from background leaking into mesh?

Since we use a relatively small noise level for image-to-image diffusion, boundary change is not very obvious. Also, since the optimization is performed from multiple camera views, it could be corrected from another view should any color leaking happened in one view.

Q3: Comparison of the texture refinement stage with texture generation methods like Texfusion and Text2tex.

Thanks for reminding us! Although we both generate texture on a fixed mesh geometry, there are several differences:

(1) These methods require a text prompt as the input. However, we don't have a prompt in our image-to-3D setting.

(2) We have a coarse texture and we want to enhance the details based on it. These methods focus on texture generation instead of refinement and will ignore the coarse texture, which is not suitable especially for image-to-3D.

We have updated the paper to discuss these methods.

Q4: Text-to-3D often generates over-saturated texture.

Thanks for mentioning this! We have updated the paper and stated this problem as a shortcoming in the limitations.

Q5: Missing references on methods addressing Janus problem without using 3D assets.

Thanks for reminding us! We have updated and discussed these methods in the paper.

Q6: There are many typos in the paper.

Thanks for correcting us! We have revised and updated the paper to fix these typos.

Q7: Why choose not to use a learnable background model?

The major reason is that we want to keep the pipeline simple, and we find that using random black or white background is enough for Gaussians to converge. We are not the first to choose this design, as Fantasia3D [2] also adopts a solid background color during optimization.

The learnable background model is majorly adopted in NeRF-based method. One possible explanation is that NeRF tends to form the background during early optimization, which is undesired since we want NeRF to form the target object. However, we observe that it's less likely for mesh or Gaussian-based method with explicit geometry to form the background, so a learnable background model is unnecessary.

[1] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).

[2] Chen, Rui, et al. "Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation." arXiv preprint arXiv:2303.13873 (2023).

We hope our responses satisfactorily address your queries. Please let us know to address any further concerns impacting your review.

Dear Authors,

Thank you for addressing my previous feedback in your rebuttal.

I acknowledge the emphasis your paper places on the speed of generation, which is indeed a significant contribution. However, I must express my reservations regarding the quality of the output. Compared to other concurrent works, the quality here seems to be notably inferior. This aspect is particularly critical, as it undermines the overall impact of the work.

Furthermore, while the contribution towards texture enhancement is noted, its value is somewhat diminished due to the overall low quality of the output. This limitation prevents the texture contribution from being as meaningful as it could be.

In light of these observations, I have decided to maintain my original score for the paper. I believe the paper should be accepted due to its notable contribution in terms of generation speed. However, given the concerns regarding output quality and the relatively minor impact of the texture contribution, I do NOT recommend it for selection as one of the top papers at ICLR.

I hope this feedback is helpful for your future endeavors in research.

Best regards,

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: The text-to-3D results are limited.

Thanks for your advice! Our paper does focus more on image-to-3D, since text-to-3D generation takes longer computing time and is more prone to suffer from the Jasus problem.

As stated in the limitations, it's promising to solve the Janus problem with recent multi-view or camera-conditioned 2D diffusion models. We have updated new text-to-3D results using MVDream [1] as the guidance model in the appendix, which shows better results for difficult prompts.

Q2: What is the advantage of the proposed refinement stage compared to Fantasia3D?

The key advantage of our refinement stage lies in its efficiency.

Unlike other methods, which typically start without initial textures and require considerable time for generation, our approach builds on an existing mesh with coarse textures. For example, the texturing stage in Fantasia3D can take up to 20 minutes using 8 GPUs, whereas our refinement stage enhances the coarse texture in just 1 minute on a single GPU.

Q3: Differences compared to GSGEN which optimizes for longer time.

There are several different designs that lead to the different conclusions about optimization time:

(1) GSGEN prioritizes high-quality text-to-3D generation, which is more challenging than image-to-3D. To address the Janus problem, they also generate a point cloud using Point-E [2] to initialize the Gaussians.

(2) They introduce a novel compactness-based densification strategy, which is more suitable for longer optimization.

(3) In addition to the 2D SDS loss using Stable-diffusion, GSGEN applies a 3D SDS loss with Point-E [2], potentially prolonging each optimization step.

In contrast, our method emphasizes efficiency, trading off some level of detail. We also observe that text-to-3D typically requires longer optimization to achieve finer details, especially with an appropriate densification strategy.

[1] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).

[2] Nichol, Alex, et al. "Point-e: A system for generating 3d point clouds from complex prompts." arXiv preprint arXiv:2212.08751 (2022).

We hope our responses satisfactorily address your queries. Please let us know to address any further concerns impacting your review.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Missing details about the model size, complexity, computation bottlenecks.

Thanks for reminding us!

The size of the generated 3D models varies depending on the input and training schedule. Typically, image-to-3D results involve around $10^4$ 3D Gaussians, while text-to-3D can reach up to $10^5$ Gaussians due to a more aggressive densification strategy.

For the mesh extraction, we apply remeshing and decimation as post-processing so the size is controlled. Specifically, we perform isotropic remeshing to an average edge length of $0.015$, and then decimate the number of faces to $10^5$.

The primary computational bottleneck in our pipeline remains the forwarding of the 2D diffusion model. We have broken down the time consumption for each operation per iteration as follows:

Our method primarily reduces generation time by requiring fewer iterations to converge (from several thousand to 500 steps). However, each iteration is still constrained by the 2D diffusion model.

For the mesh extraction, the time consumption for each operation is detailed as:

Q2: How to control the mesh complexity? How to solve non-watertight/manifold meshes?

As mentioned in response to Q1, we control mesh complexity through post-processing, limiting the number of faces to $10^5$ through decimation. These post-processing details have been added to the implementation details.

We acknowledge the potential for non-manifold or non-watertight meshes in cases of poor Gaussian convergence, but we consider repairing it is less related to the goal of this paper and leave it for future work.

Q3: The size of user study is limited.

Thanks for the advice! We have conducted additional surveys with 20 more participants, bringing the total to 60 users. The updated results are included in the paper. Due to time constraints in the rebuttal period, we plan to release the data from our experiments for public comparison to facilitate broader assessment and validation.

We hope our responses satisfactorily address your queries. Please let us know to address any further concerns impacting your review.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: The writing is poor and misses background details.

Thanks for your advice! The paper has been revised to enhance the writing quality. Additionally, we have included a preliminary section in the appendix to introduce relevant background details including SDS loss and mesh UV mapping.

Q2: What is the distribution of p and t in SDS loss?

$p$ represents the camera pose to render the 3D Gaussians, which is uniformly sampled to orbit the object center.

For example, for image-to-3D, the camera is sampled with a radius of $2$, a y-axis FOV of $46$ degree, with the azimuth in $[-180, 180]$ degree and the elevation in $[-30, 30]$ degree.

$t$ represents the timestep in denoising diffusion probabilistic model.

Early works like Dreamfusion [1] samples $t$ from $[0.02, 0.98]$ uniformly during training. We follow Dreamtime [2] to perform an annealing of $t$ from $0.98$ to $0.02$ during training, which leads to faster convergence as shown in the ablation study (Figure 7).

Q3: How to perform the linear timestep decreasing?

As answered in Q2, we linearly decrease $t$ from $0.98$ to $0.02$ with iteration step $i$ during training.

This can be viewed a simplification of the TP-SDS algorithm proposed in Dreamtime [2]. Their findings suggest that larger values of $t$ are crucial for the global structure, whereas smaller values enhance local details. Hence, a decreasing schedule aligns the SDS noise level with NeRF optimization, facilitating a coarse-to-fine generation process.

Q4: More details about the evaluation protocol and datasets.

We have expanded on our evaluation methodology in section A.1 of the appendix.

Given the absence of ground truth for 3D generative tasks, assessing generation quality poses a challenge. We follow previous works and use CLIP-similarity for evaluation. This metric calculates the average cosine similarity of CLIP embeddings between the input image and novel views from the generated 3D object. Our evaluation was conducted on a dataset comprising 30 samples.

[1] Poole, Ben, et al. "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.14988 (2022).

[2] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).

We hope our responses satisfactorily address your queries. Please let us know to address any further concerns impacting your review.