Text-to-3D with Classifier Score Distillation

[W3] Quantitative comparisons with ProlificDreamer and Fantasia3D

1. We would like to clarify that the reason that we omitted quantitative comparisons with ProlificDreamer and Fantasia 3D is that it is very time-consuming to evaluate ProlificDreamer based on public implementations (8h per prompt on an A800 GPU), and Fantasia3D requires manual tuning configuration parameters for each prompt (see their note ("What do you want?") in their official implementation). It is also worth noting that in their original paper, ProlificDreamer evaluated the performance on 15 prompts with a user study, and Fantasia3D evaluated the performance on 10 prompts with a user study.

2. Here, within the short period of rebuttal time and our constrained computational resources, we randomly select 20 prompts and conduct experiments and comparisons with CSD, ProlificDreamer (using ThreeStudio), and Fantasia3D (using official code). The table below presents the results, showing that our method is generally comparable to Fantasia3D and ProlificDreamer. Moreover, our approach tends to perform better in comparison with other approaches under a more strict evaluation criterion like the CLIP score that directly measures the exact match between text prompts and generated images. However, it's important to note that metrics like the CLIP score and R-precision primarily measure textual-image congruence and may not reflect 3D and texture quality. For a more comprehensive analysis, please refer to the detailed discussion below.

   	CSD	Fantasia3D	ProlificDreamer
   CLIP Score	81.6	77.8	80.2
   CLIP R@1	96.2	93.8	96.2
   CLIP R@5	98.8	100	100
   CLIP R@10	100	100	100

3. [Analysis of CLIP score for 3D evaluation] It's important to note that in evaluating 3D synthesized results, the CLIP score may not fully reflect the quality for several reasons: (1) As an image-level metric, the CLIP score assesses each projected image independently, thus lacking comprehensiveness for 3D results; (2) The CLIP score might not align with human perceptual judgments, which are crucial in evaluating generative results; (3) This score evaluates the match between synthesis and text prompts using its learned feature space, which might not correlate directly with quality; and (4) The CLIP score has biases toward its training data distribution, potentially differing from 3D projected images. Consequently, many methods, like [1,2,3], choose to only conduct user studies for performance evaluation. In the final version, we will include user study results for these new experiments (note that we have previously conducted a user study comparing our results with DreamFusion and Magic3D, as detailed in our paper Table 1). Please see our qualitative results at the provided anonymous link.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

[2] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. Wang et al., 2023

[3] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. Chen et al. 2023

Dear Reviewer FT6d,

We sincerely appreciate your great interest in our work and your thoughtful comments. In the response below, we address each of your concerns with the aim of clarifying and resolving any issues raised.

---

[W1] Implementation

Due to the change in the main loss function (SDS vs CSD), which obviates the need for a 'guidance scale', adjusting other parameters alongside the optimization process is necessary. We sincerely apologize for the lack of detailed hyperparameters. To address this, we have now revised and expanded the appendix to include these crucial details (i.e. appendix A.4). Additionally, we have made the code available at this anonymous link, with the repository's README containing execution instructions. We will also release our code publicly.

---

[W2] Janus problem

1. We would like to clarify that the core concepts and theoretical insights of our paper are not directly associated with the Janus problem. The fundamental challenge of the Janus problem stems not from the loss function, but from the limited 3D-aware capabilities inherent in the diffusion model. Our approach can be seamlessly integrated with techniques specifically designed to address the Janus problem, such as using the multi-view-diffusion model.
2. Here, we conduct a set of experiments using the pre-trained multi-view diffusion model from the concurrent work MVDream [1], combined with our CSD loss. We provide the qualitative results at this anonymous link. The results illustrate that our loss function can generate realistic results, free from the Janus problem, indicating that this issue is more attributable to the diffusion model's 3D-awareness rather than our loss. This set of experiments further strengthens the versatility and generalizability of our analysis. We also provide the code at the same link in W1 for your reference and we hope it can help the clarification.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

[Q1, Q2] Negative prompts

1. We use "oversaturated color, ugly, tiling, low quality, noisy" as our negative prompts. If negative prompts are not used, some generated results may show oversaturated colors or a higher occurrence of floaters. For demonstration, videos are provided at the following anonymous link.
2. It should be emphasized that our paper's focus lies in providing a new understanding of score distillation and establishing the link between classifier scores and negative prompts rather than contributing to the technique of negative prompts itself. With this new understanding and guidance, we envision that future research could explore a range of techniques to improve quality, not limited to negative prompts. This includes developing adaptive negative classifier scores, for example, we examine the connection between VSD and classifier scores in equation (12).

---

[Q3] Using all 415 prompts for calculating CLIP R-precision

Thank you for your suggestion. For calculating the R-precision, we utilized the same set of prompts as those used in the generation process, in accordance with Cap3D [4]. These 81 prompts are detailed in our supplementary materials, with corresponding video filenames indicating the specific prompts employed. Note that our comparison between CSD and other methods is fair, since they are all based on same prompts.

To further address your concern, we additionally calculated the R-precision using a larger set of 415 prompts for the caption set. As shown in the table below, the results are consistent with our original paper.

Reference:

[4] Scalable 3D Captioning with Pretrained Models. Luo et al., 2023

---

We sincerely hope that our responses have adequately addressed your concerns and thank you again for your valuable suggestions that help us improve our work.

I would like to thank the authors for their response. After running the provided code and viewing other supplemented videos from other answers, I have the following observations:

1. Regarding the 3d model quality if only optimized with 1 stage (NeRF only), the quality of CSD and SDS are similar. This is a fair comparison since the proposed method focused on the proposed loss, not on three-stage optimization. Similarly, we can take the results of SDS and then optimize further in 2 later stages (with exactly the same setting). Compared to VSD, the quality of CSD is fairly behind (also with 1-stage optimization).
2. Regarding the optimization time, it has a similar optimization time as SDS and is faster than VSD.

Therefore, I can observably draw a conclusion that the proposed CSD is more or less similar to SDS, not a big improvement. Thus, I retain my initial rating.

[W3-2] Computational cost for each run:

1). For DreamFusion, we used their official website's results, employing 4 TPUv4 chips for 15,000 iterations over 1.5 hours (6 TPUv4 hours). 2). For Magic3D, we used a single A800 GPU, with a batch size of 4 for 10,000 NeRF iterations (70 minutes) and a batch size of 1 for 10,000 iterations in the second stage (20 minutes), totaling 1.5 GPU hours. 3). For Fantasia3D, following their official code, we used 8 GPUs, each stage taking about 10 minutes, for a total of 2.7 GPU hours. 4). For ProlificDreamer, we use their original paper's settings: 25,000 VSD iterations for NeRF (3.5 hours), 15,000 SDS iterations for geometry (30 minutes), and 30,000 VSD iterations for texture (4 hours), totaling 8 GPU hours. 5). For our experiments, both stages were optimized for 10,000 iterations, taking 35 and 25 minutes respectively, amounting to 1 GPU hour.

---

[W3-3] Comparisons with Tango & quantitative evaluations:

Thank you for your reminder. We have cited this paper in our revised paper (appendix A.1) and here we present the comparison results with it. We show the qualitative comparison results at this anonymous link.

Regarding the task of texture synthesis, as mentioned in W3-1, evaluations based on CLIP may not be accurate. Most current works rely on user studies. Here, we provide the CLIP score for your reference. Although Fantasia3D performs the best on two backbones, there is not a single method that wins in all metrics, and our method performs reasonably well on all evaluated settings. More importantly, our user study in Table 1 in our main paper demonstrates a clear win of our approach in terms of texture synthesis when compared to Fantasia 3D, ProlificDreamer, Magic3D, and TEXTure.

---

[W4] CSD on more baselines:

1. Please note that for DreamFusion, ProlificDreamer, and our work, the principal differences and unique contributions are centered on the three distinct losses: SDS, VSD, and CSD. Therefore, integrating CSD into these methods essentially amounts to employing the CSD approach, albeit with some variations in the training process. For more details, kindly refer to equations 6, 8, and 12 in our paper. In fact, our loss is model-agnostic or independent of 3D representation, as evidenced by our approach in different stages. For instance, in the first stage, we utilize the DeepFloyd-IF stage-I model with NeRF representation, and in the second stage, we apply Stable Diffusion with DMTet representation. This showcases the versatility of our loss across various models and representations.
2. To show more results in more baselines, we integrated the proposed CSD into the pre-trained multi-view diffusion model from the recent concurrent work MVDream [1]. We provide the qualitative results at this anonymous link. The results indicate that our loss can be seamlessly integrated into this new framework, producing realistic contents and high-quality textures.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

---

We sincerely hope that our answers have addressed your concerns and thank you again for your valuable suggestions that help us improve our work.

[W3-1] Comparisons with ProlificDreamer and Fantasia3D:

1. We would like to clarify that the reason that we omitted quantitative comparisons with ProlificDreamer and Fantasia 3D is that it is very time-consuming to evaluate ProlificDreamer based on public implementations (8h per prompt on an A800 GPU), and Fantasia3D requires manual tuning configuration parameters for each prompt (see their note ("What do you want?") in their official implementation). It is also worth noting that in their original paper, ProlificDreamer evaluated the performance on 15 prompts with a user study, and Fantasia3D evaluated the performance on 10 prompts with a user study.

2. Here, within the short period of rebuttal time and our constrained computational resources, we randomly select 20 prompts and conduct experiments and comparisons with CSD, ProlificDreamer (using ThreeStudio), and Fantasia3D (using official code). The table below presents the results, showing that our method is generally comparable to Fantasia3D and ProlificDreamer. Moreover, our approach tends to perform better in comparison with other approaches under a more strict evaluation criterion like the CLIP score that directly measures the exact match between text prompts and generated images. However, it's important to note that metrics like the CLIP score and R-precision primarily measure textual-image congruence and may not reflect 3D and texture quality. For a more comprehensive analysis, please refer to the detailed discussion below.

   	CSD	Fantasia3D	ProlificDreamer
   CLIP Score (ViT-L/14)	71.8	65.2	72.7
   CLIP R@1 (ViT-L/14)	100	93.8	100
   CLIP R@5 (ViT-L/14)	100	100	100
   CLIP R@10 (ViT-L/14)	100	100	100
   CLIP Score (ViT-B/32)	81.6	77.8	80.2
   CLIP R@1 (ViT-B/32)	96.2	93.8	96.2
   CLIP R@5 (ViT-B/32)	98.8	100	100
   CLIP R@10 (ViT-B/32)	100	100	100
   CLIP Score (ViT-B/16)	83.1	78.6	83.2
   CLIP R@1 (ViT-B/16)	97.5	96.2	100
   CLIP R@5 (ViT-B/16)	100	100	100
   CLIP R@10 (ViT-B/16)	100	100	100

3. [Analysis of CLIP score for 3D evaluation] It's important to note that in evaluating 3D synthesized results, the CLIP score may not fully reflect the quality for several reasons: (1) As an image-level metric, the CLIP score assesses each projected image independently, thus lacking comprehensiveness for 3D results; (2) The CLIP score might not align with human perceptual judgments, which are crucial in evaluating generative results; (3) This score evaluates the match between synthesis and text prompts using its learned feature space, which might not correlate directly with quality; and (4) The CLIP score has biases toward its training data distribution, potentially differing from 3D projected images. Consequently, many methods, like [1,2,3], choose to only conduct user studies for performance evaluation. In the final version, we will include user study results for these new experiments (note that we have previously conducted a user study comparing our results with DreamFusion and Magic3D, as detailed in our paper Table 1). Please see our qualitative results at the provided anonymous link.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

[2] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. Wang et al., 2023

[3] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. Chen et al. 2023

---

Dear Reviewer pueN,

We sincerely appreciate your insightful comments and valuable suggestions. In the response below, we address each of your concerns with the aim of clarifying and resolving any issues raised.

---

[W1] Reason for DeepFloyd-IF stage-I & Fairness of comparison

1. The Stable Diffusion model is a high-resolution generative model (512x512), while the DeepFloyd-IF stage-I is a low-resolution model (64x64). Since our first stage focuses on generating rough results, we use DeepFloyd-IF stage-I.
2. We did our best to ensure fair comparisons with other methods by faithfully re-implementing their approaches based on their design principles or directly using their best available results as follows: 1) DreamFusion uses the superior Imagen model, which is not open-sourced. Fortunately, it publishes all results and thus, we directly compared our results with the best results from its official website. 2) The original Magic3D paper used eDiff-I in its first stage, which is also not open-sourced. Besides, Magic3D has not released its code or more results, so we re-implement its method upon the ThreeStudio. As the first stage of Magic3D also focuses on generating a coarse shape, we used DeepFloyd-IF stage-I in its first stage, the same as our setting. 3) ProlificDreamer needs high-resolution NeRF training in its first stage, which requires high-resolution diffusion models. Thus, we followed their setting by using Stable Diffusion in the first stage. 4) Fantasia3D disentangles the generation into geometry and texture, in which both stages need high-resolution diffusion models to guarantee high-quality results. Thus, we followed their original paper by using Stable Diffusion.

---

[W2-1] Criteria of text prompts

1. Thank you for this great question. Our criteria for selecting prompts followed three principles:
   • Sufficient Coverage and Diversity: For example, some categories in the prompt set include: 1). Animal Characters: e.g., "a koala wearing a party hat"; 2). Daily Life Objects: e.g., "a shiny red stand mixer"; 3). Transportation and Structures: e.g., "a Panther De Ville car"; 4). Fantastical and Whimsical Themes: e.g., "a humanoid banana doing homework"; and 5). Nature and Scenery: e.g., "a bumblebee sitting on a pink flower".
   • Avoidance of Overly Similar Prompts: There are many very similar prompts on the official DreamFusion website. We filtered these out to avoid such situations and achieve greater diversity.
   • Inclusion of Uniquely Creative Prompts: We retained prompts that are highly imaginative and creative, such as "a humanoid banana doing homework".
2. It's noteworthy that there is no established gold standard for evaluation prompts, and existing methods often select a subset of prompts for easier evaluation. For example, ProlificDreamer assessed their method using 15 prompts in a user study, while Fantasia3D did so with 10 prompts. Therefore, in our original paper, we adhered to this practice, choosing 81 prompts to guarantee coverage, diversity, and creativity, and to avoid redundancy in our representative evaluation prompts.
3. Given the constraints in computational resources, the brief rebuttal period, and the demands of conducting other experiments, generating results for 415 prompts exceeds our current capacity within the rebuttal timeframe. However, we have included results from additional randomly selected prompts, as illustrated in the comparison in W3-1. We pledge to release our results for all prompts once the experiments are completed.

---

[W2-2] More CLIP R-precision

We report more CLIP scores and CLIP R-precision here using different pre-trained models. Please refer to W3-1 for an analysis of the CLIP score.

---

Dear Reviewer 8VxR,

We sincerely appreciate your insightful comments and valuable suggestions. In the response below, we address each of your concerns with the aim of clarifying and resolving any issues raised.

---

[W1, Q1] Realistic

Thank you for your insightful comments. In our observations, results from CSD are not less realistic or more prone to over-saturation compared to the SDS. Indeed, negative prompts can generally be utilized to address these issues. Further, following the concept of negative classifier scores, we have presented a new insight in Equation (12) of our paper. We suggest that the reason VSD helps alleviate over-saturation might be due to its new gradient functioning like an adaptive negative score.

---

[W1, Q2] Diversity

1. Empirically, we did not observe lower diversity using CSD compared to SDS. We think that improving diversity through reducing CFG might not be a viable choice for the SDS-based method as a lower CFG value may also lead to optimization failure, which is verified by our analysis in Figure 2 of our paper. In this work, our focus is to uncover the component— the classifier-free guidance— that makes SDS work.

2. To improve diversity, we believe a better approach might be to consider more diverse prompts. We can enhance the user-provided prompts, ensuring they meet the user’s content requirements while introducing a variety of variations. This can be achieved automatically using large language models (LLMs). As an illustration, here we employed ChatGPT to enhance the given prompt 'a DSLR photo of a bear dressed in medieval armor', where we asked it to generate different variations while retaining the original meaning, resulting in the following prompts:

   • “A DSLR photo of a panda bear in stylized Eastern medieval armor, reflecting Asian historical designs.”
   • “A DSLR photo of a polar bear in ornate, silver-colored medieval armor, contrasting with its white fur.”
   • “A DSLR photo of a sun bear wearing lightweight leather armor, reminiscent of medieval archers, with a quiver of arrows.”
   • “A DSLR photo of a panda bear in ancient Chinese warrior armor, embellished with dragon motifs and vibrant colors.“

   We then applied our method (NeRF stage) to these prompts, and we provided video results at the following anonymous link. This approach successfully enhances diversity while not suffering from optimization issues.

[W2, Q3] CFG in baseline methods & Experiment with larger CFG for SDS

1. For the CFG scale of other methods, we followed the values stated in their respective papers. For ProlificDreamer, we set the CFG scale to 7.5 in both the first and third stages, where VSD is used, and increased it to 100 in the second stage, where SDS is employed for refining the geometry. For Fantasia3D, we consistently applied a CFG scale of 100 for both stages. In the case of Magic3D, we used a CFG scale of 100 for the second stage. However, for the first stage of Magic3D, since we do not have access to their unreleased eDiff-I model, we utilized the DeepFloyd-IF stage-I model with a CFG scale of 20, following the default implementation in ThreeStudio.

2. Although SDS with a large CFG value approaching infinity may approximate the effects of using classifier-free guidance alone (i.e., CSD), this raises significant optimization issues, making it impractical. Within the SDS framework, which combines generative and classifier priors, hyperparameter tuning with a large CFG scale becomes complex and often requires extensive trial and error to adjust learning rates and geometric constraint loss weights. For instance, an excessively large CFG scale can result in problems such as numerical overflow in gradient calculations due to substantial guidance weights. Moreover, this can cause a "gradient domination" effect, diminishing the impact of other geometric regularization terms.

3. To empirically demonstrate the effect of using higher CFG values with SDS loss, we have conducted experiments using the first stage of Magic3D. Note that the first stage of Magic3D is essentially the same as DreamFusion, with the difference being that DreamFusion employs Mip-NeRF representation and the more powerful Imagen model that we do not have access to. We use SDS with CFG scales of 20, 100, 200, 500, and 1000. However, simply increasing guidance weights does not guarantee satisfactory results, as confirmed in our results (refer to results in anonymous link): higher CFG values can lead to more floaters, and at extremely high CFG values, there's a risk of results collapsing.

---

[W3, W4] Revise of Figure 2

Thanks for your suggestion, we have revised the Figure 2 for better demonstration.

---

We sincerely hope that our answers have addressed your concerns and thank you again for your valuable suggestions that help us improve our work.

Dear Reviewer oCeh,

We sincerely appreciate your insightful comments and valuable suggestions. In the response below, we address each of your concerns with the aim of clarifying and resolving any issues raised.

---

[S2, Q1] Relation to Collaborative Score Distillation

Thank you for your insightful comment and the recommended work, which we have now cited in our revised paper. We find it to be a very interesting work! Interestingly, its formulation also replaces subtracting noise with subtracting a score computed from a reference image. This bears a resemblance to our discussion on DDS. We believe that substituting this aspect with CSD could potentially further enhance the quality.

Since the code for this paper has not been released yet, it is a bit challenging for us to faithfully reproduce this work during this short rebuttal period. We will continue to pay close attention to this work and its release. Thank you for your insightful comments!

To demonstrate the generality and adaptivity of our CSD formulation, we conducted additional experiments on the recently released multi-view diffusion model [1]. We provide the qualitative results at this anonymous link. The results demonstrate that our loss can be seamlessly integrated into this new framework, producing realistic 3D content.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

---

[W1] Hyper-parameter ablation study

Thanks for your suggestion. In our experiment, we found that employing appropriate sparsity and orientation losses during the NeRF optimization is crucial for geometry. Here, we show four distinct experiments to demonstrate this: A). Without any geometric constraint loss. B). Constrained only by sparsity loss. C). Constrained only by orientation loss. D). (i.e., CSD) Employing both constraints simultaneously.

We have provided video comparison results at this anonymous link. As can be observed, it's only when both constraints are applied together that we can ensure smoother normals and better 3D consistency (i.e., less Janus problem), as well as guarantee the overall success of the optimization process.

---

We sincerely hope that our answers have addressed your concerns and thank you again for your valuable suggestions that help us improve our work.