Rethinking Score Distilling Sampling for 3D Editing and Generation

Thank you for your detailed comments and valuable suggestions, which have greatly improved the quality of our paper. Below, we address your concerns and clarify potential misunderstandings:

Experimental

1. As shown in the Figure 1, we provide a quantitative comparison of our NeRF-based results, namely UDS+DreamFusion, with RichDreamer and Dreamcraft3D.

2. Thank you for your suggestion, we will include the following content in the revised version.

We conducted a user study to evaluate the performance of different methods based on human preferences. In the generation task, we showed participants a side-by-side comparison of 3D assets generated by each method. In each trial, participants received a text prompt and a rotated video of multiple candidate 3D assets generated using different methods. In the editing task, we showed the side-by-side effect of each method editing a 3D scene. In each trial, participants received a text prompt, a reference scene, and videos of multiple candidate 3D scenes edited using different methods. Figure 4 is a screenshot of the user study page. We collected responses from a total of 102 participants. Each participant randomly performed 50 to 100 trials, and their selection data were recorded for subsequent analysis. To ensure the diversity and fairness of the evaluation results, the order of presentation of candidate content was randomly arranged in 50 to 100 trials for each participant.

3.1 In our work, all methods use SD 2.1 for the generation task and SD 1.5 for the editing task.

3.2 We follow LucidDreamer to use Pointe to initialize 3D GS.

3.3 We have conducted relevant ablation experiments on these two estimation methods (Tweedie formula and DDIM multi-step denoising) in the main text.

3.4 Regarding the prompts, as described in the main text, for NeRF-based methods, we use 15 prompts from Magic3D and 415 prompts from DreamFusion (these prompts can be easily found on their official website or threestudio). For 3D GS-based methods, we will add a prompt list in the revised version. We have already shown about 30 prompts in the main text.

4. Due to ICML policy, we can't seem to provide videos. As shown in Figure 5, we provide more comparisons with multiple views of LucidDreamer. For fair comparison, we use the same hyperparameters.

References

We will add citations to the revised version, e.g. DPS [1].

[1] Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR (2023)

Weaknesses

1. In generation tasks, success depends on the initialization; if the initialization is poor, the generation is likely to fail. In editing tasks, when there is a large gap between the prompt and the original scene, failure may occur. For example, when changing the prompt from "A photo of a plant" to "a photo of balls", the result often does not generate balls on the branches, but rather places them at random positions in the scene. We will include the above content in the revised version.

2. Please refer to Experimental 4.

3. Our work focuses on exploring a natural and unified formulation of generation and editing tasks rather than on solving the consistency issue. We believe that the inconsistencies in the generated hands can be easily addressed by incorporating multi-view information and geometric constraints. However, this is not the primary focus of our work.

Other Comments

Please note that Equation 12 actually contains some important coefficients of PDS, which we simplify in the main text for easier understanding, while Equation 14 is our proposed method.

Thank you for your detailed comments and valuable suggestions, which have greatly improved the quality of our paper. Below, we address your concerns and clarify potential misunderstandings:

Weaknesses

1. We present a comparison of the optimization time of our method with that of PDS and DDS in the editing task. Please note that when using the same training steps, our method performs similarly to PDS and DDS; however, we found that the UDS method converges faster, requiring only one-third of the steps needed by PDS and DDS.

2. We compare several feed-forward methods: the early ShapE, 3DTopia, and the latest TRELLIS. We use the 43 prompts mentioned in our paper for generation. Because it is difficult to find a metric that works well for these two kind of methods, we render only the front view and calculate the CLIP score for evaluation.

[1] Shap-E: Generating Conditional 3D Implicit Functions

[2] 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors

[3] Structured 3D Latents for Scalable and Versatile 3D Generation

3. Thank you for your suggestion, we will include the following content in the revised version.

We conducted a user study to evaluate the performance of different methods based on human preferences. In the generation task, we showed participants a side-by-side comparison of 3D assets generated by each method. In each trial, participants received a text prompt and a rotated video of multiple candidate 3D assets generated using different methods. In the editing task, we showed the side-by-side effect of each method editing a 3D scene. In each trial, participants received a text prompt, a reference scene, and videos of multiple candidate 3D scenes edited using different methods. Figure 4 is a screenshot of the user study page. We collected responses from a total of 102 participants. Each participant randomly performed 50 to 100 trials, and their selection data were recorded for subsequent analysis. To ensure the diversity and fairness of the evaluation results, the order of presentation of candidate content was randomly arranged in 50 to 100 trials for each participant.

Questions

1. In generation tasks, success depends on the initialization; if the initialization is poor, the generation is likely to fail. In editing tasks, when there is a large gap between the prompt and the original scene, failure may occur. For example, when changing the prompt from "A photo of a plant" to "a photo of balls", the result often does not generate balls on the branches, but rather places them at random positions in the scene. We will include the above content in the revised version.

2. We thank you for raising this question. In fact, SDS-based methods all rely on pre-trained 2D diffusion models, but our UDS makes improvements in gradient construction. Specifically, we use latent embedding $x_0$ to replace $\epsilon$ of previous SDS-based methods. Compared with $\epsilon$, $x_0$ is closer to the data and has less noise, thereby weakening the propagation of 2D model deviations in 3D results. In other words, the unified gradient signal we obtain is more stable, which has been reflected in the experiments that the generated and edited results are richer in details and less distorted.

3. We provide ablation study on the time interval $c$ here (Figure 2. Ablation on different time interval). We provide ablation study on CFG $w$ here (Figure 3. Ablation on different CFG).

Thank you for your detailed comments and valuable suggestions, which have greatly improved the quality of our paper. Below, we address your concerns and clarify potential misunderstandings:

Methods And Evaluation Criteria

Most previous works have focused on a single 3D generation or editing task. Despite the natural and widely recognized formal similarities between editing and generation, no work has explored the two jointly. Our work attempts to investigate both tasks simultaneously and successfully uses the same framework to complete different tasks, which provides a theoretical and experimental basis for subsequent research based on SDS methods. In the generation task, most previous methods use $\epsilon$-guided reconstruction terms for optimization, while our approach uses latent embeddings $x_0$ that are closer to the real data. Since $x_0$ has less noise, this makes the generation process more stable and the generation quality is improved. For the editing task, we prove through derivation that compared to PDS, we omit the implicit identity constraint term in the gradient. This enables editing to be performed in the regular CFG setting and accelerates convergence, thereby improving the editing effect.

Theoretical

The term on the right corresponds to the gradient expression used in DDS, specifically, $\boldsymbol{\epsilon} _ \theta (\boldsymbol{x} _ t^{\text{tgt}}, y^{\text{tgt}}, t) - \boldsymbol{\epsilon} _ \theta (\boldsymbol{x} _ t^{\text{src}}, y^{\text{src}}, t)$. The estimated noise of $\boldsymbol{x}_t^{\text{src}}$serves as a reference for preserving the identity of the source. Although DDS achieves strong editing results in 2D image applications, it suffers from quality degradation when applied to 3D editing. This degradation occurs because 3D editing requires stronger identity preservation than 2D editing. In DDS, the optimization process minimizes the difference between the SDS losses of the source and target, but it does not introduce any additional constraint to explicitly preserve identity. As a result, the lack of regularization allows the optimization to drift away from the source identity. In other words, the gradient term of DDS is not independent of the identity preserving term, but lacks an explicit identity preserving constraint.

Experimental

We provide additional quantitative ablation experiments. Due to rebuttal period constraints, for the 3D GS-based generation tasks, we conducted ablation on only 20 out of the 43 prompts available in 3D GS. For the NeRF-based generation tasks, ablation experiments were performed on 15 prompts from Magic3D. For the NeRF editing tasks, we conducted ablation on 10 different prompts in the ‘man’ scene.

Table1. Ablation for different approximate methods. (NeRF-based editing)

Table2. Ablation for SDS and UDS with different generation frameworks. (NeRF-based generation)

Table3. Ablation for generation task. Add noise by DDIM inverse strategy. (3D GS-based generation)

References

Thank you very much for recognizing our work. We will discuss NFSD, ODS, and ISD in the revised version.

Thank you for your detailed comments and valuable suggestions, which have greatly improved the quality of our paper. Below, we address your concerns and clarify potential misunderstandings:

References

We will include the Gaussianeditor and Drag Your Gaussian discussions in the revised version.

Weaknesses

Thank you for your careful review of our symbols. We will correct these errors in the revised version as follows:

• $\epsilon_\theta$ -> $\epsilon_\phi$.
• $\underbrace{w(\boldsymbol{\epsilon} _ \theta (\boldsymbol{x}_t,t,y) - \boldsymbol{\epsilon} _ \theta (\boldsymbol{x}_t,t,\emptyset))} _ {\delta _ {\boldsymbol{x}_t}^\text{cls}}$ -> $w\underbrace{(\boldsymbol{\epsilon} _ \theta (\boldsymbol{x}_t,t,y) - \boldsymbol{\epsilon} _ \theta (\boldsymbol{x}_t,t,\emptyset))} _ {\delta _ {\boldsymbol{x}_t}^\text{cls}}$
• $\delta_{\boldsymbol{x}t}^\text{recon}$ -> $\delta_{\boldsymbol{x}_t}^\text{recon}$

Questions

1. We provide the training times for the generation and editing tasks using the Tweedie formula and multi-step DDIM Inversion. Please note that we found that in the generation task, the DDIM interval does not significantly affect the quality of the results, and the differences in the 3D assets produced by DDIM and the Tweedie formula are minimal. Therefore, to speed up the generation process, we directly use the Tweedie formula for the generation task. For the editing task, we found that DDIM affects the editing outcome; a shorter DDIM interval retains more of the original information, but it also increases the editing time. (We also conducted corresponding ablation experiments in Figure 4 in the main text.)

Table 1. Training time of DDIM (different time intervals) and Tweedie in the generation task. Note that all experiments are performed on a 3090.

Table 2. Training time of DDIM (different time intervals) and Tweedie in the edit task. Note that all experiments are performed on a A6000.

2. The time interval $c$ is inspired by recent approaches that use DDIM-based noise addition, such as ISM. In ISM, the gradient is guided by an $\epsilon$ that is highly correlated with the data at two time steps; in contrast, our UDS guides the gradient using a latent embedding $x_0$ that is closer to the data at two time steps. We provide an ablation study on the time interval $c$ here (Figure 2. Ablation on different time interval).