DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation

[Q2] What is decreasing timestep sampling?

We adopted decreasing timestep sampling to satisfy the equation 15 (equation 12 in the revised manuscript) at every timestep $t$. For clarification on decreasing timestep sampling, please refer to common response 3. As we discussed in common response 3, SDS differs from standard diffusion sampling in its approach to iterative timestep sampling because SDS aims to distill the diffusion networks to the 3D model. To approximate the standard diffusion sampling, DreamCatalyst and DreamTime $1$ sample $t$ consecutively (more noise to less noise) with SDS. The main difference between the decreasing timestep sampling and DreamTime lies in the sampling rate for each timestep $t$. By uniform decreasing timestep sampling, DreamCatalyst is more consistent with the standard diffusion process compared to DreamTime by almost satisfying the equation 15 at every $t$, as discussed in common response 3.

[Q3] Why is DreamCatalyst fast?

In common response 1, we clarify why DreamCatalyst achieves faster 3D editing. Please refer to common response 1. In short, our modified loss significantly reduces the required optimization steps, while FreeU further decreases editing time by removing the additional computational overhead associated with extra networks, as seen in methods like DreamBooth and LoRA. Moreover, we carefully revised the manuscript to incorporate your thoughtful suggestions (lines 341-348 and 376-377).

[Q4] Designing $\Phi$ and $\Psi$

We elaborated on why DreamCatalyst can save the editing time in common response 2. Please refer to common response 2.

Reference

$1$ Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).

Dear Reviewer p9uk,

We thank the reviewer for their instructive feedback and thoughtful comments. Below, we provide detailed responses to each of the raised questions and concerns. Please let us know if further clarification is needed at any point.

[W1, Q5] About FreeU

We remark that our loss design is technically practical and efficient. We would like to clarify that in Table 4, $b=1.0$ corresponds to DreamCatalyst without FreeU. Thus, Table 4 shows the quantitative ablation of FreeU. Notably, DreamCatalyst without FreeU already achieves state-of-the-art results compared to PDS and IN2N, as shown in Tables 1 and 4. This signifies our theoretically designed loss enables improved editing in both speed and quality. Specifically, in fast mode, DreamCatalyst operates approximately 5 to 23 times faster than other baselines, making 3D editing techniques more applicable in real-world scenarios by significantly reducing computation time.

Furthermore, DreamCatalyst introduces technical novelty by discovering FreeU’s strength in editing tasks and SDS. To the best of our knowledge, we first discovered that FreeU is suitable for editing tasks without sacrificing identity preservation, not just for generation tasks. Also, instead of utilizing Dreambooth as PDS, we first employ FreeU to remove extra training time and computation time at the inference stage for the module in SDS and editing tasks. Moreover, the proposed formulation and FreeU have strong synergy. As shown in Table 6 and Figure 19, PDS with FreeU shows inferior results than the original PDS. This is because PDS underweights identity preservation at large timesteps, as shown in Figure 2 (a), leading to insufficient preservation of identity features. Enhancing editability with FreeU further exacerbates this issue, resulting in loss of the original identity and unrealistic image results, such as over-editing and background distortions as illustrated in Figure 19. These observations underscore the synergy between FreeU and our weight design rule, highlighting the technical novelty of our approach.

We stress that the performance improvements of our loss and FreeU are almost similar. A comparison between Tables 1 and 4 reveals that the performance gains in key metrics—namely, the CLIP-Directional Similarity Score and Aesthetic Score—resulting from the modified loss weighting and FreeU are similar. This highlights the analogous importance of these two components in enhancing the editing quality within our framework. For your convenience, we provide integrated quantitative comparisons in Tables 1 and 4 as follows:

Note that Bold indicates the best result and Italic is the second-best result.

[Q1] How can DreamCatalyst be adapted to 4D editing?

Extending 3D to 4D has recently garnered significant attention such as Monst3r, which extends Dust3r for 4D reconstruction. The key challenge in extending 3D to 4D lies in temporal understanding. However, standard 2D pretrained diffusion models do not contain any temporal priors and inherently incorporate only 3D priors. To enable 4D editing, DreamCatalyst must leverage pretrained video diffusion models, i.e., CogVideoX $1$ . Such pretrained video diffusion models incorporate both 3D and temporal priors. By incorporating an extra-temporal regularizer into DreamCatalyst, 4D editing becomes feasible. As described in equation 16 (equation 13 in the revised manuscript), DreamCatalyst formulates SDS editing as an optimization problem, allowing for the inclusion of various regularizers, such as temporal regularization terms, to achieve 4D editing. We believe that the suggested theoretical framework in DreamCatalyst can be generalized to a variety of tasks by appropriately adjusting the regularization terms.

Reference

$1$ Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

Dear Reviewer p9uk,

Thank you for reviewing our paper and for providing valuable feedback. We’d like to check if you have any further concerns or comments that we can address. Please let us know if there’s anything else you’d like us to clarify or improve.

Best regards,

Authors

Dear Reviewer p9uk,

The discussion phase has only two days remaining, and we thus kindly request you to let us know if our response has addressed your concerns. If there are additional issues or questions, we would be happy to address them. Otherwise, we would greatly appreciate it if you could consider updating your score to reflect that the issues have been resolved.

Best regards,

Authors

Dear Reviewer p9uk,

We sincerely appreciate your thoughtful suggestions. As your helpful feedback, we have revised the manuscript and highlighted the changes in red as follows:

• [Q3] We have clarified how DreamCatalyst achieves fast editing in the revised manuscript (lines 341-348 and 376-377).
• [Q5] In Table 4, $b=1.0$ corresponds to DreamCatalyst without FreeU, illustrating the quantitative ablation on FreeU. We have updated the notation to explicitly label $b=1.0$ as "DreamCatalyst w/o FreeU" in the revised manuscript.

As today is the last day for submitting revisions (with six days remaining in the discussion period), we wanted to inform you that the revision phase is concluding. If you have any additional suggestions or concerns, please let us know at your earliest convenience. We are eager to discuss and address any further feedback you may have during the remaining discussion period.

Best regards,

Authors

Dear Reviewer p9uk,

We hope this message finds you well. As the discussion period is nearing its conclusion in just two days, we wanted to check if we have sufficiently addressed your concerns and questions. We would greatly appreciate any further discussion to address your concerns. Looking forward to hearing your thoughts!

Best regards,

Authors

Dear Reviewer p9uk,

As there are only 24 hours left in the discussion period, we wanted to check if we have adequately addressed your concerns. We would greatly appreciate it if you could share your thoughts and engage in further discussion regarding our responses. Once again, thank you for your valuable time in reviewing our manuscript.

Best regards,

Authors

Dear Reviewer p9uk,

With only 8 hours remaining in the discussion period, we wanted to check if we have adequately addressed your concerns. Once again, thank you for dedicating your valuable time to reviewing our manuscript.

Best regards,

Authors

[Q1] Significance of FreeU

DreamCatalyst without FreeU already achieves state-of-the-art results. We clarify that $b=1.0$ in Table 4 indicates the configuration without FreeU and we have revised the manuscript to explicitly highlight it. Notably, the quantitative results without FreeU outperform the existing methods, as illustrated in Table 1. For your convenience, we provide integrated quantitative comparisons in Tables 1 and 4 as follows:

Note that Bold indicates the best result and Italic is the second-best result.

These quantitative results provide overall quality comparisons of PDS and DreamCatalyst without FreeU, while the Batman result in Figure 6 represents only a single case. Furthermore, as in Figure 19 in the revised manuscript’s Appendix, qualitative comparisons reveal that DreamCatalyst without FreeU produces more realistic and visually appealing edits compared to PDS with FreeU. Therefore, the quantitative results demonstrate that DreamCatalyst does not heavily rely on FreeU.

[Q2] Designing $\Phi$ and $\Psi$

In common response 2, we have discussed the determination of our formulation. Please refer to common response 2. In short, we determined the $\Phi$ and $\Psi$ with the proposed two conditions and our observation (discussed in common response 2). Furthermore, our design rule demonstrates robustness, as various function families following this rule produce consistent and reliable results.

Dear Reviewer Hf3Z,

We greatly appreciate your insightful feedback and thoughtful comments. Below, we have provided detailed responses to each of your questions and concerns. Please do not hesitate to let us know if there are any points that need further explanation or additional clarification.

[W1] Comparison to PDS

We stress that our DreamCatalyst is a generalized formulation of PDS, positioning PDS as a special case of DreamCatalyst, just as SDS is a special case of VSD $1$ . Our theoretical analysis shows that PDS cannot change the weighting of loss terms and the regularization function itself (L2 loss in PDS). However, our analysis enables not only reweighting the loss terms but also the use of regularization functions. Our theory allows using other identity preservation loss terms. Thus, our theoretical analysis gives broader ways for future work such as extending DreamCatalyst to 4D editing, discussed in reviewer p9uk’s Q1.

In addition, the main difference between DreamCatalyst and previous methods (i.e., ProlificDreamer and Fantasia3D $2$ ) lies in the purpose of utilizing multiple loss terms. (1) Fantasia3D disentangles geometry and appearance by separating them into independent models, each optimized with its own SDS loss. Unlike our approach, Fantasia3D uses the SDS loss independently for optimizing separate geometry and appearance models, without regularizing the SDS loss for a single model. (2) ProlificDreamer employs two loss terms: the SDS loss for finetuning the LoRA model and a separate loss term for optimizing the 3D model. Similar to Fantasia3D, each loss term in ProlificDreamer is dedicated to optimizing a distinct model. In contrast, DreamCatalyst leverages two loss terms exclusively to optimize a single 3D model. Specifically, DreamCatalyst regularizes the SDS loss to enhance optimization for the single model, whereas ProlificDreamer and Fantasia3D employ multiple loss terms to independently optimize separate models, without regularizing for a unified objective. These regularizations can be extended to future work of 3D editing (i.e., varying the identity preservation regularizer to improve editing results) and various tasks (e.g., 4D editing with additional temporal regularizer). Thus, our theoretical analysis has strong expandability for practical usage.

[W2] The results of PDS

First, we fully followed the instructions of the official code of PDS. We observed that 3D editing results of PDS often show differences with their teaser figure as their GitHub issue $3$ . However, we carefully revised the manuscript to provide qualitative comparisons with their teaser figure as your thoughtful suggestions to address your concern. We brought the figures provided in the original PDS paper to compare with their best results. In the revised manuscript, Figure 16 has been updated to incorporate teaser results of Batman and tulip examples from the original PDS paper (please refer to the revised manuscript’s Appendix). Additionally, we have included a comparison of editing a face into a skull using results obtained from the official PDS project page $4$ . While the teaser primarily highlights two scenes, this additional example provides coverage of a different scene to further demonstrate the method’s versatility. As a result, DreamCatalyst demonstrates more realistic editing results while preserving the background details, outperforming the representative results of PDS. We sincerely appreciate your valuable feedback for the better manuscript.

Reference

$1$ Wang, Zhengyi, et al. "Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation." Advances in Neural Information Processing Systems 36 (2024).

$2$ Chen, Rui, et al. "Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

$3$ https://github.com/KAIST-Visual-AI-Group/PDS/issues/7

$4$ https://posterior-distillation-sampling.github.io

Dear Reviewer Hf3Z,

The discussion phase has only two days remaining, and we thus kindly request you to let us know if our response has addressed your concerns. If there are additional issues or questions, we would be happy to address them. Otherwise, we would greatly appreciate it if you could consider updating your score to reflect that the issues have been resolved.

Best regards,

Authors

Dear Reviewer Hf3Z,

We sincerely appreciate your helpful suggestions. In response to your feedback, we have revised the manuscript and highlighted the changes in "red" as follows:

• [W2] We have updated the qualitative comparisons between the teaser figures of PDS and DreamCatalyst in the revised manuscript (please refer to Figure 16).

As today is the final day for revisions (with six days remaining in the discussion period), we wanted to inform you that the revision period is concluding. If you have any further suggestions or concerns, please let us know at your earliest convenience. We are eager to discuss and address any additional feedback you may have during the remaining discussion period.

Best regards,

Authors

Dear Reviewer Hf3Z,

We hope this message finds you well. As the discussion period is nearing its conclusion in just two days, we wanted to check if we have sufficiently addressed your concerns, particularly regarding the comparison between DreamCatalyst and the best results of PDS. We would greatly appreciate any additional discussion to address your concerns. Looking forward to hearing your thoughts!

Best regards,

Authors

Dear Reviewer Hf3Z,

As there are only 24 hours left in the discussion period, we wanted to check if we have adequately addressed your concerns. We would greatly appreciate it if you could share your thoughts and engage in further discussion regarding our responses. Once again, thank you for your valuable time in reviewing our manuscript.

Best regards,

Authors

Dear Reviewer Hf3Z,

With only 8 hours remaining in the discussion period, we wanted to check if we have adequately addressed your concerns. Once again, thank you for dedicating your valuable time to reviewing our manuscript.

Best regards,

Authors

[W3] Significance of FreeU

First, the modified weighting of loss terms and FreeU are analogously important components in enhancing the editing quality of our framework. As demonstrated in Tables 1 and 4, the performance improvements achieved by modifying the loss function and incorporating FreeU are comparable particularly in terms of CLIP-Directional Similarity and Aesthetic Score. Notably, $b=1.0$ in Table 4 indicates DreamCatalyst without FreeU (we revised Table 4 to clarify it), which already archives state-of-the-art results.

For your convenience, we provide integrated quantitative comparisons from Tables 1, 4, and 6 as follows:

Note that Bold indicates the best result and Italic is the second-best result.

To further address your suggestion, we conducted experiments applying FreeU to both PDS and DreamCatalyst (please refer to Table 6 and Figure 19 in the revised manuscript’s Appendix). We set the FreeU hyperparameter $b=1.1$ for PDS, consistent with its configuration in DreamCatalyst. The findings are summarized below:

• Applying FreeU to PDS: We observed a slight increase in the CLIP-Directional Similarity score (from 0.161 to 0.162), demonstrating the enhanced editability with utilizing FreeU. However, Aesthetic Score decreased (from 5.437 to 5.413). This is because PDS underweights identity preservation at large timesteps as in Figure 2 (a), resulting in insufficient preservation of identity features. Enhancing editability with FreeU further exacerbates this issue, causing the method to lose the original identity and generate unrealistic image results, such as over-editing and background distortions as shown in Figure 19.

• Applying FreeU to DreamCatalyst: Integrating FreeU into our method significantly improved both the CLIP-Directional Similarity score (from 0.171 to 0.180) and the Aesthetic Score (from 5.564 to 5.688), demonstrating enhanced editability and visual quality. DreamCatalyst effectively balances editability and identity preservation by combining modified loss weighting and FreeU.

These results indicate that while FreeU can enhance editability metrics, its effectiveness depends on the underlying method's ability to preserve identity features and produce realistic images. Therefore, combining our modified loss weighting with FreeU is essential for achieving superior results in DreamCatalyst.

[Q1] Why is the approximated diffusion reverse process not compatible with PDS?

The early stages of PDS with the approximated diffusion reverse process disrupt preserving the source features. In PDS, identity preservation becomes insufficient at large timesteps due to the prioritization of editability. Consequently, SDS editing at large timesteps loses most of the source features because of strong noise perturbation and insufficient identity preservation as in Figure 2 (a). However, PDS using the approximated diffusion reverse process repeatedly samples large timesteps in the early stages of editing. As a result, the 3D model learns representations that lack source features, leading to a loss of the source identity. As shown in Figure 7, this produces results that resemble generation rather than editing, failing to preserve the source features. Therefore, PDS suffers from utilizing the approximated diffusion reverse process.

[Q2] What is uniform sampling?

We provided an explanation of decreasing timestep sampling and its relationship to uniform sampling in common response 3. Please refer to the common response 3. As discussed in common response 3, each timestep $t$ is sampled multiple times, and we adopted a uniform sampling strategy to ensure equal sampling for every $t$.

Dear Reviewer wbDz,

We sincerely appreciate your valuable feedback and constructive comments. Below, we address each concern and question in detail. Please let us know if there are any remaining issues or concerns that require further clarification. We carefully revised the manuscript to reflect the suggested qualitative and quantitative results in the Appendix.

[W1] Comparison to PDS

We sincerely appreciate your thoughtful suggestions for improving the manuscript. The key distinction between DreamCatalyst and PDS lies in the design of $\Phi(t)$ and $\Psi(t)$. We remark that PDS cannot modify $\Phi(t)$ and $\Psi(t)$ since their theoretical foundations, while our theoretical finding allows the modification because interpreting DDS as an optimization problem makes the identity preservation term a regularizer. This modification not only reduces the editing time largely but also improves the quality. We carefully revised our manuscript to reflect the main differences compared to PDS. Please refer to lines 341-348, which have been highlighted in “red” for your convenience.

[W2] Why DreamCatalyst is fast?

In common response 1, we clarify why DreamCatalyst can edit faster. Please refer to common response 1. We have carefully revised the manuscript accordingly to be clear as lines 341-348 and 376-377 as your thoughtful suggestions.

In addition, we have supplemented the quantitative and qualitative comparison results as suggested in Figures 17 and 18 to highlight DreamCatalyst’s convergence speed (please refer to the revised manuscript’s Appendix). Figure 17 quantitatively demonstrates that DreamCatalyst converges significantly faster than the baseline methods. For this evaluation, we utilized CLIP Directional similarity as a metric to reflect the editing convergence behavior, since CLIP image similarity and Aesthetic score do not adequately capture the editing convergence behavior. Figure 18 presents qualitative results highlighting the editing convergence. These results indicate that DreamCatalyst achieves substantially faster convergence compared to PDS and IN2N. Thanks for your helpful feedback.

Dear Reviewer wbDz,

Once again, thank you for taking the time to review our paper. We appreciate your efforts in helping us improve our work. We hope to inquire if you have any remaining concerns that we could address. Kindly let us know.

Best regards,

Authors

Dear Reviewer wbDz,

The discussion phase has only two days remaining, and we thus kindly request you to let us know if our response has addressed your concerns. If there are additional issues or questions, we would be happy to address them. Otherwise, we would greatly appreciate it if you could consider updating your score to reflect that the issues have been resolved.

Best regards,

Authors

Dear Reviewer wbDz,

We sincerely appreciate your valuable suggestions. In accordance with your feedback, we have revised the manuscript and highlighted the changes in "red" as follows:

• [W1] We clarified the difference between DreamCatalyst and PDS in the revised manuscript (lines 341-348).
• [W2] We clarified the reason for the increased speed in the revised manuscript (lines 341-348 and 376-377). In addition, we have supplemented quantitative and qualitative comparisons in Figures 17 and 18.
• [W3] We have included the ablation results of FreeU on DreamCatalyst and PDS in Table 6 and Figure 19.

As today is the final day for revisions (with six days remaining in the discussion period), we wanted to inform you that there is limited time remaining before the revision period ends. If you have any further suggestions or concerns, please let us know at your earliest convenience. We are eager to discuss and address any additional feedback you may have.

Best regards,

Authors

We are glad that our rebuttal addressed your concerns, and we sincerely appreciate your decision to raise the score.

Please don’t hesitate to let us know if you have any additional questions or feedback.

Dear Reviewer wbDz,

As there are only 24 hours left in the review period, we kindly ask if you could update your review score in the OpenReview system to reflect your updated comments. We would greatly appreciate it if you could do so.

Thank you very much for your time and consideration.

Sincerely,

Authors

We would like to thank reviewer wbDz for the positive and constructive comments and raising the score throughout the review process. We are glad that we addressed the concerns.