Re-imagine the Negative Prompt Algorithm for 2D/3D Diffusion

Dear Reviewer 6H1W,

Thank you for your thorough review and positive feedback on our paper. We appreciate your recognition of our novel approach in solving the Janus problem using a novel negative prompt algorithm.

We acknowledge your suggestion regarding the inclusion of quantitative fidelity results like FID, CLIP similarity, and user preference. We agree that these metrics will enhance our paper's robustness and will include them in our revised submission.

Thank you again for your support and positive comments on the value of our work!!

Dear Reviewer j2ct,

Thank you for your thorough review and insightful observations regarding our submission. Your feedback has been invaluable in enhancing the quality and scope of our research.

1-Expanding the Scope Beyond 3D Generation:

We appreciate your emphasis on the broader potential applications of our Perp-Neg methodology. Initially, our focus was predominantly on 3D generation due to its profound impact and the specific challenges presented by the Janus problem in score distillation-based methods. Recognizing the value of your suggestion, we will extend our research to encompass additional applications, such as segmentation mapping and guided bounding box generation, and leave the rest for future work.

2-More 3D Experiment

We thank you for pointing out the need for a wider array of examples. Initially, we presented results from 12 prompts, totaling 240 experiments. Based on your valuable feedback, we expanded our dataset to include 10 additional prompts, each tested with and without Perp-Neg. The diversity and increased number of experiments strengthen our findings, as detailed in the revised Table.

Please refer to the response provided to reviewer vLLW question 2nd.

3- More 2D view Experiment

For view-dependent images, we've now included varied and creative prompts, enhancing the demonstration of our method's effectiveness in 'real-world' scenarios. The updated results are as follows:

4(a)- More 3D Baseline

Thanks for raising this point! Our intention was not to directly compete with existing text-to-3D models but to show how Perp-Neg can enhance them as a plug-in when combined with them. We will clarify this in the paper. The provided examples, highlight the significant reduction of the Janus problem when our method is integrated into these baselines.

4(b)- Figure 7 Clarification

To clarify, Figure 7 in our paper is intended to demonstrate the improved results achieved by integrating Perp-Neg with the Magic3D model. We acknowledge that the absence of standalone results from "Magic3D without Perp-Neg" might have caused a lack of context and confusion.

Figure 7 only shows the result of "Magic3D + Perp-Neg." To make things clearer, we'll add some pictures to the appendix of our paper. These pictures will show what happens when we use "Magic3D without Perp-Neg" method. This will help show why our method is important and how it fixes the problems with Magic3D. For all tries "Magic3D without Perp-Neg" we observed the Janus problem.

We are truly grateful for your comprehensive review and constructive feedback, which have been instrumental in enhancing the quality and clarity of our paper. Thank you so much!

Dear Reviewer Vwgv,

We are truly grateful for your valuable feedback on our manuscript. We have carefully considered your comments and have made several enhancements to our paper to address your concerns.

1-Focus on 2D Images and Relevance to 3D Generation:

Thanks for raising this point! We will clarify in the paper that in our initial submission, we conducted 240 trials of text-to-3D generation, incorporating 12 different prompts. Our emphasis on 2D was primarily due to the significant computational demands of 3D experiments. Each 3D trial required over 30 minutes of processing, compared to just 8 seconds for 2D.

Additionally, we found that improvements in 2D were closely linked to enhancements in 3D generation, offering a more efficient pathway to examine the root cause of the Janus problem. We will add this clarification to the revised manuscript to better explain our methodology and focus.

2-Extended Experiments and Comparative Analysis with Paper [1]:

In response to your suggestion, we expanded our experimental scope. We conducted an additional 180 experiments for each prompt, with variations including the use of Perp-Neg, no Perp-Neg, and prompt debiasing (using DeepFloyd-IF). These experiments have provided more comprehensive insights into the effectiveness of our proposed method, especially in comparison to the work presented in Hong et al. (2023). We will include these additional results and a more thorough comparative analysis in our revised manuscript.

Here are the updated results from these experiments:

These findings reinforce our initial conclusions and demonstrate the practical effectiveness of Perp-Neg in various scenarios.

3-Simplicity of Generated 3D Objects and Detailed Object Generation:

Thank you so much for this comment! Our choice of simple prompts was intentional, as we aimed to isolate and address the view consistency issue without the confounding factor of complex object generation. By confounding factors, we refer to the problem of the text-to-image model missing some objects provided in the prompt. However, following your suggestion, we have now also tested our algorithm with more complex prompts and provided the result above.

4-Choice of CEBM as a Baseline and Exploration of Negative Prompts:

Thanks for this question! We would like to clarify that the Compositional Energy-based Model (CEBM) is exactly the naive negative prompt algorithm. CBEM just mathematically motivates why the negative prompt algorithm works. In the revised version of our paper, we will clarify this distinction more explicitly to ensure a better understanding of why CEBM was referenced

We want to express our sincere gratitude for the time and effort you've dedicated to reviewing our paper. Thank you again for your invaluable contribution to our work.

Dear Reviewer vLLW,

We greatly appreciate your comprehensive review and constructive feedback on our paper. Your insights are invaluable in refining our paper!

1-Regarding Perpendicular Gradient Sampling Theory:

We acknowledge your concern regarding the lack of formal mathematical proof for our perp-neg method. While a formal mathematical proof is not presented, we believe our geometric reasoning and mathematical motivations, detailed in sections 2.2.1 and 2.2.2, provide a solid foundation for our approach. We appreciate your pointing out this limitation and will consider adding a more rigorous theoretical underpinning in future work.

2-Effectiveness of Perp-Neg in alleviating the Janus Problem - Threestudio:

We acknowledge your concern about our method's limited effectiveness in text-to-3D as highlighted in the Threestudio implementation.

It is essential to clarify that the Threestudio version is a derivative of our original algorithm and does not mirror our methodology or results. In our experiments, we observed a significant reduction in the Janus problem, as presented in the appendix.

In our experiments, comprising 240 trials (120 with Perp-Neg and 120 without), we observed a significant mitigation of the Janus problem. The following table, which we plan to include in the revised manuscript, details our findings:

Overall, our method with Perp-Neg demonstrated a 29.17% improvement over the non-Perp-Neg approach (5.83%). These results are presented in the original manuscript with 12 different prompts in total. We have mentioned the success rate for each prompt in the paper.

It's crucial to note that while we haven't entirely solved the Janus problem, our method offers a significant advancement in addressing it. We will clarify this in the paper.

Following your suggestion and other reviewers, we will also include an additional 120 experiment results on text-to-3D experiments in the appendix to further substantiate our claims. The table of the new results is presented in the following:

3-On Tuning the Balance Factor Between Equations 8 and 9:

We appreciate your concern about the balance factor's tuning difficulty. We wanted to clarify that Equation 9 is used across all text-to-3D tasks. Regarding the weight of negative prompts, we found that the model is not sensitive to the weight of negative prompts if they are beyond a certain threshold, as observed in both 2D and 3D settings. This observation will be elaborated in our revised manuscript for better clarity.

Thank you once again for your thorough evaluation and constructive criticism. Your feedback has been instrumental in refining our work, and we are confident that these revisions will significantly improve our paper.

Dear authors,

I appreciate your response. These added experiments have addressed some of my concerns. However, I still have doubts about the effectiveness of your methods in text-to-3D models.

I have tried your released code, but it is still prone to generating Janus-faced objects. The interpolation weight between the negative prompts and positive prompts is hard to decide; it needs to be tuned case by case.

Besides, I also tried your code on 2D-image generation. I found that the generated image may encounter distortions and oversaturation when adopting your proposed method. I speculate it's the perp-neg sampling method in Equation 6 that poses an OOD issue for the latent decoder of diffusion models.

So, I tend to keep my previous rating.