LCGen: Mining in Low-Certainty Generation for View-consistent Text-to-3D

Thank you very much for your effort in reviewing our work! Our responses are as follows:

1. Diversified Examples

In Rebuttal File, we demonstrate how our method alleviates the Janus Problem in various other types of examples.

In Fig. C and D, we tested on "a sunflower" and "a piano" to show the effectiveness of our method when the object does not contain a "head". It can be observed that the original Prolificdreamer generates sunflowers and keyboards of piano on both the front and back, resulting in spatial inconsistency. After applying our method, it correctly renders the front and back images of the sunflower and piano, resolving the spatial inconsistency issue.

In Fig. F, we carefully designed a prompt to show the effect of our method when a large amount of descriptive language is added: "A sleek, silver-gray dolphin leaping gracefully out of the crystal-clear ocean, its body glistening in the sunlight as it arcs through the air with joyful exuberance." It can be observed that the original Prolificdreamer generates two dolphin heads, while our LCGen method correctly generates the image. In Fig. G, we also carefully designed a prompt: "A Matte painting of a Christmas cake made of cheese surrounded by a moat made of ice cream". It can be observed that in the original prolificdreamer, the front and back of Santa Claus on the cake both have faces. However, using our method, the correct samples are generated.

2. Limitation Discussion

For the failure cases, our analysis is as follows:

1. When objects are in very unusual poses, our method has a certain probability of failing. For example, "an upside-down lion." In this case, using the original certainty to separate different views fails because the bottom might be the highest certainty view. For such samples, we need to redesign the view-based guidance function to accommodate the generation of these special samples.

2. Our method may fail in multi-object generation scenarios. This is due to the limitations of current text-to-3D methods in multi-object generation tasks. As shown in Fig. H in Rebuttal File, when the text is "three corgis facing different directions," baseline method Prolificdreamer fails to generate three corgis correctly, with an overlap between different objects. On this basis, the current best view-consistent generation methods, such as DreamControl and our method, cannot directly address the Janus Problem. To solve this issue, the primary task is to solve the multi-object generation problem in text-to-3D, which is a significant research area. For example, a current mainstream idea is to generate individual assets for each object and then stitch them together[1]. Our method still has the potential to adapt to such methods.

3. Due to the lack of 3D prior knowledge, like other SDS-based methods, our method can only model 3D representations that appear more realistic from various perspectives, but cannot ensure that these 3D representations adhere to the physical laws of the real world. For example, during the generation of octopus tentacles, since there is no difference in tentacles from different views and the model does not know how many tentacles should be generated, it may produce 3D representations that do not conform to objective reality. To address this issue, we need to endow the model with the ability to understand the 3D world. One possible approach is to collect massive 3D data and design appropriate representation forms to establish a pre-trained 3D generation model (e.g., 3D diffusion). Given the enormous data requirements and training difficulty, this requires the collective effort of the entire AIGC community.

Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.

Reference

[1] Zhou, Xiaoyu, et al. "Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting." arXiv preprint arXiv:2402.07207 (2024).

Thank you very much for your efforts in reviewing our paper! Below is our response. For Figure and Table, please see Rebuttal File.

1. Diverse Examples

The main purpose of our method is to help address Janus Problem appearing on a single object in text-to-3D baselines. In the limitations section of the Main Paper (lines 291-294), we mention, "Our method performs well in generating individual objects but has limitations with complex multi-object scenes." We will provide more detailed explanations in the limitations section and include the following examples:

1. Examples without concept of heads. In Rebuttal File Fig.A and B, we show the results of 'a street lamp' and 'a tree' as you mentioned. It can be seen that both the original Prolificdreamer and our method achieve spatial consistency generation. Considering that the above two examples do not have obvious differences between different views, we conducted experiments on 'a sunflower' and 'a piano' as shown in Fig.C and D in Rebuttal File. It can be seen that the original Prolificdreamer produced spatial inconsistencies, showing multiple frontal images of the flower and multiple keyboards in single piano from different views. Our method successfully alleviated this issue, generating a sunflower and piano with correct front and back images. This indicates that our method is effective not only for objects with heads but also for spatial consistency modeling of other objects.

2. Multiple objects with multiple heads. We will supplement the discussion details in the limitations section of the Main Paper. Multi-object generation is another important and challenging field in text-to-3D tasks. For SDS-based methods, the current most important issue in multi-object generation is not the Janus problem but how to handle the relationships between different objects. In Fig.H in Rebuttal File, we show the results of 'three corgis facing different directions.' It can be seen that the original Prolificdreamer cannot correctly generate 3 corgis, and there is some degree of sticking between the objects. Both our method and the current state-of-the-art multi-view consistent generation method DreamControl cannot correctly handle this example. Once the multi-view generation issue is resolved, our work will have more exploration possibilities. For example, a current mainstream idea is to generate individual assets for each object and then stitch them together[1]. Our method still has the potential to adapt to such methods.

2. Generation Quality

1. Regarding the blue shading you mentioned, this is an normal process handled by prolificdreamer during the generation. As shown in Fig.E in Rebuttal File, the original Prolificdreamer can also produce blue shading when changing seed, and our method can generate results without blue shading.

2. As shown in the Main Paper, our method does not suffer from a loss in generation quality compared to baseline methods and can improve the CLIP-Score. Low certainty generation helps the model find an optimization direction that better aligns with specific views, while the fine-grain detail of the generated images is determined by the generation model rather than certainty. Additionally, by aiding in view consistency modeling, the final generation results exhibit spatial consistency and better overall quality.

3. We also compared some fine-designed prompts, as shown in Fig.F and G in Rebuttal File. Our method ensures high-quality generation even when dealing with finely designed objects.

3. Quantitative Comparison

Please see the Table in Rebuttal File.

It presents a quantitative comparison with other methods dealing with the Janus Problem. It can be seen that our method, while having a series of advantages, surpasses existing methods in various metrics: it achieves the best overall performance when integrated with Prolificdreamer, strongly suppresses the Janus problem, and maintains the highest CLIP-score; when integrated with Dreamfusion, it has the best suppression effect on the Janus problem.

4. Prompt Library

Due to constraint of 6000 max length, please see Prompt Library in General Rebuttal to all reviewers.

We present 30 prompts selected from the Dreamfusion prompt library. Since our focus is on the Janus Problem, we have specifically chosen prompts that can evaluate method's effectiveness.

5. The Pig Generated by Stable Diffusion

All the samples in Fig.10 in Main Paper are generated by Stable Diffusion 2.1 base (not our method) to demonstrate that Stable Diffusion 2.1 base has view biases during generation. This supports our analysis and thus introduces our method. In some samples, the pig is not generated correctly, reflecting the limitations of the Stable Diffusion pre-trained model. We generated all the images at once on Stable Diffusion 2.1 base and evaluated their views without any selecting to ensure the accuracy of the presented results.

6. Failure Cases

Due to constraint of 6000 max length, please see Limitation Discussion in respone 2 for reviewer wiP5. Thanks very much.

Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.

Reference

[1] Zhou, Xiaoyu, et al. "Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting." arXiv preprint arXiv:2402.07207 (2024).

Thank you very much for your effort in reviewing our work! Our responses are as follows:

1. Quantitative Comparison with Other Methods Dealing with the Janus Problem

We have provided the quantitative comparison results with other methods in the table in Rebuttal File (as well as in the table below). It can be observed that our method can comprehensively surpass or partially match the performance of other methods. Specifically, our method shows the best suppression of the Janus Problem when embedded in dreamfusion; when embedded in prolificdreamer, it achieves the best overall performance, strongly suppressing the Janus Problem while maintaining the highest CLIP score.

Notably, our method can achieve good performance without requiring additional 3D priors or multi-stage fine-tuning, consuming minimal extra computational power (see quantitative comparison in the next response) and can be directly applied to SDS-based text-to-3D methods.

2. Quantitative Comparison of Computation Cost with DreamControl

We conducted a quantitative comparison of computation costs with Dreamcontrol. Notably, in the text-to-3D process, Dreamcontrol consists of two stages: Stage 1 - 3D Self-Prior Generation and Stage 2 - Control-Based Score Distribution. However, only the code for the latter has been released in their official codebase, while the former remains on their to-do list. Using Stage 2 alone cannot directly perform text-to-3D generation when given a text prompt. According to the official instructions, we need to provide an obj file or a threestudio checkpoint as a condition during the inference stage. Therefore, in the quantitative computation process for text-to-3D, we need to consider the costs of both stages (baseline+stage 2 or stage 1+stage 2).

We used prolificdreamer as the baseline model for both methods, since it is one of the best text-to-3D methods currently. We conducted our experiments on a single NVIDIA RTX A6000 GPU with the max steps set to 10,000. It is worth noting that the following data were all calculated during the text-to-3D generation (i.e., generating the corresponding 3D representation from the given text, not include pre-training time), which can be considered as the inference time in other methods, since the trainable params are the 3d representation we need. The quantitative comparison of computational overhead in text-to-3D task for one sample is as follows:

It can be observed that compared to DreamControl, our method:

1. Our method does not require two-stage processing, and the model parameter count, and runtime are reduced by more than 50% compared to DreamControl, GPU memory usage is also lower than DreamControl.

2. When comparing DreamControl (Stage 2) and LCGen (Ours), our method also performs better than DreamControl in all computational overhead metrics.

3. According to the Table in response 1, the performance of our method is not inferior to DreamControl across all metrics.

Thank you again for your efforts and suggestions! We will include detailed discussions on the above points in the Main Paper and Appendix.

Dear Reviewer sRbF, FWvW, and wiP5,

Thank you for your efforts in reviewing our paper! We greatly appreciate the valuable feedback provided and carefully posted our response. We understand that the reviewers may have very busy schedules. However, we are happy to have further discussion and answer any further questions in the Author-reviewer discussion phase. Looking forward to your feedback!

Best,

Authors of submission 898

Dear Reviewers,

Thank you for your efforts and valuable suggestions! We will continue revising the paper based on your feedback. Thank you once again to everyone!

Best regards,

Authors of submission 898