3D-GPT: Procedural 3D Modeling with Large Language Models

1. Similar discoveries on ChatGPT/ GPT-4's ability on generating parameters for images [1] or 3D objects [2]. No surprise that combining LLM with a better parametric 3D modeling tool InfiniGen could produce better visual results.

We appreciate the reviewer's comments regarding the perceived limitations of our contribution. However, we respectfully disagree with the assertion that our work lacks novelty. It is essential to differentiate between image generation [1] and 3D modeling, as these areas pose distinct challenges.

Moreover, the 3D examples provided by [2] do not reflect the complexities we address. Such a way to generate Blender Python code lacks a comprehensive understanding of API functions and their alignment with natural language. The generated Python codes would have high error rates. Yet, it lacks mechanisms for verifying code executability and correctness. Furthermore, directly generating Python codes does not effectively manage dependencies between 3D contents. For instance, when generating trees, the spatial relationship between the leaf and tree branches is not explicitly modeled. As a result, this method of applying LLMs to Blender code could produce inadequate 3D models, especially in the context of complex large-scale scenes.

Also, our research demonstrates that a simple combination of LLMs with the procedural generation library InfiniGen leads to subpar results. Note that, while the idea of combining LLMs with procedural 3D modeling may now appear straightforward in hindsight, creating an effective workflow is far from simple. Our solution involves three distinct agents: the Task Dispatch Agent interprets and chooses the right API calls, the Conceptualization Agent enhances the scene description with adequate details, and the Modeling Agent comprehends how textual descriptions translate into specific API parameters. This multi-agent approach ensures effective and detailed 3D model generation from textual inputs. Please refer to the results presented in Section 4.2 (Table 1, Figure 6) of our paper, as well as the updated supplementary materials (Section 6.2, Table 2, Table 3). These results clearly illustrate the significant roles played by each component within our framework in enhancing the final output.

2. Comparison with Previous Text-to-3D Methods:

In response to the reviewer's request for a comparison with previous text-to-3D methods, we have included an additional comparison with Dreamfusion [a] in the supplementary materials (Section 6.3, Figure 7). We show that while Dreamfusion generates reasonable 3D for single objects, it struggles to generate large-scale natural scenes, as demonstrated in our supplementary materials (Section 6.3, Figure 8). In comparison, our method is effective in both scenarios, showing the advantages of our approach in handling more complex 3D modeling tasks. Furthermore, our framework supports sequential scene editing without necessitating re-training.

Furthermore, it's worth noting that previous text-to-3D methods encountered challenges when generating large-scale natural scenes, as demonstrated in our supplementary materials (Section 6.3, Figure 8). These observations emphasize the strengths and advantages of our approach in handling more complex 3D modeling tasks.

[a] DreamFusion: Text-to-3D using 2D Diffusion. In ICLR 2023

Thank you for your valuable feedback and insightful comments. We truly appreciate the time and effort you dedicated to reviewing our work. Please let us know if any questions.

1. Submission to other venues (e.g., SIGGRAPH) We appreciate your suggestion to consider SIG and TOG for our paper submission. While we agree that our work aligns with the themes of these conferences, we strongly believe that our paper is also highly relevant for ICLR, fitting squarely within its broad scope of machine learning topics. Our work uniquely intersects with the domain of large language models (LLMs) and AI agents, areas of keen interest within the ICLR community.

The core of our contribution lies in a novel approach to formulating the text-to-3D problem. We introduce 3D-GPT to overcome significant hurdles in 3D scene creation. This includes generating expansive scenes, animating 3D models, maintaining coherence in 3D spaces, and creating intricate details in 3D objects.These are issues that previous methodologies have struggled with and are of considerable interest to ICLR readers and authors.

Furthermore, our future directions, including automatically generating procedural rules and fine-tuning models, align well with the interests of the ICLR community and require machine learning-based solutions that this community can provide.

2. Describe the system in a very high-level, impossible to implement the paper.

We have significantly revised Section 3 of the paper to provide more details on the various components of our method in a concise, yet reproducible way. We highlight the key changes as red color. Extended full examples of communication between agent, scene generation script and example implementation for modeling agent are included in the supplementary material. As mentioned elsewhere we will release all code and examples on acceptance. The revised version of the paper has been uploaded. : )

We want to thank you for your helpful comments and suggestions, which really improved our paper. We're grateful for your patience and commitment during the review process.

1. Expanding the Scope of 3D-GPT Beyond Plants and Forests:

Our method, which leverages the procedural algorithm library Infinigen, was originally designed for natural scene generation. However, its versatility extends beyond this specific application. Its key strength lies in its ability to proficiently handle various 3D modeling tasks in diverse contexts without necessitating fine-tuning. Our system operates in a zero-shot manner, independently processing each modeling function by the conceptualization and modeling agents. This independence ensures that our method can readily adapt to new functions, provided the requisite information. We will provide more complex object modeling in the supplementary before the rebuttal deadline.

2. Evaluation with Alternative Open-Source LLMs:

We have also conducted evaluations of our approach using Llama2/GPT4, as outlined in the supplementary materials (Section 6.2, Table 3). Our results demonstrate that the performance of our method remains consistent when applied across different LLMs, underscoring its robustness and broad applicability.

Thank you for your valuable review. Your feedback is much appreciated, and we're committed to addressing your concerns. We hope our explanations have addressed your concerns and we remain open to further discussion.

Thank you sincerely for sharing your invaluable review with us, and for your insightful feedback on how we can improve our evaluations. Your input is highly valued, and we are fully dedicated to addressing your concerns to the best of our abilities. Please feel free to reach out if you have any additional questions or if there is anything else we should provide.

Others: Suggestions and Queries

1. LLMs in 3D Modeling and Procedural Sequences:

We value your suggestion about leveraging LLMs for procedural rules in 3D modeling, a topic we've earmarked for future exploration in our paper. Our current research primarily focuses on utilizing LLMs for efficient function control and parameter inference in 3D modeling. This approach is pivotal to our overarching aim of reducing human intervention in the 3D modeling process through advanced LLM applications.

2. Function Dependencies and Complex Tasks:

The functions we have selected do establish dependencies. For example, the generate_trees() function can autonomously design tree branches, calculate surfaces programmatically, and accurately place leaves and flowers on these surfaces. This ensures correct dependencies and spares LLMs from the intricate task of determining precise 3D leaf/flower locations, a challenging feat for pretrained LLMs.

3. Availability of Full Examples of Input and Output:

We recognize the value of detailed examples for each agent, yet their inclusion would overly extend our manuscript. To keep our narrative concise, we will showcase these examples through supplementary videos or within our user interface. This approach ensures clarity and accessibility without overburdening the text. We will release the code that contains all examples.

4. Adaptability to Arbitrary Procedural Models:

Our system functions in a zero-shot manner, with each function processed independently by the conceptualization and modeling agents. This independence ensures that our method can adapt to new functions as long as the necessary information is supplied. As shown in our paper PAGE 5 bottom block, with provided code C, example E, and function document D, our method can understand the function and infer the required parameters.

5. Integration of Functions Selected by TDA in CA:

To be clear, in our current pipeline, the conceptualization agent processes only those functions that are selected by the task dispatch agent.

6. Efficiency for Professional Blender Users:

For Blender experts that not familiar with procedural generation, in the scenario depicted in Figure 4, transforming flat terrain into mountainous landscapes would require manually rebuilding terrain and repositioning each element, which is time-intensive. For Blender experts that are familiar with infinigen libraries might find this task straightforward. However, for each new procedural library, the experts are required to understand every new function, it also poses a significant challenge for professional Blender users to quickly grasp and utilize every new procedural library and its functions. Our approach aims to simplify and expedite this process.

Evaluation Improvement

1. Ablation Studies on D, C, I, E: Impact of Zero-shot/Few-shot Learning.

Thank you for your insightful comments. In response, we have updated our submission with comprehensive ablation studies in the supplementary material (Section 6.2) . Table 1 elucidates the roles of D, C, I, and E components. Absence of function documentation (D) impairs our method's ability to interpret function input parameters, thereby reducing the CLIP Score and elevating the Failure Rate. Without code (C), while the system grasps basic function and parameter knowledge, it experiences a decline in CLIP Score and an increased Failure Rate. The lack of information (I) hinders the conceptualization agent's capacity to enrich text with vital details, leading to a significant drop in CLIP Score. Excluding examples (E), as shown in Table 4, markedly diminishes the CLIP Score and heightens the Failure Rate. We noted no notable difference in performance between one-shot, two-shot, and three-shot prompts, though introducing more examples tends to reduce parameter diversity.

2. Impact of Scene Complexity and Parameter Count on Results.

The outcomes are indeed influenced by scene complexity and parameter count. The definition of scene complexity matters: if the initial text input features complex objects beyond our procedural algorithm's scope, the quality of generated content is limited. Conversely, detailed initial text descriptions enhance the conceptualization agent's understanding, yielding results more aligned with the input. The number of parameters is critical; removing functions or sub-functions (e.g., 'flower_generation' or 'control_flower_petal') limits modeling abilities, affecting controllability and diversity. Assessing the impact of function set size and parameter count is complex due to varying function significance. For example, removing 'control_flower_petal()' may not markedly affect the CLIP score, but eliminating 'terrain_generation()' substantially reduces it, as terrain modeling becomes impossible. Consequently, we opted not to include evaluations of this aspect, but we will update some visual example in supplementary before rebuttal deadline.

Lack of Detail:

1. Which Large Language Model (LLM) is used?

We utilized GPT-3.5 for all experiments.

2. What is the size of the function set F?

The function set F comprises all functions and subfunctions detailed in this script (https://github.com/princeton-vl/infinigen/blob/main/worldgen/generate.py), except for the 'creatures' class. These functions are integral to our scene creation process.

3. Does the size of F affect the quality of generated scenes?

Yes, the size of F directly impacts scene quality. Infinigen's scene generation employs a fixed function set. Removing a function, such as 'flower_generation,' eliminates the ability to create corresponding elements (e.g., flowers) in the scene. Similarly, omitting a sub-function like 'control_flower_petal' restricts control over specific aspects (e.g., flower petal movement), impacting scene controllability and diversity.

4. How many examples are provided for each function?

We provide one example per function.

5. Is the provided example related to the user's prompt L?

No, the prompt L is user-supplied. Our examples, one per function, might occasionally resemble some user inputs L, but they are not directly related.

We thank you for bringing our attention to the contemporary work of ViperGPT. We would like to underscore the distinctive features that set our work apart from theirs:

Task Complexity: Our approach tackles the more intricate challenge of 3D modeling. To address this complexity, we have implemented multiple agents that collaboratively handle various subtasks, resulting in a cooperative solution. This stands in contrast to ViperGPT, which lacks the incorporation of agents and cooperative strategies. In particular, ViperGPT only provides its agent with function names. In our case, we have a more extensive library with intricate functions, such as the sky modeling function we showcased in the paper Figure 5. This function demands a deep understanding of the Nishta sky modeling algorithm for parameter inference. If we were to apply their approach to our work, it would be inadequate for a pre-trained language model to comprehend these functions fully and, consequently, to make informed selections regarding the relevant functions and arguments based on the given instructions.

Scope of Focus: While both our work and ViperGPT share a common vision of employing a LLM to generate executable Python code, our emphasis diverges in the specific goal and methodology. Our work is primarily centered around 3D generation, whereas ViperGPT is oriented towards visual understanding. Moreover, our work employs multiple agents to solve the task. In light of these distinctions, we believe our work makes a unique contribution to the field, providing advancements in handling complex 3D modeling tasks through collaborative agent-based strategies. We will reference ViperGPT in the related work of our paper and include a description of the differences.

Thank you once again for your valuable feedback. We are dedicated to addressing any remaining concerns and ensuring the clarity and significance of our contributions.