ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling

Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.

W1. Clarity and Scope of Method Presentation We appreciate the reviewer’s feedback and acknowledge that the inclusion of multiple modules (e.g., GPS representation, multi-path modeling, component-aware painting) may impact clarity. In the revision, we will:

• Clarify the GPS representation and shape modeling with improved notation and explanation.
• Provide a simplified overview figure with clear annotations of each stage.
• Relocate implementation details to the appendix and bring key results and analyses from the appendix into the main text. We aim to make the system easier to follow without sacrificing technical completeness.

---

W2. Texture Quality and Scope of Contribution While shape generation is the primary focus of ShapeCraft, the texture module is included to demonstrate the extensibility of the GPS representation for practical workflows. The diffusion model is invoked as a functional component within the LLM agentic pipeline, conditioned on the GPS structure and executed post shape modeling.

Our component-aware SDS adds node-level descriptions and geometry to guide texturing. We conduct an ablation study of the 10 prompts in Tab.1, this design achieves higher alignment with the prompt compared to the original SDS:

In Fig. 3, our method successfully follows texture-related prompts, e.g., generating “rust and dirt spots, primarily in brown and orange” for the air conditioner. In cases with vague or missing texture descriptions, the model falls back to common material priors (e.g., metallic for fridge, leather-like for sofa).

Compared to prior LLM-based shape modeling methods, we are the first to integrate a texture generation module, highlighting a promising direction for future research, even though visual fidelity remains limited.

We will clarify this scope in the revision and emphasize that texture generation is an optional extension of our procedural modeling framework.

---

Q1. Clarifying Use of Blender API Thank you for the helpful suggestion. We agree that explicitly stating our use of the Blender API when introducing the GPS representation would improve clarity. In the revision, we will update Section 3.1 to clarify that each code snippet $P_i​$ in the GPS graph ultimately corresponds to Blender API calls.

---

Q2. Clarification on Hierarchical Parsing and Representation Bootstrapping Thank you for the thoughtful feedback. We acknowledge that the original notation in Eq.1 and Eq.2 may appear opaque, and we will revise the paper to provide a clearer, more intuitive explanation in the main text.

Hierarchical Shape Parsing via CoT Reasoning
As described in Eq.1, DecomposeHierarchy refers to the first step of LLM reasoning, where the model breaks down the input shape description into semantic levels of abstraction. We give an example "chair" in Line 150-151.

Next, ParseNodes converts each leaf in the hierarchy into a node-level description, which is then passed to BuildGraph, a deterministic Python procedure that instantiates the GPS representation (e.g., generates component identifiers, bounding box placeholders, and empty code slots). These are stored in memory and serialized as gps.json (see examples in our supplementary material).

Representation Bootstrapping
As shown in Eq.2 and illustrated in Appendix Fig. 2, this process refines the initial GPS layout using visual feedback:

1. GenerateBoundingBoxShapeProgram: The LLM generates an initial bounding box program $G_0$.
2. ExecuteShapeProgram: The boxes are rendered into multi-view images $I$.
3. VlmFeedback A VLM is prompted with the rendered images $I_{i-1}$ and asked to evaluate completeness, correctness, and spatial coherence.
4. GraphUpdate: The bounding box coder agent revises the previous bounding box program $G_{i-1}$ with the VLM feedback $F_{i-1}$ resulting in updated results $G_i$.
5. This loop continues (or early stopping) until a satisfactory version is selected for downstream modeling. We will revise Section 3.1 and 3.2 to more clearly explain these two modules in intuitive language, minimizing notation where possible.

---

Q3. Comparison with BlenderLLM Thank you for raising this. We have conducted a comparison with BlenderLLM using the same prompts as in Tab.1. As shown below, ShapeCraft outperforms BlenderLLM across all metrics, particularly in compile success rate:

BlenderLLM is tailored for CAD modeling, and its LLM is fine-tuned on instructive prompts and simple shapes. This makes it less effective on open-ended natural language prompts from the MARVEL benchmark. We will include this comparison in the revised version and clarify the distinction in scope and generalizability between the two systems.

Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.

W1. LLM Dependency for complex or ambiguous prompts We thank the reviewer for raising this important point. To mitigate LLM inconsistencies under complex or ambiguous prompts, ShapeCraft integrates several mechanisms:

1. Hierarchical Decomposition + Graph-Based Representation
   We first decompose the input prompt into semantically coherent components using structured chain-of-thought (CoT) reasoning. To further reduce ambiguity, we augment each node's description with geometry-level details (e.g., shape, size cues), which improves the LLM’s ability to reason about structure and generate accurate shape programs. This hierarchical strategy narrows the search space by assigning each component a well-scoped modeling subtask. As discussed in Lines 131–132, each node maintains its own component-level search space, enabling the generation of more precise and consistent code. These geometry-aware sub-prompts serve as clear and focused entry points for downstream modeling.

2. Representation Bootstrapping with VLM Feedback As shown in Appendix Fig. 2, initial LLM outputs (e.g., missing the “freezer” component) are visually verified via rendered bounding boxes. Feedback from the VLM guides corrections of hallucinations, omissions, and spatial errors, improving reliability.

3. Multi-Path Sampling and Iterative Refinement
   Rather than relying on a single output, we explore multiple modelling paths (Sec. 3.2). Each is refined with VLM feedback independently (as shown in Fig. 5A, Appendix Fig. 3), increasing robustness and reducing sensitivity to any single LLM failure. Together, these design choices convert complex prompts into manageable sub-tasks, enhance alignment with visual semantics, and significantly improve stability across varied prompt scenarios.

---

W2. Effectiveness of GPS representation for abstract, fantastical and dynamic shapes We thank the reviewer for this thoughtful question. ShapeCraft is designed primarily for static, structured shapes, which already present a significant challenge for current LLMs. Our system is most effective when prompts describe geometrically grounded, component-based objects. For highly abstract or fantastical shapes, current LLMs often fail to parse meaningful structure, limiting the reliability of GPS representation. We acknowledge this limitation and plan to explore hybrid approaches (e.g., native 3D model add-on) for such cases in future work.

That said, our Blender wrapper library is modular and extensible, allowing integration with various add-ons or simulation toolkits. This flexibility lays a foundation for handling abstract, fantastical components, given the right parsing operations and prompt design.

For dynamic or articulated shapes, we provide a preliminary demonstration in Fig. 5(B) and in the Appendix video, where animations are applied directly to generated components (e.g., opening/closing laptop). This is made possible by our explicit component-based GPS representation, which avoids the need for post-hoc mesh segmentation.

---

W3 & Q2. Computational cost We report the average number of API calls and runtime for the methods evaluated in Tab. 1. API calls in our setup are categorized into code generation $\mathcal{C}$, feedback $\mathcal{F}$, and question generation $\mathcal{Q}$. We follow this taxonomy to compare against 3D-PREMISE ($\mathcal{C}$+$\mathcal{F}$) and CADCodeVerify ($\mathcal{C}$+$\mathcal{Q}$+$\mathcal{F}$), both evaluated with 3 update steps. ShapeCraft consists of two phases: GPS initialization and iterative modeling. As detailed in Appendix Sec. A L7–9, GPS initialization involves 3 API calls for bounding box generation. The iterative modeling phase issues $P\times T$ modeling attempts. Thus, the total API calls for ShapeCraft are: $3+P×T×(\mathcal{C}+\mathcal{F})$. Under a minimal configuration (P=1, T=1), ShapeCraft outperforms previous LLM-based baselines in both quality and computational cost.

We further compare against diffusion-based methods. As shown in Fig. 3 and Fig. 4, our procedural generation produces more structured and editable meshes, whereas diffusion-based outputs often suffer from marching-cube artifacts and topological noise, limiting their usability in downstream tasks.

In particular, ShapeCraft achieves comparable or better IoGT than MVDream (distill from diffusion model), while being significantly faster. Compared to TRELLIS-text-large (diffusion-based native 3D model trained on 500K shapes using 64 A100 GPUs), ShapeCraft is training-free yet approaches its performance. Notably, our evaluation set (Objaverse-MARVEL) is included in TRELLIS's training set, whereas ShapeCraft generalizes without task-specific fine-tuning.

We also demonstrate ShapeCraft’s flexibility in post-modeling edits (e.g., animation) in Fig. 5(B) and Appendix Fig. 6, highlighting its practicality for interactive applications.

---

Q1. Risk of VLM Failure in Iterative Refinement We acknowledge this limitation. While our VLM-guided bootstrapping significantly improves initial outputs, it is not perfect. To mitigate this:

• We adopt component-level refinement, so even if one node's error is missed, it does not propagate globally.
• Our multi-path sampling strategy introduces diversity and redundancy, increasing the chance of discovering a correct variant.
• In practice, VLM feedback successfully catches most common structural issues (e.g., missing parts, overlaps), as shown in Fig. 5A and Appendix Fig.2,3,4.

---

Q3. Complex material interactions We acknowledge that BRDF alone cannot fully capture translucent or subsurface effects like glass, skin, or wax. However, our texture field design is modular and extendable. By incorporating BTDF models (e.g., adding transmission coefficients and subsurface weight in Eq.(3)), our pipeline can be extended to support complex material types.

---

Q4. Failure Case Analysis and Limitations We appreciate the reviewer’s suggestion. We acknowledge two main sources of failure in ShapeCraft:

1. Ambiguous or Underspecified Prompts
   ShapeCraft relies on prompts containing explicit geometric cues. When prompts are vague or lack structural details, the shape parsing may fail, or the upsampled node descriptions may include hallucinated components. For example, prompts like “a magical object with flowing form” cannot be reliably decomposed into actionable modeling steps.

2. Complex Topologies
   Our current wrapper library is optimized for modeling structured shapes. It lacks native support for fine-grained or highly organic details (e.g., tails, ears, branches). However, we are actively experimenting with extensions, such as integrating Hunyuan3D add-ons, to support more diverse and irregular topologies. We will include a detailed discussion of these limitations in the revision, and present qualitative failure cases and possible mitigation strategies.

Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.

W1. Novelty and Comparison with Scene Generation Works

We respectfully disagree with the claim that our GPS representation is similar to prior scene generation approaches such as Holodeck, LayoutGPT, FlairGPT, or SceneCraft. These methods focus on scene layout construction using LLMs to generate graphs over object bounding boxes and then retrieve pre-existing 3D assets (e.g., from Objaverse). Their graph representations encode coarse inter-object spatial relationships, but are not designed for procedural shape modeling.

In contrast, ShapeCraft focuses on generating novel 3D shapes from scratch. Our Graph-based Procedural Shape (GPS) representation encodes component-level semantics, where each node includes both a bounding volume $B_i$ and an executable modeling program $P_i$. Unlike layout graphs, $P_i$ captures fine-grained geometric construction logic, not just object placement.

We also argue that shape decomposition is inherently more challenging than scene decomposition, as it requires semantic and functional understanding of substructures within a single object, rather than arranging independent entities in space.

We will clarify this distinction in the revised Related Work section under "LLM Agents for Scene Layout" and include a focused discussion of the above methods.

---

W2.1 Iterative refinement based on visual feedback is expensive.

We respectfully disagree that our refinement strategy is "expensive." ShapeCraft is designed to be efficient and lightweight:

1. We use open-source LLMs/VLMs (e.g., Qwen3-235B), enabling low-cost, local deployment without reliance on GPT-4.
2. Our multi-path sampling supports parallelized component modeling, as nodes are independent post-decomposition.
3. We employ early stopping based on VLM feedback scores to avoid unnecessary queries.
4. Despite being training-free, ShapeCraft outperforms fine-tuned methods like LLaMA-Mesh on both geometry and alignment metrics.

---

W2. 2 Comparison with 3D-GPT, SceneCraft and BlenderAlchemy

• 3D-GPT builds scenes using the Infinigen function library; it does not support shape modeling or procedural decomposition.
• SceneCraft (distinct name from our ShapeCraft) performs scene-level layout generation using a constraint satisfaction algorithm and retrieves assets.
• BlenderAlchemy focuses on material optimization through visual feedback loops, rather than structural modeling.

In contrast, ShapeCraft decomposes shape descriptions into fine-grained components and introduces a node-level multi-path sampling strategy. Our VLM feedback loop operates at the per-node level, comparing rendered geometry and component descriptions—not full-scene images—thereby improving both feedback specificity and modeling granularity. We will clarify this discussion in the revised Related Work section.

---

W3. Clarification on painting module The diffusion model is invoked as a callable module within our pipeline, it is tightly integrated into the LLM-driven agentic workflow—triggered after shape modeling and conditioned on the GPS structure. Thus, the painting module is not separate, but a downstream step in the procedural generation process.

Our method differs from optimization-based pipelines (e.g., MVDream) that regress entire textures. Instead, we learn a BRDF field over global and local UV coordinates, enabling a physically meaningful and post-editing-friendly texture.

Compared to original SDS, which optimizes the texture globally, our component-aware Score Distillation also introduces node-level conditioning: each component’s geometry and textual description guides rendering and optimization for its corresponding region. This improves semantic alignment, as shown in the following ablation over 10 prompts:

---

W4. Fairness for comparison. To ensure a fair and controlled comparison, we explicitly re-implemented all baselines using the same LLM/VLM setup as ShapeCraft, specifically Qwen3-235B-A22B and Qwen-VL-Max (as stated in Appendix A Line 5). This isolates the performance gain to our GPS representation and multi-path sampling strategy for iterative modeling, not LLM scaling or instruction tuning. We will make this clarification more explicit in the main paper to avoid confusion.

---

Q1 & Limitation. Computational resource and token cost. We report the average token cost, runtime and the number of API calls for all methods in Tab.1. ShapeCraft incurs higher token usage due to its generation of more detailed, component-wise programs, including multiple code snippets per object.

However, token count alone does not reflect actual computational cost. We also report runtime and API call counts. Despite modest increases in these metrics, ShapeCraft delivers up to 41% IoGT improvement over CADCodeVerify, demonstrating a favorable trade-off. Moreover, under a minimal configuration (P=1, T=1), ShapeCraft achieves both better performance and lower runtime than other LLM-based baselines.

Limitation: Compared to Trellis Compared to TRELLIS-text-large (diffusion-based native 3D model trained on 500K shapes using 64 A100 GPUs) in Tab.1, ShapeCraft is training-free yet approaches its performance. Notably, our evaluation set (Objaverse-MARVEL) is included in TRELLIS's training set, whereas ShapeCraft generalizes without task-specific fine-tuning. And ShapeCraft produces procedural generation, which is more structured (in Fig. 3 and Fig. 4) and editable meshes (Fig. 5(B) and Appendix Fig. 6), whereas Trellis outputs often suffer from marching-cube artifacts and topological noise

---

Q2. Compared to LLM with the “thinking” module. As described in Line 144, we adopt a hierarchical CoT strategy during shape parsing: the LLM first infers a global semantic root node $H_0$, then recursively decomposes it into component-level nodes $H_i$ based on semantic and structural relationships. This structured CoT constrains the reasoning space and leads to more reliable and interpretable initialization. However, we disable verbose or unconstrained CoT (indicate "thinking" module in LLM/VLM) to ensure efficiency and determinism. This decision is based on our observation that free-form CoT often introduces redundant steps or hallucinated geometry operations.

To evaluate our hierarchical CoT strategy, we conducted an ablation using open-ended “thinking” models for prompts in Tab.1. As shown below, free-form CoT reasoning fails to maintain spatial consistency across components and often produces invalid results. We report compile rate as the percentage of prompts yielding a valid result in a single run:

---

Q3. How does the LLM understand the proportion ratios? If the concern relates to how the LLM determines component scale and proportion, we clarify as follows: The LLM infers relative proportions based on its pretrained spatial priors (position embedding) and component descriptions (e.g., “a thin screen above a wide base”). During GraphInit, it generates all bounding boxes in an autoregressive run, so later boxes are conditioned on earlier ones, ensuring internal proportional coherence.

We allow the LLM to freely choose coordinate scales and apply normalization to a unit cube post-hoc to maintain consistency across shapes. Further refinement via VLM feedback ensures spatial accuracy when needed.

Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.

W1.1 Concern on LLM/VLM Dependency and Failure Modes To address potential issues such as LLM hallucination and geometric misinterpretation, we designed two core mechanisms:

1. Representation Bootstrapping As shown in Appendix Fig.2, initial LLM parsing misses components (e.g., "freezer" in the fridge example). Our bootstrapping mechanism (Sec. 3.1) introduces a VLM-based visual verification loop: the LLM’s output is rendered as bounding-box geometry and compared to the textual input. Feedback from the VLM guides the correction of omissions, hallucinations, and spatial inconsistencies.
2. Multi-Path Sampling with Iterative Updates Instead of relying on a single deterministic program, we adopt multi-path sampling (Sec. 3.2), which explores diverse shape programs in parallel. Each path undergoes iterative refinement with VLM feedback to correct execution errors or misalignments (as shown in Fig. 5(A) and Appendix Fig.3). This redundancy (lowers the risk of not having any compiled shape programs in the end) improves robustness against LLM/VLM inaccuracies and API hallucinations. As a result, ShapeCraft achieves a significantly higher compile success rate compared to baseline methods and even strong CoT-based reasoning models.

Failure Modes Even equipped with above functions, ShapeCraft may fail when provided with ambiguous or underspecified prompts. In such cases, shape parsing may produce hallucinated or incomplete node descriptions, making downstream modeling unreliable. For instance, prompts like “an amazing chair with complicated decoration” cannot be decomposed into actionable components.

---

W1.2 & Q2: Blender Wrapper Library and Its Quantitative Ablation Study Qualitative ablation study of wrapper library can be found in Appendix Fig. 5. We further conducted a quantitative ablation study over 10 prompts subset below:

Our Blender Wrapper Library is designed to reduce the complexity of the API space exposed to the LLM. And results show that Blender Wrapper Library provide better shape results and produce more concise program. It abstracts frequent modeling operations—such as primitive creation, Boolean ops, and modifiers—into a curated set of consistent, high-level commands. Instead of raw Blender Python calls (e.g., bpy.ops.mesh.primitive_cube_add with multiple parameters), we use simplified wrappers like cube(location, rotation, scale). These abstractions align better with LLM priors and reduce trial-and-error programming during code generation. Detailed API can be found in Appendix prompt 6.

---

W1.3 & Q3: Ablation on Multi-Path Sampling Thank you for pointing this out. We provide more quantitative ablations and clarifications below. In our experiments, we used P=3 and T=3 for shape modeling, and P=1, T=3 for bounding box generation mentioned in Appendix Sec. A, L7–9. These settings strike a balance between performance and efficiency.

To analyze diversity, we visualize multiple sampled paths in Appendix Fig. 3. Empirically, we observe that: 1) For simple shapes, paths tend to converge to similar modeling strategies due to low ambiguity. 2) For complex or ambiguous prompts, different paths often explore genuinely distinct decomposition strategies and shape programs, especially under higher temperature settings. In general, we observe 2–3 unique strategies across 3 paths for complex cases, demonstrating the system's capacity to explore modeling alternatives.

We also conduct quantitative ablation with varying (P, T) to evaluate the benefit of multi-path sampling. Results below:

These results show that multi-path sampling significantly improves both geometry quality (IoGT, Hausdorff) and prompt alignment (CLIP), confirming its effectiveness.

---

W2. Scalability to complex topologies. We appreciate the reviewer’s concern. While many examples focus on furniture, ShapeCraft is not restricted to simple primitives. As shown in Fig. 4, our Stichter Banjo example includes curved surfaces and interlocking parts, illustrating the system’s ability to model non-trivial geometries.

Importantly, our Blender Wrapper Library is extensible, supporting advanced operations like subdivision, bevel, and boolean. Since our agent generates tool-based programs, ShapeCraft can scale to any backend modeling API, given appropriate documentation.

Moreover, ShapeCraft’s hierarchical decomposition and modular program structure naturally support complex assemblies such as CAD parts or articulated components. In revisions, we will include qualitative results on CAD-style shapes.

---

W3. Complexity and computational cost. Our method scales well with increasing shape complexity and number of parsed components, due to the parallel execution of per-node modeling tasks. The use of our Blender Wrapper Library also reduces the overall program length by abstracting frequent operations, which helps conserve context window compared to native Blender APIs. This is supported by code length comparisons in Appendix Fig. 5 and discussed in W1.2 & Q2.

We also report the number of API calls and runtime for the methods in Tab.1. API calls are mainly related to code generation $\mathcal{C}$, feedback $\mathcal{F}$ and question generation $\mathcal{Q}$ (only used in CADCodeVertify). ShapeCraft consists of GPS representation initialization and iterative shape modeling. GPS representation initialization needs 3 API calls with one trajectory of three updates for bounding box generation mentioned in Appendix Sec. A, L7–9. The number of Iterative shape modeling depends on the $P$ and $T$. Under a minimal configuration (P=1, T=1), it outperforms previous LLM-based methods both in quality and computational cost:

We will include this analysis in the revised paper to clarify the trade-offs between program complexity and execution efficiency.

---

Q1. Bounding Box Generation and Spatial Coherence in GraphInit Thank you for the thoughtful question. During the GraphInit phase, bounding boxes $B_i$ are generated by the LLM through an upsampled node description $x_i$ that supply relative spatial relationships. The LLM uses a wrapper method in Appendix Prompt 4:

cube_bounding_box(name="node_name_bbox", position=(x, y, z), scale=(sx, sy, sz))

Although no visual feedback is available at this stage, LLMs exhibit basic spatial reasoning due to their pretraining. We do not impose fixed value ranges, allowing the model to freely choose a coordinate scale; normalization is applied afterward to ensure consistent layout.

Notably, the bounding box agent generates all $B_i$ in a single autoregressive run. This implicitly conditions each box on previous ones (i.e., models $P(B_i|B_{<i})$), helping maintain global spatial coherence even before visual grounding.

Subsequently, the bootstrapping phase Eq. 2 integrates VLM feedback to further refine placements. This two-stage design—coarse layout via LLM, fine adjustment via vision—strikes a balance between reasoning efficiency and geometric fidelity.