GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models

Q1: State-of-the-art VLLMs can already perform well on existing benchmarks asking for geometric understanding.
A1: Thanks for your valuable feedback! We agree that SOTA VLLMs (such as OpenAI-o3) can already perform well on geometric understanding. However, when annotating simple local parts, Vertex-based captioning offers two key advantages over VLLM-based captioning:

1. Cost-free: It does not require deploying any local models or invoking paid API services.
2. Fully accurate: Compared to VLLM-based captioning, rule-based captioning ensures 100% correctness in annotation by explicitly extracting angle and length information of loops.

Q2: The quality of the annotated 221k local parts are not clear. Human evaluation for at least part of them is needed.
A2: We are pleased to see that the reviewer also recognized this point. In fact, we have conducted a human evaluation of the annotated local parts in the supplementary material. Specifically, as described in lines 20–22 of the supplementary material, we manually assessed the correctness of each caption by labeling it as either "Yes" (correct) or "No" (incorrect). Based on this evaluation of 1,000 randomly sampled parts, the overall captioning accuracy is 91.3% for Qwen2.5-VL-72B-Instruct and 86.5% for GPT-4o.

Q3: Use other parameter efficient finetuning strategies as well as full-model finetuning. It is also not sure if the rank of 8 and an alpha of 3 is the optimum setting for the finetuning.
A3: In fact, we have compared LoRA with full-model finetuning in the Table 1 in the supplementary material. The results show that full-model finetuning performs comparably to LoRA-based tuning, demonstrating the effectiveness of the LoRA strategy. Here, we compare another parameter efficient finetuning strategy (i.e., QLoRA [1]) with LoRA. The results are as follows:

The results show that QLoRA performs the worst among the evaluated methods. On the other hand, increasing the LoRA rank and alpha leads to slight performance improvements. However, higher rank and alpha values result in slower convergence. To balance performance and training efficiency and ensure a fair comparison with FlexCAD, we choose rank = 8 and alpha = 32 in our main experiments. Thanks!

[1] Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized llms. NeurIPS 2023.

Q4: Will larger LLMs perform better on this CAD task after the proposed finetuning strategy?
A4: Here, we compare the performance of Llama-3-70B equipped with LoRA. The results are as follows:

The results show that Llama-3-70B achieves superior performance, but it is more time-consuming and costly, as also explored in FlexCAD. In other words, as the LLM scale increases, both generation quality and instruction-following ability improve accordingly. Thanks!

Q1:The input requires a clean CAD shape program, which limits the application scope. Many CAD shapes in the ABC dataset cannot be edited by this method.
A1: We appreciate the reviewer’s concern regarding the current scope of our method. We would like to clarify the following point:

1. Our work focuses on common and practical CAD primitives, specifically sketches and extrusions. They are sufficiently expressive to cover a wide variety of shapes, and form the basis of many real-world CAD models [1].
2. As mentioned in [1] and [2], the ABC dataset is provided in the B-rep format, with no sufficient information to recover the user's design process [1], making it unsuitable for controllable CAD generation [2]. In contrast, we focus on the sketch-and-extrusion format, which align with the CAD design process in commercial CAD tools like SolidWorks and AutoCAD.
3. GeoCAD is designed to be extensible. Our GeoCAD can be easily adapted to support more complex operations such as chamfer and revolve once relevant data becomes available for training.

[1] R. Wu, C. Xiao, and C. Zheng. Deepcad: A deep generative network for computer-aided design models. ICCV 2021.
[2] Z. Zhang, S. Sun, W. Wang, et al. Flexcad: Unified and versatile controllable cad generation with fine-tuned large language models. ICLR, 2025.

Q2: Many local patterns are difficult to describe through text.
A2: We appreciate that the reviewer also recognized this point. In fact, we have discussed this in the limitation section. As shown in lines 88-92 in the supplementary material, in practice, certain complex local parts may be difficult or even impossible to describe precisely using text alone. As a potential direction for future work, we believe that combining textual input with hand-drawn images for local geometry could enable users to convey their design intent more effectively and accurately.

Q3: the entire sketch, the extrusion length and the extrusion combination operations (union and differ)
A3: This a good question. Actually, sketch-level editing is achievable based on our loop-level editing capability. This is because the loop serves as the fundamental element of a sketch. For example, if a user selects a sketch consisting of two symmetric trapezoids and wishes to replace them with two symmetric letter "E" shapes, the following automatic process can be performed:

1. Estimate the center point of each trapezoid by averaging its coordinate points, which are extracted using string matching.
2. Determine the symmetry axis based on the two center points.
3. Generate a local letter "E" through GeoCAD replacing one of two trapezoids.
4. Reflect the generated "E" across the symmetry axis to produce the second symmetric letter "E", thereby replacing both original trapezoids.

Thus, for sketch-level editing, it is unnecessary for the model to generate the entire sketch; generating only the fundamental loop using GeoCAD is sufficient. Subsequently, the remaining loops within the same sketch can be derived from the fundamental loop through scaling, rotation, translation, reflection or pattern operations.

On the other hand, there is no need to control parameters such as extrusion length based on designed prompts. Specifically:

1. Modifying the extrusion length and performing extrusion combination operations both require only a single parameter change in the CAD text. Instead of designing detailed prompts, directly modifying the parameters in the original CAD text is more efficient.
2. For more complex extrusion modifications, users must specify detailed parameters such as 3D translations and rotations. This process can be quite unintuitive for non-experts and unnecessarily tedious for experts. In contrast, GeoCAD is designed to be user-friendly for both non-experts and experts, where users only need to select the part they wish to modify and provide geometric instructions for local parts.

Q4:Two main technical contributions are very intuitive and specifically designed only for this task.
A4: Actually, the two-stage training process are common in LLM training. We are the first to propose a two-stage LLM training pipeline in controllable CAD generation. Specifically, in Stage 1, we introduce a data augmentation strategy tailored for local CAD generation, which was recognized by Reviewer HFw7 as both novel and effective.
In Stage 2, we intentionally retain the standard training strategy with minimal modifications from FlexCAD. The reasons are as follows:
i) To demonstrate that our method is simple yet effective. Without relying on elaborate training strategies, GeoCAD significantly outperforms FlexCAD and substantially enhances geometric instruction-following capabilities.
ii) To facilitate the development of an omni CAD LLM. A standard and unified training strategy allows for better integration between different models, such as FlexCAD and our GeoCAD, bringing us closer to a general-purpose, omni-capable CAD foundation model.

On the other hand, our paper presents novelty not only in the technical aspects but also in the task formulation. Specifically, we propose and solve a new task (i.e., local geometry-controllable CAD generation, see lines 27–35 in our main paper). All reviewers recognized the value of our proposed task (Reviewer td3y: very sensible, Reviewer 6Hfr: more impactful than generating from scratch, Reviewer ZjoJ: novel and interesting, Reviewer HFw7: reasonable). To support this task, we also collected a dedicated dataset of local parts and their corresponding captions, which Reviewer 6Hfr described as a sizeable and solid contribution.

Q1: mostly of curves + extrusions limit the space of shapes. How such methods can be used for more general, complex shapes?
A1: We appreciate the reviewer’s concern regarding the current scope of our method. We would like to clarify as follows:

1. Our work focuses on common and practical CAD primitives, specifically curves and extrusions. While these primitives are conceptually simple, they are sufficiently expressive to generate a wide range of shapes. Meanwhile, they also serve as the foundation for a significant portion of real-world parametric CAD models, as mentioned in [1].
2. Our GeoCAD has extensibility to more complex shapes. Beyond curve and extrusion operations, our GeoCAD is designed with extensibility in mind. It can be easily adapted to incorporate more advanced operations, such as chamfer and revolve, once appropriate data becomes available for training. We believe GeoCAD provides a scalable foundation for advancing toward general-purpose CAD modeling.

[1] R. Wu, C. Xiao, and C. Zheng. Deepcad: A deep generative network for computer-aided design models. ICCV 2021.

Q2: How this method would fare with more specific language instructions, like "a rectangle cutout that's 45 degrees to the nearest outer wall of the shape"
A2: Thank you for raising this insightful question.
As shown in Fig. 8 in our main paper, GeoCAD is capable of precisely controlling the key dimensional parameters when handling specific language instructions. To further evaluate its capability, we tested the instruction: "a rectangle cutout that's 45 degrees to the nearest outer wall of the shape." We generated 50 new parts based on this instruction. After human evaluation, 36 of the generated parts included a rectangle cutout. 23 of them had the rectangle cutout positioned near the nearest outer wall, while 6 of them had the rectangle cutout oriented at approximately 45 degrees to the nearest outer wall. In addition, we designed 50 complex instructions involving angle or length specifications. For each instruction, we generated 10 samples. After human evaluation, the instruction-following success rate (i.e., at least one of the 10 samples met the instruction) reached 86%. These results demonstrate that our model can follow fine-grained geometric instructions.

Q3: Figure 8 shows the ability for the model to understand measurements. Is this from the LLM base understanding from Llama pretraining, or were there captions in your dataset that contained dimensional geometric instructions?
A3: It is from captions in our dataset. As shown in lines 148-150 in our main paper, we incorporate key dimensional parameters into the captions. Thanks!

Q4: On average, how many CAD instructions compose the shapes? What is the maximum number of CAD instructions that the base LLM (llama-3-8b) able to handle (both theoretically and in practice)?
A4: On average, each shape is composed of 6 CAD instructions. Each CAD instruction contains approximately 120 tokens on average. Given that the base LLM (LLaMA-3-8B) has a context window of 8,192 tokens, it can theoretically handle up to around 68 CAD instructions. In practice, the maximum number of CAD instructions contained within a shape observed in the dataset is 19. Thanks!

Q5: Is the model capable of doing multiple rounds of consecutive edits, iteratively? Especially when the second instruction depends on the successful execution of the first instruction?
A5: Yes, our model is capable of doing multiple rounds of consecutive edits, iteratively. Specifically, at the beginning, the first loop is generated based on its corresponding geometric instruction. The generated text for this first loop is then used to infill the masked region in the CAD model. In this way, it ensures that subsequent loops are generated to align with the previously generated ones. Furthermore, this iterative process is user-friendly, as it only requires the user to specify the masked parts along with the related geometric instructions.

Q1: Novelty: Small but notable improvement in the training strategy. Technical novelty is limited.
A1: We appreciate the reviewer’s recognition of our method’s notable improvements and better results. Actually, compared to FlexCAD, our paper presents novelty in both the task and technical aspects.

1. Task Novelty. We propose and solve a new task (i.e., local geometry-controllable CAD generation, see lines 27–35 in our main paper). All reviewers recognized the value of our proposed task (Reviewer td3y: very sensible, Reviewer 6Hfr: more impactful than generating from scratch, Reviewer ZjoJ: novel and interesting, Reviewer HFw7: reasonable). To support this task, we also collected a dedicated dataset of local parts and their corresponding captions, which Reviewer 6Hfr described as a sizeable and solid contribution.
2. Technical Novelty. In contrast to the one-stage training strategy used in FlexCAD, we propose a two-stage training pipeline. In Stage 1, we introduce a data augmentation strategy tailored for local CAD generation, which was recognized by Reviewer HFw7 as both novel and effective.
   In Stage 2, we intentionally retain the standard training strategy with minimal modifications from FlexCAD. The reasons are as follows:
   i) To demonstrate that our method is simple yet effective. Without relying on elaborate training strategies, GeoCAD significantly outperforms FlexCAD and substantially enhances geometric instruction-following capabilities.
   ii) To facilitate the development of an omni CAD LLM. A standard and unified training strategy allows for better integration between different models, such as FlexCAD and our GeoCAD, bringing us closer to a general-purpose, omni-capable CAD foundation model.

Q2: Practicality: Loop-level edits are simple for CAD designers, and can be quickly done manually. More complex loop-level edits aren’t shown in the paper.
A2: In fact, as shown in Fig. 7 in our main paper and Fig. A3 in the supplementary material, our GeoCAD is capable of generating not only simple CAD shapes (e.g., an isosceles trapezoid) bot also complex shapes (e.g., different letters or irregular geometric shapes). Compared to manual editing, our GeoCAD have two major advantages:

1. User-friendliness for non-experts. Users do not require any specialized CAD drawing knowledge. They only need to select the part they wish to modify and input geometric instructions. In contrast, manual editing in traditional CAD systems typically require significant domain expertise. For example, to manually create an isosceles trapezoid, a user would need to draw a quadrilateral, ensure that two sides are parallel but of unequal length, the other two are of equal length, and align the shape with the remaining parts of the CAD model. Similarly, manually designing different letter shapes requires meticulous crafting of loops to match specific shape requirements.
2. Efficiency in generation time. When tested on an A100 GPU, GeoCAD generates a valid isosceles trapezoid and a letter shape in an average of 0.43s and 0.96s, respectively, significantly faster than manual editing.

On another note, as stated in FlexCAD, DeepCAD is currently the only dataset suitable for controllable 3D CAD generation. However, this dataset contains few gears, which limits GeoCAD’s ability to modify gear features such as the number of teeth. In the future, this limitation can be solved by collecting more gear-related data. Despite this limitation, our model can still easily handle other common complex shapes such as “an ellipse with rectangular extensions at both the top and bottom” and “a heart-shaped loop”, as shown in Fig. A3 in the supplementary material.

Q3: Sketch level editing should be demonstrated while preserving the design intent.
A3: We are glad the reviewer also notice this point. For sketch-level editing, if a sketch contains multiple loops, ideally, we would like to learn the inter-loop constraints (e.g., symmetry, patterns, etc.) that define the overall structure. However, as mentioned above, DeepCAD is currently the only dataset suitable for controllable 3D CAD generation, and unfortunately, such inter-loop constraint annotations are not provided in the dataset. Fortunately, even without supervision from these constraints, sketch-level editing is still achievable based on our loop-level editing capability. This is because the loop serves as the fundamental element of a sketch. For example, if a user selects a sketch consisting of two symmetric trapezoids and wishes to replace them with two symmetric letter "E" shapes, the following automatic process can be performed:

1. Estimate the center point of each trapezoid by averaging its coordinate points, which are extracted using string matching.
2. Determine the symmetry axis based on the two center points.
3. Generate a local letter "E" through GeoCAD replacing one of two trapezoids.
4. Reflect the generated "E" across the symmetry axis to produce the second symmetric letter "E", thereby replacing both original trapezoids.

Thus, during the sketch-level editing process, it is unnecessary for the model to generate the entire sketch; generating the fundamental loop alone suffices. Subsequently, the remaining loops within the same sketch can be derived from the fundamental loop or existing loops through scaling, rotation, translation, reflection or pattern operations. More specifically, i) for pattern-based sketch, it can be achieved by providing the pattern centers and the fundamental loops generated by GeoCAD. ii) in the case of sketch consisting of an outer loop and an inner loop, the inner loop does not need to be regenerated by the model when the outer loop is modified by GeoCAD. Instead, the inner loop can be easily updated through translation, scaling, or rotation operations to preserve the original positional relationships between the two loops.

Q4: Demonstrate examples where multiple loops or faces are masked.
A4: Our GeoCAD exhibits the generalization ability to generate multiple loops or faces when they are masked simultaneously. Specifically, we consider two cases:

1. Independent Loops. If the target loops are not constrained relative to each other, we generate them iteratively during inference. In each step, a single loop is generated based on a corresponding geometric instruction, and the resulting loop text is used to infill the masked region. This strategy ensures that subsequent loops are generated in a way that is consistent with previously generated ones. This process is also user-friendly, as it only requires the user to specify the masked region and desired geometric instructions.
2. Constrained Loops. If the loops are interrelated by design constraints (e.g., symmetry, patterns), GeoCAD supports complex structure generation by leveraging its loop-level editing capability, as discussed in A3.

Similarly, as faces are composed of one or more loops, the same strategy can be applied during the masking and generation of multiple faces.
According to the official NeurIPS policy, we cannot provide visualization results during rebuttal. We will add them in the revised paper. Thanks!

Q5: Demonstrate if design intent is preserved when editing the shape
A5: As mentioned in A3, our GeoCAD has the ability to preserve the design intent (e.g., symmetry, patterns) solely based on loop-level editing. We argue that the loop serves as the fundamental element of a sketch. When generating a sketch consisting of constrained Loops, it is unnecessary for the model to generate the entire sketch; generating the fundamental loop alone suffices. Thus, in a sketch consisting of an outer loop and an inner loop, the inner loop does not need to be regenerated by the model when the dimension of the outer loop is reduced by GeoCAD. To preserve the existing positional relationships between the two loops, the inner loop can be easily updated through translation, scaling, or rotation operations.
According to the official NeurIPS policy, we cannot provide visualization results during rebuttal. We will add them in the revised paper. Thanks!