FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models

Q1: Why are LLMs the good choice to do this task?
A1: As mentioned in Lines 68–76, the main reasons are as follows:
(1) LLMs have demonstrated remarkable success in handling diverse user queries with a single, unified model. Thus, we explored and successfully introduced LLM-based FlexCAD to achieve controllable generation across seven different hierarchies within a unified framework.
(2) LLMs may have acquired CAD-related knowledge during pre-training by learning CAD-specific code, such as JSCAD. As shown in Table 2, this pre-training knowledge contributes to performance gains.
(3) Before the advent of LLMs, small transformer-based models were explored for CAD-related generation tasks, highlighting the potential of transformers for these tasks from a different perspective.
(4) Leveraging pre-trained weights, our FlexCAD inherits instruction-following capabilities of LLMs, allowing it to understand instructions that never appeared during fine-tuning (see A5 for more details).

Q2: Could LLaMA-3 do CAD generation in a zero-shot manner? How helpful is the proposed fine-tuning method compared with the zero-shot setting?
A2: LLaMA-3 (both 7B and 72B) cannot do CAD generation in a zero-shot manner. Without fine-tuning, the output format is largely incompatible with our CAD textual representation. For instance, point coordinates for circles, arcs and lines are often incomplete, and the end of each hierarchy is always missing. In contrast, with our proposed fine-tuning method, the output fully adheres to the expected format, and the generated CAD text can be successfully rendered, demonstrating the effectiveness of fine-tuning.

Q3: The evaluation for prompt-result alignment is unclear, e.g., intent understanding and instruction following.
A3: (1) The prompt-result alignment can be evaluated by checking whether the type and number of hierarchies in the generated CAD match those specified in the prompt. We utilize string matching for automated evaluation. The overall score across all hierarchies is over 98%, with the score for the most challenging level, the curve level, reaching 95%. We did not include this in the original version as it is relatively simple for LLMs, while we will add it in the revision for comprehensive evaluation.
(2) We also use the Realism metric to evaluate whether the newly generated parts under the corresponding prompts integrate seamlessly with the remaining unmasked elements without conflict. As shown in Table 1, our FlexCAD significantly surpasses baselines in terms of the Realism metric.

Q4: The instructions used to prompt GPT-4o.
Thank you for pointing this out. We have added a detailed GPT-4o prompt example in Fig. 18 in our paper.

Q5: Does the proposed method inherit some advantages of LLMs, such as instruction following and in-context learning?
A5: Yes, it does. With pre-trained knowledge, our FlexCAD inherits the instruction-following and in-context learning capabilities of LLMs, allowing it to understand instructions that never appeared during fine-tuning. To provide further evidence, we perform additional experiments. Taking sketch-level generation as an example, we add additional instructions (e.g., 'Ensure that the sketch contains at least one circle') at the end of the original instructions. Consequently, the proportion of newly generated sketches containing circles surges from 37% to 89% out of a total of 1000 sketches generated. Moreover, for in-context learning, when we provide FlexCAD with five exemplars where all the answer sketches contain circles, the proportion of newly generated sketches containing circles increases from 37% to 65%.

Q6: Can users express their intent using natural language and then the transformation task is given to LLMs?
Currently, users can primarily express their intent for controllable generation by masking the corresponding field. Users can also utilize natural language (i.e., text) to instruct our FlexCAD to meet their intent, as discussed in A5. However, achieving complex text-to-CAD alignment remains challenging. Specifically, there is a lack of relevant textual descriptions for each CAD model, such as 'a regular hexagon' or 'a desk with four legs', which are necessary for fine-tuning. We believe that by constructing a comprehensive text-to-CAD dataset and further fine-tuning our FlexCAD in the future, users will be able to more easily and effectively guide CAD generation using natural language.

Q1: Additional CAD datasets.
A1: We appreciate your feedback and will evaluate additional CAD datasets once a suitable one is found or collected. Currently, DeepCAD is the only dataset suitable for evaluation, and the detailed reasons are explained below:
(1) DeepCAD is a large-scale 3D CAD dataset, consisting of more than 178k samples.
(2) Compared to 2D sketch datasets, DeepCAD better reflects our controllability, not only on sketches but also on extrusions.
(3) Compared to other 3D CAD datasets, such as the ABC dataset (Abc: A Big CAD Model Dataset for Geometric Deep Learning, CVPR 2019), DeepCAD includes sketch-and-extrusion sequences, which align with the CAD design process in commercial CAD tools like SolidWorks and AutoCAD.
(4) Given the characteristics of DeepCAD, baselines (i.e., SkexGen [ICML 2022] and Hnc-cad [ICML 2023]) also use only DeepCAD, making it more convenient to compare performance.
(5) Beyond our controllable generation task, for other tasks such as Text-to-CAD, the paper (Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts, NeurIPS 2024) also uses only DeepCAD.

Q2: Including OpenECAD and CadVLM would strengthen the context and positioning of the work.
A2: Good suggestion. We will cite and discuss them in the revision. Notably, unlike our focus on local controllable generation, OpenECAD and CadVLM focus on the image-to-CAD task using VLMs.

Q3: During training, what is the average mask ratio per training sample, and is there any observed relationship between performance and the mask ratio?
A3: Instead of masking a fixed ratio for each sample, our masking mechanism is to mask a hierarchy from CAD, sketch-extrusion, extrusion, sketch, face, loop and curve, as shown in Fig. 4. The average mask ratios per training sample for the CAD, sketch-extrusion, extrusion, sketch, face, loop and curve hierarchies are approximately 100%, 49.8%, 7.5%, 42.2%, 40.6%, 9.79% and 8.42%, respectively. Generally, as reported in Table 5, coarser-grained hierarchies (e.g., CAD and sketch-extrusion) with larger mask ratios outperform finer-grained hierarchies (e.g., face, loop, and curve) with smaller mask ratios. This is likely because the new generations produced by our FlexCAD for the masked part strictly align with the remaining (unmasked) conditions. In other words, the new generations avoid potential conflict with the remaining conditions in 3D space. As the mask ratio decreases, the remaining conditions (i.e., constraints for the masked part) become longer and more complex, thereby limiting the overall quality and diversity of the masked part.

Q4: As shown in Figure 5, which output is selected for the final evaluation?
A4: For the quantitative comparison (i.e., Table 1), as mentioned in Lines 315-316, we randomly selected 1,000 CAD models from the test set. For each method, we generated 10 predictions per CAD model, resulting in a total of 10,000 CAD models, all of which were used for the final evaluation. For the qualitative comparison (i.e., Fig. 5), we also generated 10 predictions for each CAD model and randomly selected four newly predicted models for each method, as mentioned in Line 365.

Q1: Novelty is limited. The training methods are very standard for tuning LLM.
A1: Actually, this paper is a novel use of LLMs for CAD design, aligning with the feedback from reviewer 4Ahh. Here, we highlight our novelty and explain the training methods as follows:
(1) Baselines (SkexGen and Hnc-cad) generally utilize VQ-VAE in combination with small transformers to produce numerical codes for CAD generation. On the other hand, we shift this generation paradigm by introducing LLMs for controllable CAD generation. To further clarify our novelty, we will explain the difference between our LLM-based FlexCAD and transformers of baselines in A2.
(2) Yes, the training methods are very standard (see Line 261). We deliberately maintain standard training methods with minimal changes, considering this a key advantage of our approach. The reason is threefold:

• Demonstrating the Power of Simplicity. Our simple method significantly outperforms baselines that rely on complex models and training techniques. By showcasing this to the community, we aim to encourage future work to build on this foundation, advancing the field with practical and effective solutions.
• Evidence from Proven Approaches. In other domains, similar efforts using standard LLM training methods [Refs A, B, C, D, E] have shown significant impact. We believe our approach will similarly transform the CAD generation domain.
  Ref A: PointLLM: Empowering Large Language Models to Understand Point Clouds (ECCV2024)
  Ref B: Motiongpt: Finetuned LLMs Are General-Purpose Motion Generators (AAAI2024)
  Ref C: AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (Neurips2024)
  Ref D: UniAudio: Towards Universal Audio Generation with Large Language Models (ICML2024)
  Ref E: Fine-Tuned Language Models Generate Stable Inorganic Materials as Text (ICLR2024)
• Alignment with AGI Goals. Achieving artificial general intelligence (AGI) requires models capable of generalizing knowledge across diverse tasks and domains. The autoregressive Transformer and the next token prediction employed by LLMs are considered as promising methods towards achieving AGI. By using standard training methods of LLMs with minimal modifications, our work in CAD design can seamlessly integrate with efforts in other domains, contributing to the broader pursuit of AGI.

Q2: LLMs vs. Transformers in SkexGen and Hnc-cad
A2: While both our method and baselines utilize transformers, there are significant distinctions. Our approach primarily involves fine-tuning LLMs, whereas baselines rely on VQ-VAE frameworks. The differences are introduced below:
(1) As mentioned in Line 94, unlike SkexGen and Hnc-cad, which require multi-stage training to train multiple transformers, our FlexCAD has only one transformer decoder and achieves end-to-end training in a unified framework. (Specifically, Skexgen has five transformers: topology encoder, geometry encoder, extrusion encoder, sketch decoder and extrusion decoder. Hnc-cad has three transformers: model encoder, code tree generator and model generator.)
(2) As mentioned in Lines 60-65, unlike SkexGen and Hnc-cad, which have limited controllable abilities, our LLM-based FlexCAD achieves controllable CAD generation across all hierarchies, including sketch-extrusion, extrusion, sketch, face, loop, and curve. Besides, as shown in Table 1 and Fig. 5, compared to SkexGen and Hnc-cad, our FlexCAD enhances both generation quality and controllability.
(3) The parameter sizes are not on the same level; the parameter count in the largest transformer used in SkexGen and Hnc-cad is no more than 10M, while that of our smallest LLM is more than 8B. These parameters of LLMs are used to retain pre-trained knowledge, which has been demonstrated to enhance performance (see Table 2).
(4) Unlike SkexGen and Hnc-cad, whose input and output are in numerical format, we transform the CAD representation into a textual format that is easily understood by both LLMs and users.
(5) Relying on pre-trained knowledge, our FlexCAD inherits the instruction-following and in-context learning abilities of LLMs, enabling it to comprehend instructions that never appear during fine-tuning, a capability that small transformers in SkexGen and Hnc-cad cannot achieve. To provide further evidence, we conduct additional experiments. Taking sketch-level generation as an example, we add additional instructions (e.g., 'Ensure that the sketch contains at least one circle') at the end of the original instructions. As a result, the proportion of newly generated sketches containing circles increases from 37% to 89% out of a total of 1000 sketches generated. Furthermore, for in-context learning, when we provide FlexCAD with five exemplars where all the answer sketches contain circles, the proportion of newly generated sketches containing circles increases from 37% to 65%.

Regarding A3: Thank you for the clarification about the performance metrics.

Regarding A4: Thank you for adding the ethics statement.

Regarding A5: I am aware of and have seen research showing that LoRA can outperform full model fine-tuning in certain cases. My question is specific to your experiments: why does LoRA demonstrate this behavior in your findings? Some qualitative analysis and discussion is particularly helpful.

I think the explanations are helpful for the community. Thank you!

Regarding A6: Have you conducted any experiments using few-shot examples with FlexCAD? It would be helpful to see how FlexCAD performs in a few-shot setting and how it compares to zero-shot settings and vanilla model + few-shots.

Q7: Highlight the unique contributions of your approach beyond the application of existing methods.
A7: (1) Representing CAD as concise and structured text is a unique contribution (see Section 3.1). In standard LLM fine-tuning, the text format that can be easily processed by LLM is naturally existing, such as natural language or Python code. However, existing CAD representation cannot be easily understood by LLMs. In prior works, Hnc-cad and SkexGen both use one-hot vectors to represent tokens in CAD sequences. Specifically, SkexGen employs a one-hot vector with a dimension of 4101 to represent a coordinate token, while Hnc-cad adopts the same representation. To address this limitation, we propose representing CAD models as concise and structured texts, making them more accessible for LLMs.
(2) Our hierarchy-aware masking mechanism is another unique contribution (see Section 3.2). In standard LLM fine-tuning, randomly masking a continuous set of tokens is generally utilized. In contrast, we mask a specific hierarchy for each CAD text at each epoch. Additionally, we incorporate hierarchy-level information into the mask token (e.g., [face mask]). As shown in Table 3, these designs help LLMs focus more effectively on generating local hierarchies, thereby improving both generation quality and diversity.

Benefiting from (1) and (2), as mentioned in Lines 60-65, our LLM-based FlexCAD surpasses SkexGen and Hnc-cad, which have limited controllability, by enabling controllable CAD generation across all hierarchies, including CAD, sketch-extrusion, extrusion, sketch, face, loop, and curve.