Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models

Dear reviewer,

Thank you for the valuable comments. We are pleased to know that you are happy with our evaluation details including the qualitative results and user study. We hope the following clarifications address your concerns.

More References

Thanks for pointing out these relevant works. Below is our discussion to them, and we will add them to our updated manuscript.

1. CADLLM [1]: It is a work on unconditional CAD generation. We referenced similar works such as DeepCAD [3] and SkexGen [4]. Our work is different to them by focusing on a conditional task that deals with text-guided CAD generation.

2. Vitruvion [2]: It focuses on generating 2D sketches from hand drawings. Our work, in contrast, does not focus on CAD reconstructions that takes visual inputs, but more abstract text inputs instead. We cited similar works to Vitruvion but on 3D reconstructions [5] [6].

Baseline Methods

As all reviewers acknowledge, the text-to-CAD task is relatively new - we have highlighted this in the introduction. As a result, there are only a few open-source baseline methods that are directly comparable to our approach. We sincerely thank you for your understanding on this matter.

You are welcomed to have a look at our response to reviewer 3nTM, in which we reported a comparison between our method with an adaption of Skexgen for text-based generation. We hope this confirms that our framework demonstrates superior capability among all existing works.

Details of User Study

Our user evaluation was conducted as a blind study. Participants were asked to rank the rendered CAD objects (without knowing their origin) based on 1. their alignment to the given textual descriptions, and 2. the quality of the CAD models. The users had college-level (or higher) education but we did not restrict their majors. Since our evaluation focuses solely on the visual appearance of the generated models, we believe this is appropriate.

We have included an description of our human evaluation setup in Appendix C.2. However, we realize that this section may not be entirely clear and could be difficult to follow. We will update and clarify this section in the camera-ready version to provide more detailed instructions and improve overall readability.

DPO: Static vs. On-policy

Yes, our DPO implementation is not entirely "static". By splitting DPOs into smaller iterations and making it iterate with the sequential learning, the model being optimized is much closer to the policy model generating the sample pairs. It is a step towards an "on-policy" DPO.

[1] Seff et al., Vitruvion: A Generative Model of Parametric CAD Sketches, ICLR 2022.

[2] Wu et al., CAD-LLM: Large Language Model for CAD Generation, NeurIPS 2023 Workshop.

[3] Wu et al., DeepCAD: A Deep Generative Network for Computer-Aided Design Models, ICCV 2021.

[4] Xu et al., SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks, ICML 2022.

[5] Ma et al., Draw Step by Step: Reconstructing CAD Construction Sequences from Point Clouds via Multimodal Diffusion. CVPR 2024.

[6] Khan et al., CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention. CVPR 2024.

Dear reviewer,

Thank you for your thoughtful comments and for recognizing our contributions, particularly in the design approach and experimental setup. We appreciate the opportunity to clarify the points you raised.

Claim Could be a Bit More Precise

We realize this problem and apologize for that. We will tone down relevant parts such as "significant performance improvements" in the updated version. The limitation about complex models (demonstrated in figure 11) will also be brought to the main body.

Could Different VLMs Give Different Scores?

First, VLM's scores depend on their base capabilities. We will add relevant discussions in the appendix. The followings are some discoveries we have made:

1. LLaVA (non One-Vison) cannot give correct scores. It tends to give very high scores (e.g., 9 or 10) or very low scores (e.g., 2 or 3) randomly.
2. How LLaVA-One-Vision (LLaVA-OV) scores rendered CAD objects is quite similar to how GPT-4o does.

Second, the scores are partially sensitive to the formation of the evaluation prompts. Specifically, models randomly over-focus on aspects such as texture and colors of the rendered CAD objects, and score some rendered CAD objects lower than others based on that factors, which leads to inconsistency. Consequently, we instruct the VLMs not to focus on these aspects (details can be found in Line 752 and 800). After this modification, the scoring seems stable with prompts.

Comparisons with Text2CAD

We have the results where Text2CAD uses the same prompts as ours (i.e. the solid one). It is the comparison between row 3 and 4 in Table 3. The result we selected for presentation in Table 1 is the setup favors Text2CAD the most, which does not affect the fairness of our comparison. We will add the results with the same prompts to the main sections to make the comparison more comprehensive.

Dear reviewer,

We sincerely appreciate your thoughtful comments and the opportunity to clarify the concerns raised.

How Well Captions Describe CAD Models and Model Accomodates to Real Users

During development, we considered similar concerns and would like to share our findings:

• VLM-generated captions are effective for moderately detailed descriptions, such as structure, topology, and key component relationships. However, they struggle with precise numerical specifications, such as exact dimensions (e.g., "13.5 × 4.3 × 2"). Since designing is an iterative process, moderately detailed descriptions still provide substantial guidance to CAD modeling.

• We provided guidelines to human users and empirically find that they quickly learned to interact with our model by following them. We provide more detail of the guideline and an example at the end of this response.

Handling Numerical Parameters

Currently, users should manually adjust numerical parameters as a post-processing step. In a pratical designing scenario, they normally create rough designs first and refine the numerical details iteratively. We believe our model provides a sufficient response automating the former step, and optimistically anticipate future work that supports the latter, aiming to fully automate the design process.

More discussion for HNC-CAD

We have discussed HNC-CAD in the introduction and related works as an unconditional CAD generation method, and cited it in our paper. Following your suggestion, we will add more discussion about HNC-CAD. Specifically,

1. HNC-CAD is also an autoregressive model. However, it trains a transformer VQ-VAE on codebook representations on sequential signals, while ours leverages a pretrained LLM (decoder only) on stringified representations on both sequential and visual signals.
2. The auto-completion task in HNC generates a random completion. It can be used jointly with the text-based CAD generation method. By providing a CAD sequence prefix and text guidence to the framework, we achieve a text-guided completion.

User's Ability to Modify Generations with Additional Prompts

We would put it as a promising future direction called text-based editing. It relies on different model capabilities from text-based generation. For instance, model may need to learn how to localize the place for edition based on additional prompts and make changes to that specific location. This requires not only text understanding but also CAD understanding.

Generation Diversity Regarding the Same Prompt

We did include exploration on this part! Please refer to Figure 10 in Page 18. It corresponds to Appendix C.7.

[1] Khan et al., Text2cad: Generating sequential cad models from beginner-to-expert level text prompts, NeurIPS 2024.

[2] Xu et al., SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks, ICML 2022.

(Appendix) User Guidelines for Prompting

A good prompt follows a structured description: (1) shape overview, (2) shape details, and (3) shape applications. Given the varying shape complexity, we encourage but do not enforce describing item (2) and (3). Below is an example caption retrieved from Figure 8, Row 2, Item 3, demonstrating this approach:

[Shape Overview] The 3D shape consists of a large, flat rectangular slab with two evenly spaced, identical cylindrical protrusions extending vertically from its surface. [Shape Details (Optional)] The slab provides a stable base with significant length and width compared to its thin height, while the cylinders are relatively short and have small diameters. [Shape Applications (Optional)]The overall design is symmetrical and balanced, potentially serving as a mounting base or connector.

We believe that such structured descriptions are clearly formatted and easy for users to interpret. If the reviewer finds it necessary, we are happy to include detailed instructions in the camera-ready version of the paper or the code repository upon release.

Dear reviewer,

Thank you for your feedback. We appreciate the opportunity to clarify your concerns.

SL(no VF) & SL-VF(pro)

We are unsure what SL-VF(pro) refers to. We presume you are mentioning SF-VF(rpo). If so, we would like to point out that this is not our main method, but an ablation using existing techniques to regularize and stabilize DPO (Line 410). We included it to highlight the effectiveness of our iterative visual-sequential training (line 413-426).

In Table 2, our main method is SL-VFSL(5) in the last row, where a considerable improvement on VLM score can be observed.

Evaluation Criteria

We regret to point this out, but there is a factual misunderstanding in the statement "Evaluations for whether or not the generation truly follows the text description is not obvious."

1. In our human evaluation, the textual description are shown alongside the generation results, and annotators were asked to evaluate the alignment between the CAD model and text (Line 808). Thus, the human rankings directly reflect instruction following.

2. Second, our VLM score also supports this evaluation by design (Line 317). Reviewer X4me calls this "a reasonable score", which supports the soundness of our metrics.

We would appreciate it if you could reconsider this claim after reviewing the referenced lines.

Technical Contribution

We would like to highlight that the motivation behind introducing DPO is itself a contribution. Observing limitations of sequence learning, we proposed using visual signals to adjust model preference: an approach has never been previously explored. Moreover, as the rendering process is not differentiable, directly encoding visual signals and backpropagating them is infeasible. Consequently, we propose DPO as a solution.

Furthermore, integrating DPO into our setup is not a trivial application of existing techniques. Table 2 (Rows 3, 4) shows that directly applying DPO or standard stabilizing algorithms (RPO) does not yield strong results. We hypothesize that this is due to the cross-modality nature of our task. A significant portion of our work focused on overcoming challenges though SL-VFSL training.

Additionally, we would like to highlight how other reviewers acknowledge our contributions. Reviewer jpmw calls our handling of the non-differentiable rendering process "clever and appropriate," and Reviewer X4me refers to our approach as "interesting" and a "nice way" of tackling the problem.

Simple Shapes in Figures

Figure 8, 9 includes CAD generations with more complex shapes and textual descriptions specifying CAD structure and topology details. We hope these results satisfy you.

The reason Figure 5 contains 'simple' shapes is that we had to choose samples from the intersection of valid generations across models. We are happy to move some instances in Figure 7, 8 to the main manuscript at your wish.

Data Size and Captioning

1. While we acknowledge that 20k human-annotated samples could be further expanded, we choose this setting for some reasons:

• First, we annotated data and trained models iteratively. At 20k samples, our method already outperformed baselines (Table 1, main experiment).
• Second, the cost of annotating 20k samples is affordable for research groups of any scale. By keeping the dataset size reasonable, we aim to ensure the reproducibility of our results and provide meaningful insights to the broader community.

2. VLM itself is reliable. The CADFusion variant trained on only VLM data with sequential signals (Table 4, row 2) outperforms baselines (Table 1, row 1, 2), demonstrating improvement.

COV/MMD/JSD Scores

The Text2CAD paper does not report these metrics. We attempted to compute them using our evaluation code but encountered challenges when applying it to Text2CAD's code and results. However, we have reported other metrics that effectively reflect the our method's efficacy.

Comparing to DeepCAD/SkexGen/HNC-CAD

We did not compare against them as none take text as input or claim text-based capabilities. Adapting their frameworks for textual input requires modifications that constitute a new research project rather than a baseline.

Although this requirement is doubtful, we implemented a text-guided SkexGen variant by integrating a text encoder with SkexGen’s decoder for Text-to-CAD generation. Training on the same setups, our results are:

CADFusion outperforms SkexGen in multiple metrics, reinforcing its effectiveness.

Collisions

The term "collisions" refers to part intersections within a CAD model. VLM does not explicitly prevent collisions but evaluates their presence. This scoring mechanism benefits the subsequent visual-feedback stage, where we empirically observed an improvement in collision avoidance.