Targeted control of fast prototyping through domain-specific interface

Is there any way to automatically benchmark the effectiveness of your method?

Thanks for the comment. We would like to clarify that we aim to directly assess the targetedness of each individual design instructions, e.g., "make the spout narrower". This measure is somewhat subjective, as designers must evaluate whether desired changes were implemented while undesired changes were suppressed.

Our work requires step-by-step assessment of each instruction within a sequence leading to a final product, whereas existing datasets typically evaluate only the final result after multiple instructions. This fundamental difference means that while conventional methods can create groundtruth references in advance, our approach relies on designers' on-the-fly instructions, making it impractical to prepare ground truth data (e.g., point clouds) beforehand. This is now discussed in the revised manuscript.

Can you elaborate on how the objective in Eq. 6 is optimized?

Thanks for the question. The optimization of Eq. 6 is achieved through iterations alternating construct expansion and feasibility validation. During the construct expansion phase, the heuristics adjust exploration strategies based on the interface’s state and feedback from prior iterations. When diversity is insufficient (e.g., limited variation in designs), the heuristics broaden exploration breadth by prompting the LLM with directives like “generate diverse handle configurations for teapots”. Conversely, if diversity is high, the heuristics narrow exploration breadth by prompting with constraints. When constructs are overly abstract (e.g., “refine shape”), heuristics increase exploration depth by decomposing the constructs into atomic operations. If constructs are excessively granular, the heuristics reduce exploration depth by encapsulating low-level commands into functions (e.g., “smooth contour”). In the feasibility validation phase, heuristics enforce alignment with the modeling engine’s capabilities by pruning constructs incompatible with CAD engine's constraints. This is now discussed in the revised manuscript.

There are various sampling techniques used in the paper that are not ablated. I would suggest the authors consider the following baselines:...

Thanks for the suggestion. In selecting our interfaces, we explored several alternative methods and evaluated them using intermediate metrics (soundness, completeness, and granularity alignment) that contribute to final performance. The methods we examined are framed as Single-MCMC (single-scale sampling with ranking) and Multi-MCMC (multi-scale sampling with reranking). Specifically:

(i) Single-MCMC aligns with the first suggested baseline for sampling and ranking constructs within a single chain but lacks multi-chain diversity.

(ii) Multi-MCMC mirrors the second suggested baseline for exploring diverse constructs via parallel chains but omits iterative optimization.

(iii) Our full method (Converged MCMC) extends these by integrating multi-scale sampling and iterative refinement.

Results are available here, validating our design choices. This analysis is now included in the revised manuscript.

The related work discussion is very concise.

Due to the space limit, please refer to the response to the first question by reviewer h3kU.

I find the reranking and sampling method in the paper interesting, but (1) very few baselines are considered, and (2) the evaluation is only based on an unreproducible user study.

Thanks for the comment. Regarding the concerns about baselines, we'd like to clarify that our work is not proposing an alternative to LLM-based CAD generators, but rather an interface to improve the performance of those generators. This functions as the first stage of an explicit two-stage approach, with LLM-based CAD generators serving as the second stage. To the best of our knowledge, there are no established baselines for this complete two-stage pipeline. We have incorporated state-of-the-art methods for the second stage.

Concerning reproducibility, we acknowledge that different prompts can yield different results. To address this, we input identical design instructions to all compared pipelines in each design step. While this cannot completely eliminate prompt-dependent variations, it represents our best effort to standardize the experimental protocol for measuring the inherently subjective concept of targetedness, providing a reasonable foundation for comparison. This is now discussed in the revised manuscript.

I suggest the authors provide more documents on their user study, and the final figures generated for each method.

Thanks for the suggestion. The records are available here, which are now included in the revised manuscript.

The authors have not provided a Related Work section in the main paper to present an overview of the studied topic.

Thanks for pointing this out. Now we incorporate a discussion contextualizing our work within the domain of targeted control in fast prototyping.

Fast prototyping is a key process in industrial design, enabling rapid iteration and tangible feedback without the constraints of production-level precision [1,2]. Unlike production-ready modeling carried out by engineers [3], fast prototyping prioritizes exploratory design adjustments, making intuitive and high-level control crucial [4]. Inspired by the concept of targeted therapy in medicine, targeted control in fast prototyping aims to maximize desired modifications while minimizing undesired distortions. This requires an interface that aligns with designers' high-level thinking, focusing on components, structures, and relationships rather than low-level modeling commands [5].

Existing LLM-based CAD generators primarily assist modeling processes rather than fast prototyping, often requiring engineering-level language instructions or sketch instructions. For instance, Query2CAD refines CAD models through natural language queries [6], while Free2CAD translates sketches into CAD commands via sequence-to-sequence learning [7]. Recent advances leverage multi-modal capabilities, such as OpenECAD, which translates 3D design images into structured 2D and 3D commands [8], and CAD-Assistant, which employs tool-augmented vision-language models for general CAD tasks [9]. While these approaches enhance CAD generation and editing, they do not directly address the needs of fast prototyping.

Our proposed interface complements existing LLM-based CAD generators by bridging this gap. Rather than directly alternating CAD generation models, it serves as an auxiliary module, enabling designers to exert precise, high-level control in fast prototyping workflows. This discussion is now included in the revised manuscript.

[1] Burns, M. Automated fabrication: improving productivity in manufacturing. Prentice-Hall, Inc., 1993.

[2] Hallgrimsson, B. Prototyping and modelmaking for product design. Laurenceking, 2012.

[3] Barnhill, R. E. Geometry processing for design and manufacturing. SIAM, 1992.

[4] Uusitalo, S. et al. "clay to play with": Generative ai tools in ux and industrial design practice. In ACM DIS, 2024.

[5] Hannah, G. G. Elements of design: Rowena Reed Kostellow and the structure of visual relationships. Princeton Architectural Press, 2002.

[6] Badagabettu A. et al. Query2cad: Generating cad models using natural language queries. arXiv, 2024.

[7] Li C. et al. Free2cad: Parsing freehand drawings into cad commands. ACM Transactions on Graphics, 2022.

[8] Yuan Z. et al. OpenECAD: An efficient visual language model for editable 3D-CAD design. Computers & Graphics, 2024.

[9] Mallis D. et al.CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers?. arXiv, 2024.

Failure case analysis is also not presented to help the readers to understand the limitations of the proposed method.

Thanks for pointing this out. We have qualitative results as follows. The results are presented through links. Failure cases shape the boundary of our method and success cases .

Failure cases:

(i) For qualitative spatial constraints like "vertical" or "parallel", LLMs sometimes fail to map them correctly to precise positions and orientations due to their weak spatial reasoning.

failure case 1

(ii) For certain abstract and complex instructions---such as operations involving Bezier curves---current methods sometimes fail to capture the correct approach.

failure case 2

Success cases:

(i) Our-Int effectively maintains and guides the transformation of basic shapes and their modifications, mapping abstract designer instructions to concrete shape changes.

success case 1

(ii) Our-Int can automatically infer a commonsense spatial distribution of components when designer instructions lack explicit spatial constraints.

success case 2

(iii) Our-Int translates modifications at a finer granularity, identifying which component and which attributes are affected.

success case 3

These analyses are now included in the revised manuscript.

The benchmarking metrics seemed like proxies to what would really matter?

Thanks for the question. Fast prototyping allowing designers to explore brainstormed ideas without elaborating their instructions into modeling engineers' language. Our approach is explicitly two-stage, with our proposed interface serving as the first stage and LLM-based CAD generators as the second. Our metrics were selected to measure specific aspects of the process. Specifically, we use rendering consistency to measure how well the interface captures designers' intentions, essentially evaluating if desired elements appear and undesired ones are suppressed. Meanwhile, information clarity metrics quantify how effectively information transfers between designers' high-level language and modeling engineers' fine-grained requirements. We have clarified the rationale for these measurement approaches in the revised manuscript.

The work is heavily focused on a specific application domain with unclear generalizability to other spaces.

Thanks for the comment. Our main motivation is that fast prototyping allowing designers to explore brainstormed ideas without elaborating their instructions into modeling engineers' language, which in general investigates into the construction of a interface to bridge designers' language with modeling engineers' language. In broader context, our work provides a proof-of-concept for improving communication between domain-specific executors (Part-B) and stakeholders (Part-A). We totally agree that further investigation on generalization would be a promising direction. This is now discussed in the revised manuscript.

Limited discussion of computational efficiency and scalability; and lack of case studies useful in application papers.

Would like to see more qualitative examples to concretize a lot of the failures

Thanks for pointing these out. We have validated the computational efficiency by calculating the cost of targeted control, on the scale of eight domains.

We use OpenAI’s GPT-4o API for domain adaptation and real-time execution. Designing a domain-specific interface costs about \$10 per domain, running 1,000 iterations on a MacBook with an M2 chip to ensure convergence. During prototyping, costs remain low: executing the pipeline for targeted control costs \\$0.10 for translation and \$0.30 per ten refinement iterations for CAD generation.

In the revised manuscript, we add qualitative results on failure cases and success cases. Due to the space limit, please refer to the response to the second question by reviewer h3kU.

However, claims about the interface reducing cognitive load for designers aren't directly measured.

The paper claims their approach is generalizable across domains, but only tests on eight product categories which seem relatively similar.

Thanks for pointing out these confusions. We've revised our wording to emphasize the concrete benefit of "reducing designers' manual efforts" in fast prototyping. Regarding generalizability, we acknowledge our testing was limited to physical consumer products, a key reference for industrial design. To better reflect our scope, we've adjusted "across domains" to "across categories" in the revised manuscript.

The human study design with 50 participants is adequate, but I question whether 10 iterations per task is sufficient to capture the full design process.

Thanks for the question. Fast prototyping differs from full design. While full design requires master models for mass production, fast prototyping serves as an exploration for primary design ideas.

Design studies often use two setups: (i) counting iterations to reach certain resultsor (ii) evaluating improvements within a fixed iteration count. We adopt the latter to assess the improvements of each iteration with the same instruction, therefore relatively invariant to the number of iterations.

The current choice of 10 is informed by discussions with professional industrial designers to capture the typical lifecycle of fast prototyping, and would be further investigated in further study. This is now discussed in the revised manuscript.

This paper could benefit from discussing and/or connecting to such work!

Thanks for the insightful comment. Our automated interface design shares conceptual similarities with program synthesis in achieving hierarchical abstraction through iterative sampling and refinement. However, two key distinctions exist:

(i) Knowledge representation: program synthesis relies on structured knowledge, using exemplar DSL programs to generate higher-level libraries. In contrast, our approach samples from unstructured commonsense knowledge bases (e.g., LLMs) and organizes knowledge into a DSL;

(ii) Machine learning paradigm: program synthesis follows a supervised approach, leveraging I/O pairs with task specifications, whereas our method is unsupervised, emerging from commonsense knowledge bases.

This is now discussed in the revised manuscript.

I'd happy to raise my score if the authors could expand on what makes the proposed method work better than the other baselines, especially why it's able to beat LLM based approach by significant margin.

It's not immediately clear to me why this method is the obvious approach over all other possibilities.

Thanks for the comment. Our work is not an alternative to LLM-based CAD generators, but rather an interface to improve the performance of LLM-based CAD generators by bridging designers' language with modeling engineers' language. The fundamental challenge is that while modeling languages are hierarchically documented, designers' language is unstandardized in the wild. This necessitates a systematic representation of concepts described in designers' language---from product categories to structures, components, attributes, and operations. This requirement aligns with word learning in cognitive science, where humans learn systems of interrelated concepts rather than isolated terms.

Drawing inspiration from cognitive development, we mirror how people learn concept networks by sampling from the environment and organizing these samples into hierarchical structures through DPMM. This spectral clustering approach captures multi-level attributes rather than clustering based on overall similarity [1]. In our approach, we substitute environmental sampling with LLM-generated samples, as LLMs are recognized repositories of commonsense knowledge [2]. We then apply DPMM to cluster these samples into a hierarchy of structures, components, attributes, and operations. This systematic representation allows natural instructions from designers to be decomposed into fine-grained elements that align with modeling language constructs, enabling targeted control in fast-prototyping.

Therefore, adding our interface to alternative approaches---such as forcing LLMs to directly translate between languages or inputting designers' instructions directly into LLM-based CAD generators, should intuitively benefit the performance.

[1] Tenenbaum et al. How to grow a mind: Statistics, structure, and abstraction. Science, 2011.

[2] Yildirim et al. From task structures to world models: what do LLMs know? Trends in Cognitive Sciences, 2024.

Lacks some analysis (ideally quantitative) and ablations on what advantage the proposed method has over the other baselines other than the final performance.

Thanks for the comment. In the revised manuscript, we establish a more comprehensive evaluation beyond the previous final performance, to guarantee the three major qualities of the LLM interface:

(i) Soundness: ensuring all language constructs used by designers can be implemented in the modeling process;

(ii) Completeness: ensuring all modeling process constructs are represented in designers' language;

(iii) Granularity alignment: ensuring proper overlap between the finest-grained constructs in designers' language and the coarsest-grained constructs in the modeling process.

Quantitative results for these metrics are available here, demonstrating the advancement of our interface. This superior systematic representation directly contributes to the improved final performance when integrated with an LLM for CAD. This analysis is now included in the revised manuscript.

The proposed method only compared to two other baselines, which seems a little slim to me. For example, the LLM prompting baselines could have very different performance if you prompt it differently.

Thanks for the comment. We would like to first clarify that our work is not proposing an alternative to LLM-based CAD generators, but rather an interface to improve the performance of LLM generators.

Our approach is explicitly two-stage, with our interface serving as the first stage and LLMs for CAD as the second. To our knowledge, there are no established state-of-the-art baselines for direct comparison of the entire two-stage pipeline. Comprehensive baselines exist for the second stage, so we have adopted the current SOTA to evaluate the first stage, s.t. our proposed interface.

We fully acknowledge the reviewer's point about prompt sensitivity. Actually, this is also one of our major concerns. To address it, we designed our evaluation protocol to minimize the influence of varying prompts by inputting the same prompt to all compared pipelines for each design step. While this approach cannot completely eliminate prompt variability, it offers a straightforward comparison by controlling this variable as much as possible. Investigating the influence of different prompts represents a promising direction and is now discussed in the revised manuscript.