GenBen:A Genarative Benchmark for LLM-Aided Design

1. I tried to access the anonymous link provided by the authors for the benchmark, but it seems to be unavailable. Could the authors provide a working link during the rebuttal phase? This would allow me to evaluate the quality of the dataset

2. Another minor concern: Although I really appreciate the multimodal and contamination-free dataset, I personally feel that this paper might be better suited for a hardware-focused conference.

The paper is not much relevant to the ICLR community, and does not provide a strong mathematical foundation. A more suitable venue for this paper could be ICLAD Conference or some Design Automation Conference like DAC or ICCAD.

1. Data Imbalance in Test Categories: Table 3 presents disparities across task categories (e.g., 99 Design tasks vs. 57 Debug tasks). While task diversity is commendable, the authors should clarify why such an imbalance exists and discuss potential implications for model performance assessment, especially if specific categories are overrepresented. An explanation of how this might affect the reliability of the benchmark would strengthen the dataset's construction rationale (Section 3.3.3).
2. Model Selection Motivation: Section 4.1 introduces nine models across five families but lacks justification for these particular choices. Clarifying why specific models (e.g., GPT, Claude, LLaMA, QWEN, and GLM families) were included could strengthen the argument, especially if these models are particularly relevant for LAD tasks. For instance, what LAD-specific capabilities do these models bring? Including performance characteristics in multimodal tasks or LAD-related features in each model would better contextualize the selection.
3. Difficulty Tiering: The paper divides tasks into three difficulty levels (L1 to L3) with proper descriptions (Table 2). However, detailing how each tier’s tasks correspond to LAD challenges (e.g., debugging complexity or resource optimization) would add valuable context. The authors could enhance Appendix A.4 by providing specific examples that illustrate the distinctions between L1, L2, and L3, particularly within each task type (knowledge, design, debugging). Quantitative data on average model performance per tier might also reveal trends that could guide future benchmarking improvements.
4. Detailed Case Study on Perturbation: The perturbation strategy is briefly explained in Section 3.4, with an overview of surface and semantic techniques. However, a specific example, especially for semantic perturbation, would make this section clearer. Adding a detailed case study would provide a more straightforward demonstration of how the data were perturbed, also for semantic perturbation, due to the randomness, is there a manageable gap of difficulty to change? For instance, showing a task before and after both types of perturbation would help readers understand the differences in difficulty introduced by each type. Furthermore, since semantic perturbation could change task difficulty, addressing how fairness is maintained would strengthen the case for using this approach.
5. Elaboration on Workflow: The "Dynamic Test Kit" component in Figure 2 and Section 3.2 would benefit from additional detail. For instance, does the Dynamic Test Kit contain tools that adjust task difficulty, or is it limited to generating variants of test inputs? Furthermore, clarifying the connection between the scoring system described in Section 4.1 and the metrics reported in Table 5 could make the scoring more interpretable. Specifically, explaining if the pass rate is calculated per task or per difficulty level would aid in understanding the benchmark's scoring logic.
6. Experiment Results Interpretation: Section 4.2 and Figures 4-12 illustrate experiment results, but a breakdown of model performance by metric (e.g., Syntax, Synthesizability, Function) would be helpful. Discussing why Syntax scores are generally higher while Function scores lag behind could provide insight into specific model weaknesses. Additionally, including a statistical analysis (e.g., variance in pass rates across difficulty levels) would offer a clearer picture of each model’s strengths and limitations.
7. Supporting Literature for Multimodal Justification: Section 3.5 could be enhanced by citing specific LAD scenarios or prior research that demonstrate the need for multimodal tasks (e.g., combining schematics with HDL code) to support the claim “This feature is particularly important because real-world design processes often require the integration of various forms of data, such as textual specifications, diagrams, and architectural schematics”. Including practical examples from hardware design projects that require multimodal support would illustrate the real-world relevance of GenBen’s multimodal component. This would also substantiate the statement that real-world LAD tasks are inherently multimodal.
8. Future Work and Limitations: The paper would benefit from a Limitations and Future Work section, discussing GenBen's adaptability to emerging LAD requirements (e.g., integration with more complex EDA tools or AI-driven hardware design) could provide a roadmap for GenBen’s evolution. Additionally, mentioning scalability limitations (e.g., handling larger designs or more complex verification tasks) would provide a balanced view of the benchmark’s potential applications and restrictions.
9. Evaluation Metric Granularity: While Table 5 provides an overview of evaluation metrics, further clarification on the weighting or relative importance of these metrics (e.g., Syntax vs. Synthesizability vs. QoR metrics) could improve transparency. Explaining if QoR metrics are prioritized over syntactic correctness for certain tasks, such as those targeting manufacturable designs, would give a better understanding of how the benchmark defines "success." Additionally, addressing any trade-offs between these metrics, especially in cases where models might excel in one metric while underperforming in others, would be helpful (e.g., does a high Syntax score compensate for lower Synthesizability?).
10. Benchmark Customizability: GenBen is described as an end-to-end LAD benchmark. A brief mention of how the benchmark might be tailored for different LAD applications (e.g., specialized RTL verification or energy optimization) in Section 3.2 would clarify the benchmark’s flexibility.
11. Clearer Description of Multimodal Task Processing: Although Section 3.5 explains the importance of multimodal tasks, more detail on how GenBen processes or evaluates multimodal inputs would be beneficial. For instance, specifying how different data forms (e.g., schematics vs. HDL code) are presented to models and if specific evaluation criteria are adapted to multimodal tasks could provide clarity. Addressing whether the models are expected to interpret these inputs sequentially or concurrently, and if multimodal inputs impact the difficulty tiering, would improve understanding of the benchmark's multimodal handling.
12. Clarification of Pass@5 Evaluation and Scoring: The pass@5 scoring strategy in Section 4.1 is introduced without much context on its relevance to LAD tasks. Explaining why pass@5 is chosen over other scoring metrics (e.g., pass@10, exact match) and how it aligns with real-world LAD evaluation (e.g., tolerance for minor errors in preliminary passes) would strengthen its justification. Additionally, describing if scoring varies by task difficulty or complexity would make the scoring methodology clearer.

1. The dynamic perturbation strategy aims to prevent memorization but may inadvertently introduce ambiguity into test cases. This can create inconsistencies in evaluating LLM performance, especially if slight changes in prompt phrasing lead to variations in model responses. The reliance on superficial perturbations (surface-level changes) may not effectively challenge models in understanding complex circuit design concepts.
2. The paper identifies issues with synthesizability due to non-IEEE-compliant code in pre-training datasets, but it lacks concrete methods for systematically identifying and filtering these cases. This limitation could lead to significant variability in QoR metrics, particularly for synthesis tools that adhere strictly to IEEE standards, reducing the benchmark’s reliability.
3. Timing issues are briefly discussed, but the GenBen benchmark does not appear to account for complex timing closure tasks, such as handling Total Negative Slack (TNS) and Worst Negative Slack (WNS) for high-frequency designs. The benchmark could fall short in evaluating LLMs’ ability to generate designs that meet stringent timing requirements, which is critical for high-performance applications.
4. Although the benchmark includes debugging tests, it may not fully capture the complexity of real-world hardware debugging, particularly for stateful designs or asynchronous circuits. The current debugging scope seems limited to relatively straightforward syntax and functional errors, without addressing state-based errors, race conditions, or setup/hold timing violations that are common in complex designs.
5. While the inclusion of multimodal tests is innovative, the integration of textual and visual data (e.g., diagrams) lacks specific detail on how visual data is processed or scored. This lack of clarity may lead to ambiguous scoring for models that perform differently across multimodal and text-based tasks, making it challenging to standardize assessments.
6. The QoR metrics focus on synthesizability, power, area, and timing but do not assess finer aspects like pipeline balancing, parallelism optimizations, or state-machine efficiency. These factors are crucial for high-performance designs and should be part of a rigorous hardware benchmark that targets comprehensive design quality.

The writing quality of this paper is subpar, primarily due to the lack of practical examples. The authors should include more concrete examples to clarify concepts such as "adding some perturbations." Additionally, Table 4 lacks substantive information, and it would be more effective if it included illustrative examples. Furthermore, Section 3.3.1 provides only a superficial overview of the end-to-end verification flow, lacking sufficient detail and making the article less informative.

Moreover, it is unclear how the perturbation process ensures the creation of sound problems, avoiding ambiguity or unclear meanings. The authors should address this aspect to reinforce the reliability of their approach.

Some types on writing: Section 3.2 "GenBen then generates test tests from the test dataset D using scripts", test typo. Figure 2 and Figure 4-12 are very hard to read. Use contrasting colors then patterns to discern between different metrics. line 213: test tests from