The Graph's Apprentice: Teaching an LLM Low-Level Knowledge for Circuit Quality Estimation

We will answer together the concerns of the weakness section, and those of question 1, as they are closely linked.

Small vs large designs and Question 1: This is an excellent point. To clarify our goals, the use-case we have in mind is precisely large circuits for which computing the netlist is too slow to be used to provide near-immediate feedback to a circuit designer. It is also why we restricted ourselves to comparisons with ASTs, since they are some of the few alternatives to raw Verilog that can be obtained fast enough to fit this use-case. We hope this clears any misunderstandings.

Regarding LLM context length, we used the Verilog LLM with the largest context length that is publicly available (16k tokens). Most importantly, in a real practical application, we would expect that a company interested in implementing the method could fine-tune a larger model if desired. Indeed, the latest LLMs now boast context lengths so large that context length does not become a constraint anymore: as extreme examples, the huggingface model repository now stores an increasing number of large context open-source LLMs that can handle above 1M tokens (e.g. Llama-3-8B-Instruct-Gradient-4194k by gradient.ai), which at 30 tokens/line of code would represent over 33k lines of code. Although such models has not been fine-tuned on Verilog code, there would not be any obstacle in principle for an interested company to do so, now that there is a use-case for it.

Question 2: This is an excellent question. In principal, either AIG or LUT could be used to train the teacher model. In our initial experiments conducted on a subset of simpler circuits, we found that the GNN trained on LUT outperforms the GNN trained on AIG by a small margin. We suspect this was the case since graph neural networks perform best when graphs are small and node features are rich, and LUT graphs fit closer to this ideal than AIGs. In addition, the LUT representation of the circuit has more resemblance to the final Netlist. From a practical perspective, the LUT is more compact in representation , with up to 4x less nodes than the AIG. Thus, it is easier to fit large circuits into the memory with the LUT representation.

Proofreading: We went over the paper thoroughly and fixed every typo we could find. Thanks for pointing it out.

As the discussion period nears its end, we hope that we have effectively addressed and resolved your concerns. If so, we would be grateful if you could reconsider your rating of our paper.

Thank you very much for your review. We will address each point in turn:

Layer unfreezing: This is an excellent suggestion! Unfortunately, we do not currently possess the computational resources to undertake this experiment, but it could be an interesting way to potentially push the performance of our approach even further.

Different LLMs: Absolutely, we agree this could be a concern! To demonstrate the impact of KD with respect to the base LLM, we employ CodeV-DeepSeek and CodeV-CodeQwen, which utilize deepseek-coder-6.7b and CodeQwen1.5-7B-Chat as the base models. These two models are two variants of CodeV, which have undergone the same Verilog fine-tuning procedure as CodeV-codeLlama. These two models are two variants of the standard CodeV model based on CodeLlama-7b-Instruct, and were trained with the same procedure. As can be seen in the table in Appendix A, the resulting performance only varied marginally after switching to those different LLMs.

Proofreading: We made a new pass and corrected everything.

Question 1: We would like to clarify that by stating "knowledge distillation has almost no effect on the AST-GNN model", we are referring to the performance of the model. As shown in the results, the performance of AST-GNN + KD does not improve significantly over AST-GNN. Furthermore, the tSNE figure is a bit misleading due the fact that the latent representation space is projected to 2D from 512D. Still, the t-SNE shows that while AST-GNN with KD aligns with the teacher model, there are some mixups in the points visible. We have modified the text to make this point more clear.

Question 2: We have added this result to Appendix B.

Question 3: It is true that the current choice of model takes a context of 16k tokens, which corresponds to about 500 lines of Verilog code. However, there has been a steady stream of new models coming out with increasingly larger contexts - for example, the huggingface model repository now stores an increasing number of large context open-source LLMs that can handle above 1M tokens (e.g. Llama-3-8B-Instruct-Gradient-4194k by gradient.ai). Thus, although the current Verilog LLMs are smaller context, there is no obstacle to fine-tuning, in principle, larger LLMs to fit the use case would the need be.

Question 4: Currently, the model is effectively trained to produce QoR estimates conditional on a fixed recipe, because of the way the labels were produced during data generation (i.e. always using the same fixed recipe). OpenROAD provides two optimization recipes for the logic synthesis process: "ABC_AREA=1" for area optimization and "ABC_SPEED=1" for timing optimization. The experiments in the paper adopted area optimization recipe. To test the robustness of VeriDistill under a different synthesis setting, we re-run synthesis for speed optimization (ABC_SPEED=1 for OpenROAD hyperparameter setting). We train and evaluate all the methods under the new setup. As seen in the table below, all our claims in the paper stand well. We have added this table to Appendix D.

The performance of different Verilog models on the test dataset under the speed optimization setting:

We should emphasize however that this is a minor point that can easily be modified if this is a concern. For example, would the practitioner be interested in the delay and area obtainable under the optimal recipe, one could imagine during label generation to run logic synthesis with 10 different recipes, and keeping the smallest obtained delay and area as labels. Our model trained on such labels would then produce an estimate of the best area and delay obtainable for a given circuit. Or, if one wanted flexibility in prediction, one could imagine adding the parameters that control the synthesis recipe as arguments of the model, so that the model would produce estimates of area and delay conditional on specific choices of synthesis parameters. We did not consider such use-cases in our work to not overburden the text, but they would be easily accommodated if the need arose.

Thank you very much for your review, and we particularly appreciate hearing a practitioner agree with the real-life potential of such a model! Here are answers to your comments.

Alternative LLMs: To demonstrate performance of VeriDistill with different LLMs, we employ CodeV-DeepSeek and CodeV-CodeQwen, which utilize deepseek-coder-6.7b and CodeQwen1.5-7B-Chat as the base models. These two models are two variants of the standard CodeV model based on CodeLlama-7b-Instruct, and were trained with the same procedure. As can be seen from the table, the final performance is only marginally different from using different Verilog LLMs. We have included this result in the appendix.

The performance of VeriDistill with different Large Language Models:

Training resources: Because the LLM is kept frozen during training, it was possible to save training time by extracting the forward pass through the LLM only once and saving it. We performed this phase on a machine with 8 Nvidia V100 GPUs with 32GB of memory and 32 Intel(R) Xeon(R) Gold 6140 CPUs. Once the hidden state was saved, we then trained each model following the procedure detailed in the paper on the same machine using a single V100 GPU with 1024 minibatch sizes. The training times for each model are summarized in the following table.

Training times for the various models:

We have added this description and the table in the appendix.

Question 1: This is a very interesting question, and we had attempted to investigate this by looking into the Verilog code where VeriDistill made the largest errors. In general, we could not find particular patterns in the code for the errors. We noticed however that the model tended to do worse on instances with large memories, e.g. one example was the declaration of a memory with 4096 16-bit elements "reg [15:0]mem[0:4095]" in a SRAM design. It is possible that Verilog LLM was not well trained to comprehend the impact of large memory size on circuit delay, leading to underpredict the delay. If this is true, then addition of more such cases in pre-training data would likely enhance the performance of VeriDistill on these cases.

Question 2: Please see answer to "Training resources" above.

Question 3: Our training set consists of Verilog instances of various complexity. The simplest instances contain various implementations of common building blocks in Verilog (such as adders, subtractors, different logical gates, flip-flops, counters, etc.). Given that VeriDistill achieves a high accuracy on these simpler instances (as shown in figure 4), we can attest that VeriDistill can easily handle basic functionalities specified in Verilog with near-zero error.

Question 4: Yes, absolutely, we do not see any obstacles in principle. However, the recent HDL LLMs are mostly trained with Verilog data parsed from Github and textbooks. While VHDL is more popular in Europe, there is not as much VHDL data available in the web, which limits the development of LLMs for VHDL designs. If the VHDL data is not an issue, possibly for some universities or large corporations, we believe that the proposed approach could handle VHDL just as well.

Thank you for your detailed review and the insightful questions you posed. Could you please provide additional feedback if there are any aspects we have not fully addressed? Thank you for your time and effort in helping us enhance our work.

We thank the reviewer for the thorough comments.

Weakness 1: Following your comments, we rewrote and reorganized the methodology section. The model is now clearly differentiated from the training strategy, which is introduced in two steps. We have also redone Figures 1 and 2, to better match the text description. We hope this addresses the issues.

Weaknesses 2--3: We rewrote Section 4.1 accordingly.

Weakness 4: We were not able to run MasterRTL (Fang et al. 2023/2024b) as a baseline because its feature extraction step relies on a path-level slack prediction model, which is not provided in the opensourced repository. We did nonetheless try to compare to the SOG representation proposed in their work, which is an intermediate representation between RTL and netlist. However, the SOG extraction process, which relies on Yosys, PyVerilog and a Verilog-to-graph parser, failed to produce SOGs for a large proportion (27%) of our dataset, comprised almost entirely of the designs with large delays in the dataset. This emphasizes the complexity of relying on third-party tools for pre-processing Verilog files. On the other hand, our proposed VeriDistill takes the design Verilog files as inputs without any pre-processing requirement, and is able to provide design quality estimation instantaneously during inference. Moreover, the rich representation obtained from LLMs can be fused effectively with teacher GNN embedding, which is not the case for the GNN-based representation as shown in the AST-KD setting.

Weakness 5: We had attempted to investigate the outliers by looking into the Verilog code where VeriDistill made the largest errors. In general, we could not find particular patterns in the code for the errors. We noticed however that the model tended to do worse on instances with large memories, e.g. one example was the declaration of a memory with 4096 16-bit elements "reg [15:0]mem[0:4095]" in a SRAM design. It is possible that Verilog LLM was not well trained to comprehend the impact of large memory size on circuit delay, leading to underpredict the delay. If this is true, then addition of more such cases in pre-training data would likely enhance the performance of VeriDistill on these cases.

In addition, we would like to point out that even though there are sparse outlier data points in all plots in Fig 4, VeriDistill has much fewer outlier points and performs generally better on these outlier data points. To illustrate this, we outline the distribution of absolute errors across all methods in (full results can be found in Appendix 5). While the baselines AST-XGBoost, AST-GNN, AST-GNN w/ KD, and CodeV + Decoder have 26, 33, 30, 47 points with absolute error larger than 0.5, respectively, VeriDistill only has 17 such points.

Distribution of absolute percentage errors for the (log) delay prediction task. Each cell specifies the number of points with the absolute percentage error falling in the range specified by the column:

Weakness 6: We thank the reviewer for pointing out the error in our explanation for the t-SNE plot. We have made changes to the text to address the errors and clarify our explanation.

Typographical and grammatical errors: We made a new pass and corrected everything.

We wanted to follow up on our previous response and the improvements we've made based on your insightful review. As the rebuttal period nears its end, we would like to ensure that we have addressed all your concerns. If so, we would be grateful if you could reconsider your rating of our paper.

Thank you for your time and effort in helping us enhance our work.

We look forward to your response.