DynamicRTL: RTL Representation Learning for Dynamic Circuit Behavior

Although learning representations on CDFGs of programming languages is common, to the best of our knowledge, there is no open-source work capable of building complete operator-level CDFGs of Verilog. This task is more challenging compared to programming languages because bit-level operators are commonly used, which makes the CDFG-building process significantly more complex. Our work overcomes this challenge by building operator-level CDFGs from Verilog and using simulation results generated by a simulator as supervision.

We tested our model at the module level of RTL, but this does not mean that our experiments were only conducted on circuits of small scale. One of our key motivations for acquiring word-level CDFG representations of Verilog is that word-level representations can reduce the graph's scale while preserving the semantic information of the original RTL code. We present the statistics of the lines of original Verilog, Verilog IR, and Aiger (an intermediate representation of netlist-level circuits after decomposing) in the tables below. These statistics reveal that the actual scale of circuits is not small; a portion of the circuits exceeds the representation capability of most current LLMs and GNNs. For instance, an RTL-level circuit graph with 100 nodes can be transformed into an AIG graph with more than 10,000 nodes after synthesis, resulting in the loss of the semantic information of Verilog.

Overall, thank you again for your valuable feedback. We hope that our response clarifies the contributions and novelty of our work and encourages you to reconsider your previous rating of the paper.

Lines of Verilog

Lines of Verilog IR

Nodes of Aiger

Thank you for the reply and clarification. Score raised.

Thank you for your valuable review.

Weakness1: The proposed model architecture is largely based on the Heterogeneous Graph Transformer.

R: We acknowledge that we use the basic structure of HGT and GRU, which are common techniques in ML. The main contribution of our work lies in transferring these models into the circuit domain, which is a novel practice for this field. The difficulty lies in how to map a circuit into a heterogeneous graph. To address this, we designed the CDFG data structure to complete this task. We introduced novel designs for position embedding and operation embedding, and explored the embedding pattern of circuit sequence data. These designs are all tailored to circuit graph learning, and we believe they will be helpful for future hardware researchers. Specifically, we implement position embedding to help the model recognize the order of operands during message passing, as some operators do not satisfy the commutative property.

Weakness2: Missing Stronger and More Relevant Baselines

R: We should clarify that the research you referred to focuses on different tasks and circuit levels compared to our work. These works primarily address prediction tasks like PPA prediction, which are based on circuit graphs closer to the netlist (e.g., And-Inverter Graphs). However, our work focuses on learning circuit functionality at the RTL level and performing tasks like branch prediction and assertion prediction. Using models designed for netlist learning in RTL-level tasks is extremely challenging. For instance, branches in RTL-level circuits are difficult to map to wires in circuit netlists. Additionally, the graph scale differs significantly. For example, an RTL-level circuit graph with 100 nodes can be transformed into an AIG graph with more than 10,000 nodes after synthesis. It is not feasible to use such models to effectively learn circuit functions at the RTL level. Furthermore, we compare our model against Design2Vec [1], which focuses on the same task as ours and is trained on the same dataset. Experiments show that our model significantly outperforms Design2Vec.

Weakness3: Over-smoothing issue

R: Thank you for your valuable advice. We will explore the techniques referenced in your work and incorporate them to further improve our model's performance.

Weakness4: The source code and generated circuit dataset are not provided

R: We have open-sourced our framework and dataset at https://anonymous.4open.science/r/DynamicRTL-FA33, as shown in the Appendix. We have also provided the model hyperparameters in the Appendix. We believe this will help hardware researchers reproduce our work.

Q1: The split of train, test, val dataset. The hyperparameter tuning performed in this case.

R: There are several key parameters that greatly influence the model's performance, such as the GNN layer and the GRU layer in the GNN. During the training stage, we observe the accuracy of the pretraining task (branch prediction) in the first few epochs under different hyperparameter settings. We then use the best parameters to complete training and perform evaluation.

Q2: What is the variance in accuracy scores reported in Tables 1 and 3?

R: Table 1 and Table 3 correspond to different tasks. Table 1 shows the model's performance on the branch prediction task, while Table 3 evaluates the model pretrained on the branch prediction task for the assertion prediction task. These experiments demonstrate that our model learns dynamic information about circuits and can be transferred to other tasks related to the dynamic functionality of circuits.

Q3: The model’s accuracy vary across circuit designs with different depths?

R: The average depth of circuit CDFGs in our dataset is 7, and the depths of most circuit CDFGs are under 10 (as noted in line 372 of the paper). This low depth is due to the characteristics of circuits. If a circuit's depth is too high, its latency becomes very large, degrading its performance. Consequently, circuits are typically implemented with parallel structures, which is a key difference between Verilog and other programming languages. For your third question, we have supplemented the experimental results as the table shown below.

Overall, thank you again for your valuable feedback. We hope that our response clarifies the contributions and novelty of our work and encourages you to reconsider your previous rating of the paper.

[1] Vasudevan et al. "Learning semantic representations to verify hardware designs". NIPS 2021

Lines of Verilog

Lines of Verilog IR

Nodes of Aiger

Thanks authors' response. My remaining concerns are as follows:

The difficulty lies in how to map a circuit into a heterogeneous graph. To address this, we designed the CDFG data structure to complete this task.

The conversion of RTL to the CDFG data structure appears primarily as an engineering step. At line 132 of the manuscript, the authors themselves even note that utilizing a CDFG is a common way to understand the function of RTL. Thus, the conversion step may not represent a novel advancement but rather a standard practice within the field

we implement position embedding to help the model recognize the order of operands during message passing, as some operators do not satisfy the commutative property.

First of all, the authors do not mention position embeddings in the list of contributions in the Introduction section, leading readers to believe that this is not a key contribution of this work. Moreover, there are numerous positional embedding methods (some detailed in GraphGPS[7]), none of which are compared against in this study. I believe a more rigorous empirical comparison would make the efficacy of the proposed method more convincing.

These works primarily address prediction tasks like PPA prediction, which are based on circuit graphs closer to the netlist...Using models designed for netlist learning in RTL-level tasks is extremely challenging.

While the referenced methods focus on GNNs specifically designed for netlists, it's unclear why the authors have not considered applying these models directly to the CDFG generated from RTL in this study. For example, I see no apparent technical barrier that would prevent replacing the aggregation step in this paper with the aggregation functions proposed in [3].

Over-smoothing issue

It seems authors haven't resolved this concern yet. The current manuscript still mentions the proposes approach can mitigate GNN over-smoothing, while not being supported by empirical results.

The split of train, test, val dataset. The hyperparameter tuning performed in this case.

I would like to clarify my previous question. Since authors split the circuit dataset into only training and test sets, with no validation set, do authors only use training set to perform hyperparameter tuning? This experimental setup is unclear from either the manuscript or authors' response.

Given all the concerns mentioned above, I have decided to maintain my current score.

[7] Rampášek et al., "Recipe for a General, Powerful, Scalable Graph Transformer", NeurIPS'22.

Thank you for your review.

For the first question, we acknowledge that our model cannot handle all the dynamic features of circuits. We focus on a specific area that has been neglected by previous research: the dynamic functionality at the RTL level. Compared with other dynamic features, such as PPA, RTL-level functionality prediction requires learning a more fine-grained representation of circuits with simulation data. As for the iterative algorithms you mentioned, they indeed present a good scenario for GNN models to improve upon. We believe that with sufficient specific training data, our model could address this question, and this could be considered one of our future directions.

For the second question, there are numerous works that focus on predicting the PPA of circuits, and most of them predict PPA at the netlist level, which is closer to the implementation and actual scale of circuits. What sets our work apart is that it focuses on the dynamic behavior representation of circuits at the RTL level, which is closer to the semantic information and functionality of circuits and is more readable for humans. If someone wants to predict the PPA of circuits, they can synthesize the RTL circuits into netlists to obtain more precise PPA values. Moreover, the process of synthesis is not time-consuming.

When it comes to the scale of circuits, the motivation for acquiring the word-level CDFG representation of Verilog lies in its ability to downsize the scale of the graph while preserving the semantic information of the original RTL code. We present the statistics of the lines of original Verilog, Verilog IR, and nodes of Aiger (an intermediate representation of a netlist-level circuit after decomposing the circuits) below. These statistics reveal that the actual scale of circuits is not small; a portion of the circuits exceeds the representation capability of most current LLMs and circuit learning models. This highlights a significant contribution of our paper.

Overall, we sincerely thank you again for your valuable feedback. We hope that our response clarifies the contributions and novelty of our work, encouraging you to reconsider your previous rating of the paper.

Lines of Verilog

Lines of Verilog IR

Nodes of Aiger

Thank you for the replies, but some of the replies are still not quite clear.

1. If the design can also handle iterative designs, it will be good to take iterative designs such as graph processing accelerators into consideration.

2. "What sets our work apart is that it focuses on the dynamic behavior representation of circuits at the RTL level, which is closer to the semantic information and functionality of circuits and is more readable for humans.". RTL-level representation has better semantic advantages, but specific benefits more than than "readable for humans" are expected. By the way, PPA indicates power, performance, and area. I agree that power and area are usually good enough using logic synthesis, but performance is not supported in most of the synthesis tools. In fact, if the dynamic features such as branch and iterative execution are sufficiently considered, it will produce better performance prediction and can be compared to netlist-based prediction.

3. The benchmark looks reasonable for me. I think the setups in the experiment section is bit misleading which can be improved later.

Thanks a lot for your valuable advice.

Q1 & Q2: It will be good to take iterative designs such as graph processing accelerators into consideration.

R: Maybe I start to understand the meaning of your advice after thinking about it. The advice 1 meanings that we can take algorithm circuits implemented in hardware into consideration. And possibly, the meaning of advice 2 is to consider the potential of evaluating the performance of the algorithm circuits implemented in hardware, not the delay of logic gates. Because if the meaning of “performance” there is the delay of logic gates in circuits, it still can be derived easily by netlist-level representation. May I ask if my understanding is correct?

We haven’t explored the potential of our work on algorithm circuits implemented in hardware, as our initial goal was to derive representations for general circuits. To achieve this, we collected data from the Internet and ensured that the data distribution was balanced. Considering this focus, we do not have much experience with algorithm circuits implemented in hardware. These circuits are somewhat niche and are often implemented using FPGA. However, we plan to expand our work into this field in the future.

Q3: The benchmark looks reasonable. I think the setups in the experiment section is bit misleading which can be improved later.

R: Thanks a lot for your advice, we have revised it.

Thank you for your review.

Weakness: The task that was used seems to be too simple to say that it is effective. Seems like the tasks are simply binary classification tasks which is not sufficient.

R: Branch prediction accuracy is a commonly used metric, as seen in Design2Vec [1]. Although it is simply a binary classification task, accurately predicting it is quite challenging because it is strongly related to the input sequence and fine-grained circuit functionalities. We have also tried to enrich the metrics in our paper, this is why we introduced the assertion prediction metrics, which further demonstrate that we have learned a comprehensive representation of circuit dynamic behaviors using just branch prediction tasks. Branch coverage prediction and assertion are both important tasks in the ASIC verification flow, which accounts for about 40% of the time spent on IC design and tape-out.

Q1: Can you provide more details about the test set for the two tasks? For example, size of the dataset, is it balanced, ... Without that information, it is possible for the data to be biased.

R1: We have provided details about our dataset in Appendix C. To show that our dataset is unbiased, we applied the t-SNE method, and the results are also included in the Appendix.

Q2: It seems that the number of nodes in the circuit are too small. How does this scale for much larger ones? At what point does the accuracy drop significantly for table 2.

R2: One of our key motivations for acquiring the word-level CDFG representation of Verilog is that word-level representation can reduce the graph's scale while preserving the semantic information of the original RTL code. We present statistics on the lines of original Verilog, Verilog IR, and nodes of Aiger (an intermediate representation of netlist-level circuits after decomposing the circuits) in the tables below. These statistics reveal that the actual scale of circuits is not small; a portion of the circuits exceeds the representation capability of most current LLMs and GNNs. For example, an RTL-level circuit graph with 100 nodes can be transformed into an AIG graph with more than 10,000 nodes after synthesis, leading to the loss of the semantic information of Verilog.

Q3: How is the speed of inference and training?

R3: Our model requires approximately 1 hour of training time per epoch on a single A800 GPU for the training dataset. The inference time for each batch is about 0.9 seconds, with a batch size of 128. It is worth mentioning that although the number of our CDFGs is about 6,300, each CDFG includes 30 simulation traces, resulting in approximately 190,000 data entries.

O4: For the two tasks, how does it compare to human evaluation or any pre-existing programs?

R4: Our model requires significantly less time compared to commercial hardware tools, such as Synopsys VCS, to obtain branch prediction results. The key advantage of using our model for branch prediction is that we can compute the branch hit results for a large number of input sequences simultaneously using a GPU, which is much faster than human evaluation or hardware simulators. This is particularly useful in hardware verification scenarios, where fast but less accurate screening of all generated test cases is required.

Overall, thank you again for your review.

Lines of Verilog

Lines of Verilog IR

Nodes of Aiger

[1] Vasudevan et al. "Learning semantic representations to verify hardware designs". NIPS 2021

Thank you for your review.

Weakness1: The focuses of proposed work primarily on branch prediction as the main indicator of dynamic behavior. How about other aspects of circuit dynamics such as timing, power consumption, or glitch behavior.

R1: We acknowledge that there have been numerous works focusing on PPA prediction, and many of these works have already achieved very good performance. Therefore, we do not aim to further improve their methods in our work. Instead, we focus on other dynamic tasks that have been neglected by previous methods—branch prediction and assertion prediction. Note that our dynamic tasks are quite different from PPA prediction. PPA prediction usually involves learning the global dynamic behavior of circuits, whereas branch prediction focuses on fine-grained dynamic behavior. Most previous PPA prediction works are conducted at the netlist level of circuits, which are closer to the implementation and factual scale of circuits. In contrast, our work focuses on the dynamic behavior representation of circuits at the RTL level, which is closer to the semantic information and functionality of circuits, making it more readable for humans. The significance of our work lies in its focus on dynamic RTL-level circuit representation, which has the potential to enable digital circuit verification. This allows RTL coders to check whether the functionality and semantic information of the RTL satisfy the constraints they set.

Weakness2: It will be better if the authors have some discussion on the limitation of proposed method, especially the effectiveness and efficiency of proposed method for large-scale circuits.

R2: We did test our model at the module level of RTL, but this does not mean that our experiments were only conducted on circuits of small scale. Our motivation for acquiring the word-level CDFG representation of Verilog lies in its ability to downsize the graph's scale while preserving the semantic information of the original RTL code. We present the statistics of the lines of original Verilog, Verilog IR, and Aiger (an intermediate representation of netlist-level circuits) after decomposing the circuits in the tables below. These statistics reveal that the actual scale of circuits is not small; a portion of the circuits exceeds the representation capability of most current LLMs and GNNs. For example, an RTL-level circuit graph with 100 nodes can be transformed into an AIG graph with more than 10,000 nodes after synthesis, leading to the loss of the semantic information of Verilog.

One limitation of our work is the complexity of building CDFGs from Verilog. Given the lack of existing work on Verilog CDFG construction, we have not yet attempted to construct repository-level CDFGs but have instead focused on module-level Verilog. Nevertheless, repository-level Verilog CDFG construction is feasible. The original motivation for our work stems from the scarcity of Verilog source code available on the Internet, which significantly weakens the Verilog generation capabilities of LLMs compared to mainstream programming languages. The main purpose of designing digital circuits is computation. Although LLMs can generate Verilog through next-token prediction, they have little understanding of control flow, data flow, and computation. Additionally, the lack of quantity and diversity in Verilog source code limits their ability to generate more correct, creative, and practical Verilog code. Consequently, we believe that CDFG representation learning of Verilog is significant for circuit design automation.

Overall, thank you again for your valuable feedback. We hope that our response clarifies the contributions and novelty of our work and encourages you to reconsider your previous rating of the paper.

Lines of Verilog

Lines of Verilog IR

Nodes of Aiger