Reinforcement Learning Policy as Macro Regulator Rather than Macro Placer

Thank you for your valuable and constructive comments. Below please find our response.

Q1 Figure 3(a) is not clear

We are sorry for the unclear figures. The red macro $M^1$ denotes the macro that has been adjusted; the yellow macro $M^2$ is the current macro that is adjusting; the blue macros $M^3$ and $M^4$ are the macros that are unadjusted and will be adjusted in the future steps. We will revise this figure and accompany explanation in the caption to make it clear. Thank you for bringing this to our attention.

Q2 How does the model determine adjusted and unadjusted macros? Are the macros adjusted sequentially?

Yes, the macros are adjusted sequentially. As we stated in lines 145-147 of the paper, the order of the macro sequence is determined by the corresponding net and size of a macro, and the number of connected modules that have been adjusted, which is the same as in previous studies [1-2]. We will revise to make this clear. Thank you.

[1] MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning. NeurIPS, 2022.

[2] Macro Placement by Wire-mask-guided Black-box Optimization. NeurIPS, 2023.

Q3 No feasible space is available?

We sincerely appreciate the reviewer's insightful question regarding the model's behavior in situations where no feasible space is available. In fact, throughout our entire training and test process, we did not encounter any instances where the final adjusted solution had overlaps. This can be due to the following three reasons:

1. Prioritized adjustment sequence. We employ a specific adjustment sequence that prioritizes elements with greater impact on the layout, based on many aspects, as we answered in Q2. This will mitigate the risk of overlap.
2. Position mask: MaskRegulate uses a position mask that effectively selects non-overlapping positions, fundamentally reducing the risk of overlapping occurrences.
3. Integration of regularity. MaskRegulate is designed to encourage elements to align with edges, which not only enhances the overall layout but also further reduces the probability of overlapping.

Given these strategies, we have not encountered any instances of element overlap during our training and testing phases. However, we acknowledge that the possibility of overlap may exist in extreme scenarios. As you suggested, re-adjusting previously adjusted macros is a good idea for improving the model's robustness, which can lead to legal placement result by additional trials. Besides, we can introduce a negative reward to the overlapping scenario, which would penalize adjustments that lead to overlaps, thereby guiding the model towards better adjustments.

We are grateful to the reviewer for highlighting this potential issue. We will implement this mechanism in the future. This enhancement will further improve our model's robustness and adaptability, enabling it to handle a wider range of complex layout scenarios, including potential space constraint issues. Thank you very much!

Q4 Is $\alpha=0.7$ in Table 1 and Table 2?

Yes, we set $\alpha=0.7$ by default in our experiments, including those in Tables 1 and 2. We apologize for not explicitly stating this. We will revise to clarify this setting. Thank you for bringing this to our attention.

Further improvement 1: including some visualizations from Figure 6 in the main paper.

Thank you for your appreciation of our visualizations! We're glad that you found them valuable. In the light of your suggestions, we will adjust the layout and attempt to incorporate these visualizations into the revision of our paper. We believe this will enhance readability and improve the overall presentation of our research. Thanks for your constructive feedback.

Further improvement 2: adding training convergence graphs

Thank you for your valuable suggestion. We agree that including these figures would provide additional insights and further strengthen the quality of our work. However, due to space limitation of the response phase, we couldn't include these graphs currently. We will include them in the revision of our paper. Thank you.

We hope that our response has addressed your concerns, but if we missed anything please let us know.

Thank you for your valuable and constructive comments. Below please find our response.

Q1 Technical details are not clear.

Thank you for carefully reading our paper and providing these detailed and valuable comments. We are sorry for the unclear presentations.

Q1-1 Termination function.

MaskRegulate sequentially adjusts all the macros in a certain order. All macros will be sorted according to certain rules and then adjusted one by one, as we stated in line 146 of the main paper. The MDP will terminate when all macros have been adjusted.

Q1-2 Would the occupied block be switched by the current block?

No. When MaskRegulate is adjusting a macro, only the macros that have been adjusted (i.e., macros that appear earlier in the adjustment sequence) are considered as occupied, as illustrated in Figure 3(b) of the paper. The macro to be adjusted can temporarily occupy the position of unadjusted macros, and the HPWL is calculated in their respective nets.

We will revise these parts to make them clear. Thank you very much.

Q2 Calculation of WireMask

The WireMask is calculated by all the other macros' current locations, whether they are adjusted or not, as illustrated in Figure 5 of our paper.

Q3 Model inference in the generalization part.

Yes. We frozen the pre-trained model and only do forward inference in the generalization experiments. We will make this clear in our final revision. Thank you.

Q4 Integrate unknown heuristics

Thank you for your insightful comment and valuable suggestion. We would like to address your point as follows:

In the chip design domain, heuristic knowledge plays a crucial role, and state-of-the-art chips still heavily rely on expert knowledge. We acknowledge that there may exist other unknown heuristics that could potentially impact the performance of the placement process. Incorporating these additional heuristics is an important direction for future research. In our upcoming work, we plan to explore techniques like reward modeling to effectively capture and integrate a wider range of expert knowledge to guide the training of our RL regulator.

This work represents an initial attempt to incorporate advanced heuristic knowledge, specifically the idea that macros should be placed near the periphery of the chip, into the current practice of RL-based placement. Our results demonstrate that this approach yields significant improvements in placement quality. We believe that this work opens a door to enhance RL-based approaches by incorporating heuristic knowledge to make them more suitable for real-world placement tasks.

We greatly appreciate your suggestion and will include a discussion of these potential future directions in the revised version of our manuscript. Your feedback has been invaluable in helping us identify areas where we can clarify and expand upon our contributions.

We hope that our response has addressed your concerns, but if we missed anything please let us know.

Thank you for your valuable and positive comments. Below please find our response.

Q1 Discussion of regularity in other paper

Thanks for your valuable comments.

[1] proposes a multi-level approach for macro placement that can leverage the hierarchy tree and effectively explore structural information at the RTL stage. The hierarchical partitions facilitate the integration of information such as dataflow, wirelength, and regularity. Then, [2] attempts to "mimic" the interaction between the RTL designer and the physical design engineer to produce human-quality macro placement results by exploiting the logical hierarchy as well as regularity and connectivity of macros.

However, no studies have integrated regularity into RL. In our paper, we not only propose a new RL regulator framework but also integrate regularity into it. Additionally, our proposed RegularMask can be used to improve other methods, such as MaskPlace (as shown in Table 8 of the paper) and WireMask-BBO (as shown in the following answers to Q2). We will add this discussion to our revised paper. Thank you very much.

[1] RTL-aware dataflow-driven macro placement. DATE, 2019

[2] RTL-MP: Toward practical, human-quality chip planning and macro placement. ISPD, 2022.

Q2 Discussion of using RL as regulator. Can the regular mask be used in WireMask-BBO?

Thank you for your valuable comments. Compared to RL, BBO relies on iterative evaluation and cannot obtain a generalizable policy capable of rapid adjustment through forward inference alone. We believe other methods (such as BBO) can also utilize our proposed RegularMask. To investigate whether WireMask-BBO can benefit from incorporating RegularMask, we conducted additional experiments on the ICCAD 2015 benchmark to compare WireMask-EA and WireMask-EA + RegularMask. WireMask-EA + RegularMask comprehensively places and adjusts based on both WireMask and RegularMask. As shown in Table R3, WireMask-EA + RegularMask improves both global HPWL and regularity in all the cases, demonstrating the versatility of our proposed RegularMask. We will add this discussion and experiment to our revised paper. Thank you.

Q3 Why does GP HPWL improve since only MP HPWL is considered in the learning process?

Macros (i.e., individual building blocks such as memories) have much larger sizes than standard cells (i.e., smaller basic components like logic gates), which significantly impacts the overall quality of the final chip. A common practice in placement is to first fix macro locations and then place standard cells [1-3]. After determining the positions of macros by optimizing MP HPWL, placing the remaining standard cells typically yields good final results, i.e., GP HPWL. However, to consistently improve GP HPWL, it's necessary to consider the sparse signal in the GP stage, which is one of our future areas of work as we stated in our paper. We will add this discussion to our revised paper. Thank you very much.

[1] MaskPlace: Fast chip placement via reinforced visual representation learning. NeurIPS, 2022.

[2] AutoDMP: Automated DREAMPlace-based macro placement. ISPD, 2023.

[3] Macro placement by wire-mask-guided black-box optimization. NeurIPS, 2023.

We hope that our response has addressed your concerns, but if we missed anything please let us know.

Thank you for your detailed and valuable comments. Below please find our response.

Q1 Limited benchmark scope.

Thanks for your valuable comments. In our work, we use the ICCAD 2015 contest as our benchmark, which is currently one of the largest open-source benchmarks that allows us to evaluate congestion, timing and other PPA metrics. To the best of our knowledge, ICCAD 2015 is the benchmark that closely reflects the current practices in the EDA industry. We agree with you that adding benchmark scopes would further strengthen our work. Thus, we add experiments on ISPD 2005 contest benchmark, which is also a popular benchmark in AI for chip design but does not have sufficient information for PPA evaluation. The results can be found in Table R1 in our PDF file. For detailed settings and discussions, please refer to the following Q2. Thank you.

Q2 Generalization ability

Thank you for your valuable comments. In our paper, we have tested MaskRegulate's generalization ability in Table 2. We have now further tested the generalization on the ISPD 2005 benchmark by directly using the pre-trained models on superblue 1, 3, 4, and 5 (i.e., the same models in Table 2) of MaskPlace and MaskRegulate to place and regulate the eight chips. As shown in Table R1, MaskRegulate still outperforms MaskPlace in most cases, demonstrating our superior generalization ability and robustness. We will include this experiment in our revised paper. Thank you very much.

Q3 Clarity of method description.

Thank you for your suggestions. However, we cannot update the paper now due to the rules of rebuttal phase. We will carefully revise our paper according to your suggestions, including adding pseudo-code and introducing regularity and MDP in detail. Thank you very much.

Q4 Can MaskRegulate improve the quality of any method and any quality of initial macro placement solution?

Thank you for your valuable comments. Yes, MaskRegulate can be used to adjust any initial macro placement solution. According to your suggestions, we have conducted additional experiments to demonstrate this capability. We used the pre-trained model on superblue 1, 3, 4, and 5 (i.e., the same models in Table 2 and R1) to adjust different placement results obtained by MaskPlace, AutoDMP, and WireMask-EA. The results are shown in Table R2. MaskRegulate consistently improves regularity on all four unseen chips and enhances global HPWL on three chips. We will include this experiment in our revised paper. Thank you very much.

We hope that our response has addressed your concerns, but if we missed anything please let us know.

Thank you for your valuable comments. Below please find our response.

Q1 Compared with MaskPlace, the contributions of this paper lie in the expertise or experiences in the expert chip design area .... One thing I am concerned about is that NeurIPS may not be a good venue to discuss this contribution.

Thanks for your comments. However, we may not agree with you respectfully. Compared with MaskPlace, the main contributions of our work are two folds:

1. Problem formulation: MaskRegulate shifts the focus of RL from placing macros from scratch to refining existing macro placements, which allows for more effective learning from structured state and accurate reward information, significantly enhancing the learning efficiency.

2. Integration of regularity: We introduce regularity, a critical yet overlooked metric in chip design, into the RL training framework, which not only aligns with industry practice but also enhances the chip PPA quality.

We have conducted comprehensive experiments to show the superior performance of MaskRegulate:

• Our main experiments on the popular ICCAD 2015 benchmark have shown the significant improvements of MaskRegulate in PPA metrics, demonstrating the practical applicability and effectiveness of the RL regulator.

• For the problem formulation, Table 6 in the paper has shown that Vanilla-MaskRegulate is better than MaskPlace. The only difference between them is the problem formulation, and all the other components (e.g., state and reward) are the same. Thus, the results in Table 6 clearly demonstrates our motivation, highlighting the advantages of our regulator problem formulation.

• For the integration of regularity, Table 8 in the paper and Table R3 in the PDF file show that our proposed RegularMask can be integrated into other methods such as MaskPlace and WireMask-EA.

Next, we will explain why NeurIPS is a good venue for our work. Recently, AI for chip design significantly expands the application domain of current AI technologies and enhance their impact [1]. In recent years, researchers have discovered that RL can assist chip design, as demonstrated in Google's Nature paper [2]. Since then, numerous top-tier venues (e.g., NeurIPS, ICML, and ICLR) have emerged with related research, aiming to further improve its effectiveness from different perspectives [3-9], just to name a few. Our work not only proposes a new placement paradigm but also introduces practical experience from the chip design field to guide the algorithm, which opens new possibilities for the application of RL in chip design. Therefore, we believe that a top-tier AI conference like NeurIPS is a proper venue for our work. Thank you.

[1] Machine learning for electronic design automation: A survey. TODAES, 2021

[2] A graph placement methodology for fast chip design. Nature, 2021.

[3] On joint learning for solving placement and routing in chip design. NeurIPS, 2021.

[4] MaskPlace: Fast chip placement via reinforced visual representation learning. NeurIPS, 2022.

[5] Chipformer: Transferable chip placement via offline decision transformer. ICML, 2023.

[6] Macro placement by wire-mask-guided black-box optimization. NeurIPS, 2023.

[7] CircuitNet 2.0: An advanced dataset for promoting machine learning innovations in realistic chip design environment. ICLR, 2024.

[8] Reinforcement learning within tree search for fast macro placement. ICML, 2024.

[9] A hierarchical adaptive multi-task reinforcement learning framework for multiplier circuit design. ICML, 2024.

Q2 Discussion about combinatorial solver.

Good points! Thanks for your insightful comments. Placement is a very large-scale non-linear optimization problem. Using combinatorial solvers for placement has a long history [1-2], which belongs to analytical placement approach. They were initially used for placement of small-scale chips, but have gradually been replaced by more effective gradient-based approaches [3-4] in recent years. Currently, combinatorial solvers are typically used for small-scale and constrained problems, such as floorplanning [5].

The state-of-the-art analytical placement method is DREAMPlace [4], which is an important baseline in our work. Compared to DREAMPlace, our RL regulator has many advantages, as demonstrated in our paper. According to your comments, we will introduce the history of analytical placers in detail to provide a comprehensive understanding of related techniques for readers without the placement background. Thank you for bringing our attention to this point.

[1] Analytical approaches to the combinatorial optimization in linear placement problems. TCAD, 1989.

[2] BonnPlace: Placement of leading-edge chips by advanced combinatorial algorithms. TCAD, 2008.

[3] Replace: Advancing solution quality and routability validation in global placement. TCAD, 2018.

[4] DREAMPlace: Deep learning toolkit-enabled GPU acceleration for modern VLSI placement. TCAD, 2021.

[5] Global floorplanning via semidefinite programming. DAC, 2023.

Q3 PPO can handle continuous action space, is there a possibility of improvement using continuous action space.

Thanks for your suggestions. Due to the large action space of placement, previous studies propose to discretize the layout into grids, thus reducing the action space and improving learning efficiency. We agree that learning in the continuous space can enhance decision flexibility, but this also brings challenges such as low convergence rate and under-exploration. We will consider it in the future. Thank you.

Q4 Future directions of RL

Our main future work includes two aspects:

1. Further improving the generalization of the RL policy by using better architectures and training methods.
2. Utilizing some MO techniques (e.g., MORL) to obtain a set of Pareto-optimal placement results with different preferences across multiple objectives.

We hope that our response has addressed your concerns, but if we missed anything please let us know.