Reinforcement Learning with learned gadgets to tackle hard quantum problems on real hardware

The claims are generally clear, but are not always as supported by the evidence as they should be. One of the most important aspects of any ML for quantum work is the scalability, especially for circuit optimization...

We appreciate this critical feedback regarding scalability evidence, which we have addressed in our revised manuscript as follows:

(1) $**Scaling:**$ We demonstrate the applicability and advantage of GRL up to 6 qubits (in the revised manuscript), showing that compard to state-of-art RL approaches, GRL achieved a better accuracy with smaller circuit;

(2) $**Scaling of gadget extraction algorithm:**$ As point out in the response of the previous reviewers we, in the revised manuscript demonstrate that gadget extraction maintains computational tractability for shallow circuits, successfully scaling to 5-qubit systems with synthesis times ranging from 61 seconds (2-qubit, depth-5) to 1.3 hours (3-qubit, depth-10). Notably, the 2-qubit TFIM gadgets discovered through this process prove sufficient for solving 6-qubit problems, highlighting their transferability. While deeper circuits present challenges that will require future optimizations, including parallelization techniques and stitch algorithms [Proceedings of the ACM on Programming Languages 7.POPL (2023): 1182-1213], our current approach demonstrates practical scalability for intermediate-scale quantum systems relevant to near-term hardware.

---

The experimental analysis was in general sufficient. However, they would benefit from more baseline analysis...

We appreciate this valuable feedback regarding baseline comparisons. In our study, we specifically benchmark against CRLQAS (ICLR 2024), the current state-of-the-art RL approach for quantum circuit optimization, using identical agent-environment settings to ensure fair comparison. Our results demonstrate consistent improvements over CRLQAS in both solution quality (lower approximation errors) and efficiency (reduced gate counts) across all tested system sizes (2-6 qubits). While we acknowledge the existence of other RL and non-RL methods, we focused on CRLQAS as it represents the most relevant and competitive baseline for our reinforcement learning framework. The demonstrated scaling properties of our GRL approach, particularly its linear time complexity and gadget transferability, provide compelling evidence of its advantages over existing RL methods for quantum circuit optimization tasks. We will include additional comparisons with classical optimization methods in future work to further strengthen these claims.

---

The return dots in Figure 7 don't really add much/are hard to interpret.

We agree with reviewer. In the revised version we remove the dots from Fig 7.

---

The gadget approach makes it somewhat unique in the world of RL for quantum circuit optimization (of which there are quite some papers)...

We appreciate this important observation regarding our work's positioning. In the revised manuscript, we have significantly expanded the Related Works section to provide a more comprehensive survey of RL approaches for quantum circuit optimization, including recent advances in quantum circuit optimization including https://arxiv.org/abs/2109.03188 and https://ojs.aaai.org/index.php/AIIDE/article/view/7437 also https://arxiv.org/abs/2103.07585 (cited in our original submission).

---

The related works are generally presented, but lack sufficient integration. The key points that distinguish this work from the works presented is not sufficient.

We make sure in the revised version of the manuscript the difference with GRL to the other state-of-art approaches are clearly stated by rewriting the related works and the results section.

---

I think this paper would benefit from a LaTeX algorithm section to actually walk the reader through what's going on at each step...

We thank the reviewer for these constructive suggestions.

[1] Figure 1(b) has been redesigned with improved annotations and a clearer visual hierarchy to better communicate the key concepts upon first reading.

[2] The abstract has been significantly condensed to a single, focused paragraph that more concisely presents our core contributions while maintaining all key technical claims.

[3] Following the reviewer's suggestion, we have moved Figure 4 to Appendix and incorporated its most salient points into a revised Figure 2, creating a more streamlined presentation of results in the main text.

I don’t find the claim that GRL is “a versatile and resource-efficient framework for optimizing quantum circuits, with potential applications in hardware-specific optimizations, variational quantum algorithms, and other challenging quantum tasks” to be well-supported. How is GRL resource-efficient if it takes hours, and in some cases, days to train on two-qubit examples?

We appreciate this important question regarding computational efficiency. While the initial training phase for our GRL framework requires substantial time (up to 30 hours), we emphasize two key points about resource efficiency:

$**Optimal solution convergence:**$ The agent typically finds optimal solutions within 5-6 hours (as shown in our results), with the full training period primarily serving to verify solution robustness.

$**Linear scaling:**$ Training time scales linearly with qubit count, making the approach practical for intermediate-scale systems.

We acknowledge the training time can be further improvements by exploring parallelization and transfer learning techniques. Hence we remove the term "resource-efficient" from the revised manuscript.

---

The TFIM is a reasonable test case, especially for many qubits. However, isn’t the one-dimensional TFIM exactly solvable (and therefore so is the two-qubit TFIM)? What happens on harder instances?

We appreciate this thoughtful question regarding our choice of test case. While the one-dimensional TFIM is indeed exactly solvable, even for small systems like the two-qubit case, its analytical tractability provides a crucial benchmark for validating our method’s accuracy and efficiency.

Importantly, the challenge lies not in solving the model itself but in deriving hardware-efficient circuit implementations—a task that remains non-trivial and practically relevant for near-term quantum devices. Our results show that GRL performs particularly well in the hard regime TFIM regimes, where it maintains low errors while using fewer gates and depth than RL baselines. Here are the results.

Beyond exactly solvable cases, the method’s strength comes from its ability to extract reusable gadgets—fundamental building blocks of quantum dynamics—that generalize across system sizes and Hamiltonian parameters. This is shown by our scalability analysis in the revised manuscript where these patterns recur in larger systems, as well as in additional tests on more complex systems. While TFIM provides a controlled starting point, GRL’s true value lies in its automated, scalable optimization for problems where classical methods become impractical. We explore much bigger systems and deeper circuits in future work by utilizing parallelization and the stitch algorithm [POPL (2023): 1182-1213].

---

What was the connectivity graph of the three-qubit TFIM? Is it linear? All-to-all?

Thank you for the question. For 3-qubti and higher we consider all-to-all connectivity. The action space of the GRL can be tweaked to incorporate the restrictions of the connectivity of real-hardware.

---

Isn’t the one-dimensional TFIM exactly solvable? If so, why is the two-qubit TFIM a good test case?

We appreciate the important question. Kindly check the response above.

---

What hope does this approach have at scaling if it took hours to train on a two-qubit example with exact solutions?

We acknowledge the computational time required for training while emphasizing three key points: (1) The 30-hour maximum reflects total training time rather than time-to-solution (optimal configurations typically emerge within 5-6 hours), (2) Demonstrated linear scaling makes the approach practical for intermediate-scale systems as shown in our 6-qubit results (which is trained for less than 30 hours and achieves en error in order of $10^{-3}$ in $6$ hours of training), and (3) The upfront cost is amortized through reusable solutions across multiple circuit optimizations. While we're actively investigating transfer learning and parallelization to further improve efficiency, the method's value lies in its ability to automate hardware-aware optimizations where manual approaches become impractical - a trade-off that becomes increasingly favorable as system complexity grows.

The benchmark experiment only involves small systems (2- and 3-qubit). The limited system size may weaken the authors’ claims about the scalability and generalization of their methods.

Thank you for highlighting this weakness in the paper. In the revised manuscript, we address this limitation by extending the applicability of our method to 6-qubit TFIM problems. Here are the results:

We observe that GRL not only provides a better approximation to the ground energy in hard regimes but also achieves this with significantly fewer gates than state-of-the-art RL approaches such as [CRLQAS, The Twelfth International Conference on Learning Representations (2024)] (denoted as RL in the table). This advantage is achieved by transferring gadgets discovered while solving the 2-qubit TFIM in the easy regime. These results demonstrate that the scalability of GRL is enabled by transferring gadgets from smaller to larger and harder systems.

$**Scalability of program synthesis:**$ We further believe these results can be improved by introducing more gadgets. In the revised manuscript, we include an extensive analysis of different TFIMs (ranging from 2 to 5 qubits) with varying depths, showing that for up to 4 qubits, the program synthesis approach rediscovers gadgets from smaller systems using shallow circuits. For more complex gadgets, deeper circuits or higher qubit counts are required. Our future work will explore parallelization and the stitch algorithm [Proceedings of the ACM on Programming Languages 7.POPL (2023): 1182-1213] for deeper circuits. Currently, our method demonstrates practical scalability for intermediate-scale systems.

---

Other Comments Or Suggestions:

• Line 213:..
• In the sentences following Equation (2):..

Thank you for pointing out the typos. In the revised manuscript we correct this.

---

Any physical intuitions why gadgets found in smaller TFIMs or smaller h can be generalized in a larger system?

Thank you for this insightful question. In the revised manuscript (section Scalability of program synthesis), we analyze TFIMs ranging from 2 to 5 qubits with varying depths. We observe that for systems up to 4 qubits, the majority of the recurring gadgets match those identified in earlier program synthesis studies with shallow circuits. Furthermore, when examining deeper circuits, we find that complex gadgets often contain substructures identical to those discovered for smaller systems. This hierarchical recurrence strongly suggests that gadgets derived from smaller TFIMs can indeed be effectively transferred to larger, more realistic problems.

---

How to justify the scalability of this approach for realistic quantum systems?

To demonstrate the scalability of our GRL approach to realistic problems, we systematically reuse gadgets discovered in 2-qubit TFIM systems across larger systems (up to 6 qubits) in the hard regime. This transfer learning approach enables GRL to outperform state-of-the-art quantum architecture search methods in both accuracy and gate efficiency, as quantitatively demonstrated in our results table (see previous response). The key insight is that the fundamental building blocks (gadgets) governing quantum dynamics in small systems remain physically relevant when scaled to larger problems.

---

No source code is provided.

In the revised version of the manuscript we mention source code. Which the reviewer can also access by: https://anonymous.4open.science/r/Gadget_RL-460D/README.md

---

Is it possible to incorporate a noise model (to improve noise resistance) in the design of the reward function?

Thank you for the question. We incorporate noise at the gate level rather than in rewards because minimizing cumulative reward naturally yields noise-resilient solutions, as the agent must implicitly overcome gate noise to optimize its policy. This preserves standard RL while ensuring robustness. Though we recognize that specialized noise-aware reward functions could be valuable for certain regimes and plan to explore this in future work.

The appendices are organized and well-written.

Thank you for the kind comments.

---

The proposed method expands the search space of existing RL approaches by introducing techniques from program synthesis.

The circuits found by the agent are tested on real quantum hardware, demonstrating real-machine compatibility and noise resistance.

We are grateful of the reviewer for pointing out the strengths of our paper.

The idea of using composite gates to expand the action space of RL exploration of the variational circuit ansatzes seems incremental to me...

We thank the reviewer for commenting on the significance of GRL for QAS. GRL's key innovation lies in its hierarchical exploration strategy: $**Modular gadget discovery**$ extracts optimal shallow circuit motifs (depth $d<5$), while $**hierarchical composition**$ Hierarchical composition builds deeper circuits ($d\geq10$) by reusing these verified gadgets. This approach fundamentally addresses QAS scalability by: (1) amortizing exploration costs through reusable structures, and (2) enabling knowledge transfer across problem sizes (demonstrated for 4-6 qubits in $**Section 4.3**$ in revised manuscript, we show this via a table in the next response). While performance gains vary by task, the method provides a systematic framework for tackling the curse of dimensionality in variational quantum optimization.

---

The experiments are conducted on small tasks involving only 2 or 3 qubits with a few dozens of gates. The scalability is not validated by the experiments...

Thank you for highlighting the primary drawback of our manuscript. We acknowledge the scalability issue in the modified version of the article we extend our results upto 6 qubit TFIM.

$**Scalability of Program Synthesis (Section 4.2):**$ We demonstrate that gadget extraction remains computationally feasible for shallow circuits, scaling up to 5-qubit systems ($**Table 4, Appendix F**$). Synthesis times range from 61 seconds (2-qubit, depth-5) to 1.3h (3-qubit, depth-10), with 2-qubit TFIM gadgets proving sufficient for 6-qubit problems. While deeper circuits require future optimizations (e.g., parallelization, stitch algorithms. Future work will explore parallelization and stitch algorithms [ Proceedings of the ACM on Programming Languages 7.POPL (2023): 1182-1213] for deeper circuits.]), our method shows practical scalability for intermediate-scale systems.

$**Scalability of Gadget Reinforcement Learning via Transfer (Section 4.3, Figure 4):**$ We show that gadgets learned from 2-qubit TFIM transfer effectively to larger systems (4-, 5-, and 6-qubit, see Fig. 4 in the manuscript), achieving lower error rates than baseline RL methods. Crucially, even a single reused gadget outperforms existing approaches [such as CRLQAS., The Twelfth International Conference on Learning Representations (2024) denoted as $**RL**$ in table below], confirming that GRL scales well via component reuse without large overhead. The results are summarized as follows:

---

The proposed method does not directly consider the effects of noise, while minimizing noise is a major claim in the paper’s contribution...

We appreciate the opportunity to clarify the performance advantages of GRL and its robustness under noise

• Advantage of GRL on noisy quantum hardware (Appendix I) In the following table to find the forund state for 3-qubit TFIM we show that the GRL provides better PQCs compared to RL which are more noise robust in QPU. | QPU backend | GRL| RL | | -------- | -------- | -------- | | fake_torino| $\mathbf{−3.366}$| -3.351| | fake_kawasaki| $\mathbf{-3.397}$| -3.319| | fake_quebec| $\mathbf{-3.379}$| -3.318|

• Noise resilience of gadgets (Appendix G) Here we analyze GRL under a realistic Pauli noise model (50% probability of Haar-random unitary errors per controlled rotation). We observe that performance decay of GRL mitigates as more gadgets are added, suggesting diminishing marginal damage from additional noise. | Algorithm| Error| | -------- | -------- | | GRL (1 gate)| $\mathbf{4.5\times 10^{-13}}$| | GRL (noisy 1 gate)| $4.5\times10^{-10}$| | GRL (noisy 2 gate)| $4.5\times 10^{-12}$|

---

Essential References Not Discussed:

Thank you for suggesting the references. We found them very relevant to our work and have added a discussion of relevant program synthesis techniques from the programming languages community, particularly syntax-guided synthesis (SyGuS) and recursive quantum unitary programs. These changes are highlighted in red color in $**Related works**$ section i.e $**Section 2**$.

---

The presentation is clear and easy to follow.

Thank you for this kind feedback.

---

Is $J$ fixed for all the experiments? What is the ratio of h over $J$?

$J$ was fixed across all experiments for consistency, with $h/J$ varied to explore different TFIM regimes.