TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search

We sincerely thank you for the valuable feedback on the manuscript, and appreciate that you agree on the interesting idea motivating our work. In the following we systematically provide the response to your comments.

There are a decent number of typos in the paper (which is okay, as it is very possible that English is not the first language of the authors). Regardless of the reason, the paper needs to go through another round of copy editing...

We acknowledge the presence of typos and missing definitions, we will perform a thorough proofreading of the manuscript to correct all exposition issues.

Why is bond dimension 2 sufficient for these simulations? I would have expected these simulations to need a bond dimension of $2^5$-$2^7$ for the 8 qubit case, and more for a larger simulation

You have raised a valid concern regarding the use of a low bond dimension, and we now clarify that this choice is in fact intentional. For the relatively small quantum chemistry problems considered, a larger bond dimension would be sufficient to produce solutions that are already within chemical accuracy of the ground state energy. In this sense, the low bond dimension is not a limitation but rather an intentional choice: using a larger bond dimension would allow us to solve the problem classically, without the need of any additional RL-based quantum computing resource.

We emphasize that the current study is a proof-of-concept intended to validate the proposed combination of TN and RL-QAS, and we start testing it on small systems due to the heavy computational requirements needed to train RL agents (see the next discussion). Applying the same methodology on a larger number of qubits would probably require larger bond dimensions in the MPS/DMRG, which is consistent with your expectation.

We also clarify a technical point: for an MPS (with open boundary conditions), the maximum required bond dimension to represent an arbitrary n-qubit state exactly is $2^{⌊n/2]}$. For n=8, this means that a bond dimension of $2^4$ is the theoretical maximum needed to express any arbitrary state exactly, not $2^5$ or $2^7$ as suggested. We have clarified this in the revised manuscript to avoid confusion.

If bond dimension 2 is sufficient for these simulations, why haven't you scaled the simulation to significantly larger system sizes (50+ qubits) when in principle an 8 qubit matrix product state implementation should take a very small amount of memory to store classically?

Thank you for the interesting question. We want to clarify that the main computational bottleneck in our simulations does not lie in the MPS/DMRG simulations (it is inexpensive as long as the bond dimension is not too big), but in the RL training phase. Each training episode requires simulating and optimizing a parameterized quantum circuit, and computing its expectation value multiple times equal to the number of (episodes X steps). Already for ~12 qubits, this becomes computationally intensive due to the number of episodes and steps required for convergence/solving the task. This currently limits our ability to explore system sizes beyond ~20 qubits, as already at these sizes training the RL agent can take approximately 48 hours, which is the maximum runtime available on the University computing cluster which is currently available to us. Future work will address this by further alleviating the computational requirements of the RL training process, and investigating more performant quantum circuit simulators.

Simulating quantum circuits is computationally very hard in general, which is indeed the reason why we propose to combine a classical warm-start with an RL agent that completes the task by accessing quantum resources directly. But since actually accessing quantum hardware is hard because of limited access by providers, we are bound to do simulations of quantum circuits. Finally, to the best of our knowledge, we remark that our work reports the largest RL-based QAS simulation to date with 20 qubits.

What tensor network implementation are you using?

We use the Quimb Python library for all tensor network computations and MPS manipulations, including DMRG simulations and mapping to quantum circuits.

How can you guarantee that these results will continue to scale after the point where classical algorithms break down?

We acknowledge this is a central open question for all hybrid classical-quantum algorithms, including ours. Like most near-term quantum algorithms, we cannot provide theoretical guarantees that our TensorRL-QAS approach will outperform classical methods at much larger scales. However, we believe our strategy of maximizing classical pre-processing with tensor networks not only will help reduce the quantum resources required, which is especially important for the near- and mid-term regime, but also very much aligns with the current expectation that useful quantum advantage will be achieved with a tight combination of quantum and classical high-performance computing. Overall, we are confident that TensorRL-QAS provides a necessary starting point for future investigations on larger sizes, where a quantum advantage may manifest. We have added explicit discussion of these considerations and limitations in the revised manuscript.

We hope these addresses your concerns and questions and further helps you to re-evaluate our paper. Kindly let us know if you have any further comments or/and questions.

Thank you very much for your thoughtful comments and for pushing us to examine the scaling of our RL optimization with respect to the initial DMRG bond dimension. We have now performed additional benchmarks specifically as you suggested, on a 12-qubit LiH molecule, using 100 RL episodes per run (this is a result with just one seed, in the camera ready version we will have the same outcome with multiple initilizations of the neural net). For each bond dimension, our target was to reach an energy below that from simulated annealing (2.5×10⁻² Hartree). Our updated results are included in the following table:

Our results show that both the optimization time and the episode time decline noticeably as the initial DMRG bond dimension increases. We think this is a natural consequence of providing a better starting point for RL optimization. That said, we completely agree with your main point: for these modest system sizes, where DMRG with higher bond dimension is already feasible, increasing DMRG bond dimension is both straightforward and effective. We do not see a compelling speed or efficiency advantage for RL under these circumstances. In our view, RL’s potential value might be in scenarios with stronger resource constraints either larger or more entangled systems.

Additionally, we would like to clarify how our timing data relates to the question of estimating quantum runtime. The "estimate of runtimes for computing the expectation values" reported in the table above reflects the total simulation time required to complete one RL episode on our classical simulator. Each episode in our setup is divided into 50 epochs, where in each epoch, expectation values must be computed to update the RL policy. Thus, as an estimate, the time required to compute the expectation values in a single epoch on our simulator corresponds to "time per episode" divided by 50. For example, in the $2^4$ bond dimension case, this amounts to roughly 2.18 seconds per epoch on our simulator.

While this episode timing is specific to our classical simulation environment and does not directly translate to quantum hardware, it provides an indirect indication of the runtime required per epoch for expectation value estimation, which we hope is helpful for benchmarking and planning for future quantum implementations.

Regarding the question of computational bottlenecks we would like to clarify that we are using a relatively modest HPC setup: each run was carried out with a single A100 GPU, 2 CPU cores, and 4GB RAM per core. For the system sizes presented here (up to 12 qubits), most runs complete well within the requested wall time. However, achieving higher accuracy-such as reaching error below chemical accuracy (1.6×10⁻³)-can require considerably more resources, and in some cases the wall time may approach the 48-hour maximum allowed on our available cluster. Thus, the computational cost in practice depends not just on system size, but also on the desired accuracy threshold. Moving forward, we plan to further reduce these computational requirements and explore more efficient quantum circuit simulators.

With respect to your summary question on applicable regimes: Based on our results, we do not find an advantage for RL over DMRG in the currently accessible, classically tractable regime. The RL approach is mainly motivated by scenarios where DMRG becomes infeasible due to system size or entanglement-potentially relevant for future quantum hardware (NISQ or fault-tolerant), though we acknowledge that the sampling overhead and training costs on such devices are still an open research challenge. At this stage, our work serves as a proof of concept and benchmark; further investigation is needed to pinpoint the hardware requirements and practical regimes where RL could provide clear benefits beyond classical methods. We appreciate your emphasis on this important point.

We hope that our response clarifies your main concerns. Kindly let us know if you have any more comments or suggestions.

Thank you very much for your careful and thoughtful review. We are grateful for your statement regarding the memory-time trade-off in our approach and your detailed analysis of the key scaling considerations. Your observations regarding how the optimization complexity depends polynomially on system parameters are consistent with our own findings, and have helped us reflect more deeply on the limitations of our method.

We fully agree that these factors, in practice, present significant constraints on scalability. As you rightly point out, the memory-time trade-off enables us to reach useful approximations with lower classical resources initially, but it does shift the primary bottleneck to the subsequent quantum optimization stage. While our results suggest that this hybrid strategy can offer practical benefits in select cases, particularly when classical DMRG requires very large bond dimensions—we recognize that these improvements are often modest and highly dependent on the specific system under study. We agree this does not represent a universal advance over classical DMRG.

Taking your valuable feedback into account, we will be careful to make these trade-offs and scaling limitations more explicit in the camera-ready version. In particular, we plan to adjust the title to use “enhanced” instead of “scalable” to more accurately convey the nature of our contribution, as you have suggested. We feel this change better reflects both the spirit and the scope of the work.

We sincerely appreciate your constructive comments, which have been invaluable in helping us better articulate the contributions of our work and clearly define its limitations. We are very grateful for your updated scores.

Responsible reviewer！Kind reviewer！

Thank you very much for your thoughtful feedback. We appreciate your acknowledgment of the novelty in integrating tensor network methods and reinforcement learning. In the following we systematically address your concerns:

...To better isolate the value of RL, could you add a "TN-VQE" baseline? This would involve using the COBYLA optimizer directly to optimize the newly added gates, without any RL framework...

Thank you for your insightful suggestion. In response, we have added the "TN-VQE" baseline to our results table, as shown below. This baseline utilizes the DMRG-initialized circuit as a starting point and employs the classical COBYLA optimizer to optimize each newly added gate, without RL framework.

The table show that TN-VQE consistently underperforms the RL-based approaches, both in terms of solution accuracy (error) and circuit efficiency (depth, CNOT, and ROT). Notably, for the 6-, 8-, and 12-qubit molecules, TN-VQE is unable to find the ground state with comparable accuracy to the RL-based methods, and in some cases fails to solve the problem. For the 10-qubit, TN-VQE does find the ground state, but the achieved accuracy remains significantly worse than other strategies. This demonstrate that having a high-quality initialization alone is insufficient; the RL framework provides a substantial advantage by better guiding the circuit construction and parameter optimization. Thus, our RL approach is not merely leveraging a good starting point, but also substantially improves the search strategy, enabling the discovery of circuits that are both more accurate and more efficient. We will update the Table 1 with these results in the Results section.

...How much does this choice affect the final performance? Have you explored other mapping schemes, such as generating deeper or differently structured initial circuits? Could the current results be overstated due to this highly optimized initial structure?

Thank you for raising this important point. Due to time constraints and substantial code changes required for generating differently structured initial circuits, we were unable to explore alternative circuit structures in this work. Instead, we focused on varying the number of TN layers (1-3) to increase the circuit depth within the same mapping, and evaluated across 6-, 8-, and 10-qubit cases. We acknowledge this limitation and plan to investigate different TN circuit structures in future. The RL performance and circuit parameters are in the table below:

For 6 qubits, both 2- and 3-layers TN initializations perform well, yielding similarly high success rates and competitive circuit costs; however, for 8- and 10-qubit, increasing the TN initialization depth to 3-layers actually causes the RL framework to fail (resulting in zero successful episodes), while the 2-layer initialization maintains reasonable performance. This indicates that deeper or more complex initial circuits, in this brickwork pattern, can hinder rather than overstate algorithmic performance. Thus, our results suggest that the method’s success is not simply due to an overly optimized or "overfitted" initial structure; rather, our chosen TN initialization offers a practical balance between expressivity and learnability. While further exploration of diverse and deeper circuit mappings is a valuable direction for future work, these findings demonstrate that our current initialization does not artificially inflate our results. We will include these additional results and analyses in the appendix in revised manuscript.

...How does the search difficulty of this new problem compare to the original one?

Thank you for your important question. We would like to clarify that the TN initialization provides a warm start for the RL agent by supplying good initial parameters, potentially allowing more efficient convergence by helping avoid less promising regions of the optimization landscape. However, since the underlying Hamiltonian, whether for quantum chemistry, TFIM, or Heisenberg, remains unchanged, the fundamental computational complexity of the ground state search (which is QMA-hard [[Kempe, J., Kitaev, A., & Regev, O. (2006). The complexity of the local Hamiltonian problem. SIAM Journal on Computing, 35(5), 1070-1097.]) is not altered; thus, while TN-initialization can make the process more practical, it does not change the intrinsic difficulty of the original problem. We hope this addresses the impact and intent of the TN-init method in our work. We will clarify this in the Methods section.

..The mere 1% improvement for the 20-qubit system in Appendix E.6 suggests this MPS structure may struggle to represent entanglement in more complex systems, casting doubt on its generality and true scalability.

Thank you for raising such a crucial question. It should be noted that the improvement (compared to the initial TN energy) depends on action space construction, the bond dimension in DMRG and number of layers in the mapping. Here to address your concern instead of considering (RX, RY, RZ, CX) as action space we add more expressive gates such as (XX, YY, ZZ) to the action space which enhances the improvement from 1% to 9%:

Hence the bottleneck of not improving in the ground state preparation task can be addressed by a well defined action space construction. In future work we plan to explore other factors mentioned above to achieve a greater improvement. We will update the Table in Appendix E.6 in the revised version of the manuscript.

The title argues for scalability, yet the experiments are demonstrated up to 20 qubits. In the current stage, "scalable" is more appropriately used for systems approaching 50+ qubits, making the claim seem premature.

Thank you for your feedback on our manuscript title. We appreciate your point that the original use of "scalable" may be premature. To more accurately reflect the current contribution we have revised the title to:

"TensorRL-QAS: Reinforcement learning with tensor networks for enhanced quantum architecture search"

We believe, this new title places emphasis on the methodological advancements our approach offers to quantum architecture search.

We hope these modifications are sufficient and addresses all your concerns. Kindly let us know if you have any more comments, questions or suggestions.

We sincerely thank the reviewers for their thoughtful and encouraging feedback. We greatly appreciate your recognition of the novelty of our proposed method, the comprehensive and rigorous nature of our experimental comparisons across RL-based and non-RL-based baselines, and our efforts in providing a reproducible codebase for the community. In the following we address your questions and suggestions.

Is it possible to cooperate the proposed method with real hardware noise model?

Thank you for this valuable suggestion. We fully agree that validating the proposed method on real hardware is crucial for demonstrating its practical applicability. Due to current limitations, most notably the long queue times associated with available QPUs, our primary studies focus on simulations using realistic noise models. To ensure robustness, we also evaluated the method under elevated noise levels (5x and 10x the baseline) to mimic harsher hardware conditions, as discussed in [Patel, Yash J., et al., "Curriculum reinforcement learning for quantum architecture search under hardware errors," ICLR (2024)]. As shown in that work, simulated noise models often yield trends consistent with actual QPU runs.

To further respond to your point, we conducted an additional experiment: we selected several randomly chosen successful RL-generated circuits (i.e., those achieving chemical accuracy) for 8-$H_2O$ found in noisy simulations (using depolarizing noise), and ran these circuits on $`IBMQ-Brisbane`$ using $`Qiskit`$. The results are shown in the table below:

These findings further demonstrate the practical relevance of TensorRL-QAS and support the observation from [Patel et al., ICLR 2024] that the best-performing circuit in simulation may not always be the best on real quantum hardware. This emphasizes the importance of evaluating candidate architectures directly on QPUs when possible and motivates our future work toward a tighter integration with specific hardware noise profiles. We will include this insights and results in the appendix of the revised manuscript.

How would the "fixed TN-init" model, which relies on preparing an initial statevector, be implemented in practice on current hardware? This step seems non-trivial and may introduce significant overhead not accounted for in the simulations.

Thank you for raising this important point regarding the practical implementation of the "fixed TN-init" model, which relies on preparing an initial statevector.

1. Compiling TN-init to QPU: In practice, we would compile the TN-init (tensor network-initialized) state into a quantum circuit using gates natively supported by the target hardware. This allows us to adapt the initialization procedure to the device-specific gateset while ensuring physical realizability.

2. Maintaining TN-init on QPU: If the compiled circuit for the TN-init differs in fidelity from the ideal statevector because of noise, we can employ error mitigation strategies, such as tensor error mitigation [Filippov, Sergei, et al. arXiv preprint arXiv:2307.11740 (2023).], to bring the prepared state as close as possible to the ideal TN-init.

3. TensorRL atop the TN-init: With the TN-init physically implemented as a compiled circuit and error mitigation in place to counteract hardware noise, we can then apply the TensorRL using this fixed initialization. The trained policy and subsequent circuit evolutions can thus be performed atop a stable, experimentally feasible TN-init, while continuing to apply mitigation strategies to maintain fidelity throughout the computation.

Overall, this approach is practically feasible, clearly at the cost of using error mitigation techniques —coming with their unavoidable overheads— to counteract hardware noise. Due to current limitations in accessing QPUs and limited time on GPUs from our side, a thorough exploration and experimental demonstration of this workflow is left as future work.

How does the performance of TensorRL-QAS change if a different TN-to-circuit mapping is used, or if a different tensor network structure is used for the warm-start?

Thank you for raising this important question. Due to time constraints, we were not able to comprehensively investigate alternative TN-to-circuit mappings or entirely different tensor network (TN) structures. However, we sought to probe related effects by varying the number of TN layers in our initialization (from 1 to 3 layers), which closely emulates changes in TN structure and, implicitly, circuit entanglement patterns. For each configuration, we measured the success rate, as well as circuit and error statistics for 6-, 8-, and 10-qubit molecules. The proposed RL agent circuit performance and parameters are summarized in the table below:

For 6 qubits, both 2- and 3-layer TN initializations perform well, achieving comparably high success rates and competitive circuit costs. However, for 8 and 10 qubits, increasing the TN initialization depth to 3 layers leads to RL failures (resulting in zero successful episodes), whereas the 2-layer initialization maintains reasonable performance. This indicates that deeper or more complex initial circuits, even within the same TN family, can hinder, rather than artificially boost, the overall algorithmic performance. Thus, our findings suggest our method’s success is not attributable solely to an over-optimized or "overfitted" warm-start structure. Instead, the chosen TN initialization achieves a practical trade-off between expressivity and learnability.

While a systematic exploration of alternative TN-to-circuit mappings and fundamentally different TN structures remains a valuable direction for future work, our present experiments with varied TN layers serve as an initial, closely related probe. We will include these additional results and analyses in the appendix of the revised manuscript.

We hope our response properly addresses all your questions and suggestions. Kindly let us know if you have any more questions and/or comments.

Dear reviewer,

We thank you again for your valuable feedback and suggestions on our submission and for raising important points. Below we have carefully addressed each of your comments with detailed clarifications and additional analysis. We kindly invite you to review our responses and share your thoughts, especially regarding the practicality of our proposed approaches on current quantum hardware and their possible overheads. Your further evaluation and guidance will be greatly beneficial to enhance both the work and its presentation.

We look forward to your feedback and a fruitful discussion.

Warm regards,

Thank you very much for your thoughtful and positive feedback. We are pleased that you find TensorRL-QAS interesting and promising. Moreover we appreciate your positive remarks on the clarity, reproducibility, and robustness. In the following we systematically address all your questions and suggestions.

Can the proposed method be extended to larger systems beyond 12 qubits?

Thank you for the question. In the current article in appendix we show that TensorRL can be utilized upto 20 qubits, we present these results explicitly in the table below:

The improvement is over the TN initialization energy.

What is the challenge to present practical experimental results using the proposed method?

The main challenge of our current approach is the computational bottleneck in the RL training phase. More specifically, each episode requires simulating and optimizing a parameterized quantum circuit, and computing its expectation value multiple times (equal to the number of steps i.e. the maximum depth of the circuit set as a hyperparameter). Already for ~12 qubits, this becomes computationally intensive due to the number of episodes required for convergence/solving the task and the increasing number of query to the quantum simulator.

Why is RL especially useful for the given task?

We thank the reviewer for this valuable question. To show the effectiveness of the RL agent we conduct an experiment where instead of utilizing an RL algorithm to choose and optimize quantum gates we use a hardware-efficient ansatz (HEA) after the fixed TN-init (namely TN-VQE). Each gate of the HEA is explicitly optimized by COBYLA optimizer individually, the results are summarized below:

This reveals that without an RL training we can optimize the DMRG-initialized circuit to a certain extent but it consistently underperforms the RL-based approaches, both in terms of solution accuracy (error) and circuit efficiency (depth, CNOT, and ROT counts). Notably, for the 6-, 8-, and 12-qubit molecules, TN-VQE is unable to find the ground state with comparable accuracy to the RL-based methods, and in some cases fails to solve the problem altogether. This solidifies the fact that having a high-quality initialization alone is insufficient; the RL framework provides a substantial advantage by better guiding the circuit construction and parameter optimization. We will add these insights in the Results section in the revised version of the article.

The authors have demonstrated some results about noisy cases, while it would be useful to provide some experimental results on real quantum device to demonstrate its practical use.

Thank you for this valuable suggestion. We agree that experimental validation on real quantum hardware is important for demonstrating practical relevance. Due to current limitations, primarily long queue times for available QPUs, we focused our main studies on simulations with realistic noise models, including tests with noise levels elevated by 5 and 10 times to probe the robustness of TensorRL-QAS. As shown in [Patel, Yash J., et al., "Curriculum reinforcement learning for quantum architecture search under hardware errors," ICLR (2024)], simulated noise models generally align closely with results from real QPUs. To further address your point, we also evaluated randomly chosen successful (which passes the chemical accurayc) RL circuits generated from noisy simulation with depolarizing noise for 8-$H_2O$ and then run the circuit on $`IBMQ-Brisbane`$ with $`Qiskit`$; the results are displayed below which further supporting the practical applicability of TensorRL-QAS.

These results solidifies one of the claim in [Patel, Yash J., et al., "Curriculum reinforcement learning for quantum architecture search under hardware errors," ICLR (2024)] that the best performing circuit in simulated scenario might not be the best performing in QPU.

...the authors may consider to include the results of a pure TN-compiled circuit without RL refinement.

Thank you for your suggestion regarding a more detailed ablation study. In response, we provide a table of ground-state energies using only pure TN-compiled circuits without RL refinement (TN-energy) for several molecular systems. The error from the reference (true energy) energies serves as a baseline for our method, isolating the contribution of TN compilation alone.

It would be useful to broaden the baseline pool: add recent evolutionary-search and gradient-based QAS methods.

Thank you for the valuable suggestion. In the revised version of the manuscript we added a gradient-based QAS (GQAS) [He, Zhimin, et al. Neural Networks 179 (2024): 106508.] and DQAS [Zhang, S. X., Hsieh, C. Y., Zhang, S., & Yao, H. (2022). Quantum Science and Technology, 7(4), 045023.] in the table below. This will be added in the appendix. Unfortunately for evolutionary-based QAS we could not find any benchmarks that is in line with our optimization method that is why we cite the reference [Zhang, Anqi, and Shengmei Zhao. arXiv preprint arXiv:2212.00421 (2022).] in the Related works.

We hope that our response states all the concerns and questions raised by you. Kindly let us know if you have any more suggestions and questions.

Thank you for helping us strengthening our work. We are happy that you appreciated our response. We will include these changes in the camera ready version. Kindly let us know if you have any further feedback.