Reinforcement Learning for Quantum Control under Physical Constraints

Reviewer K5by

Thank you for recognising our incorporation of physical constraints, which significantly advance the state of the art, as a valuable contribution.

Weaknesses

"contributions are somewhat limited, as the approach builds on a standard RL framework for quantum optimal control, incorporating two novel techniques (limiting the agent’s action space and shaping the rewards)."

A4.1 While we agree that our work lies within applied machine learning, our implementation of physics-informed constraints enables a highly-parallelisable learning algorithm for arbitrary quantum systems, achieving state-of-the-art fidelities. This facilitates easy applications of RL to pulse-controlled quantum systems, aiding in the discovery of optimal and robust solutions, significantly surpassing other RL and non-RL methods.

To highlight our contribution, we have conducted an ablation study in the noise-free Lambda system explained in Sec. 5.1. We replace PPO with two alternative algorithms, DDPG [1] and TD3 [2]. Additionally, we compare to a vanilla version of each algorithm, where the reward function includes only a linear fidelity term (as done in many previous works, e.g. [3]).

The ablation results (see Table below) highlight the efficiency and effectiveness of our approach in achieving high-fidelity solutions. Notably, none of the vanilla algorithms achieve a fidelity above 0.99. Our PPO implementation significantly outperforms both DDPG and TD3, reaching fidelity >0.99 in less than 1% of the time. In our revised manuscript, we will include this baseline comparison along with a plot illustrating the learning dynamics.

"Due to data (interaction) driven nature of the approach this method will require huge numbers of uses of a quantum device and/or unrealistically strong simulators."

A4.2 While our method requires a large number of quantum device interactions, it has been demonstrated in [4] that this is feasible in real world systems, given their fast gate times.

We also want to highlight that most quantum systems have extremely well understood theoretical models, once the model parameters are correctly identified, our method can be applied to real physical devices. This is demonstrated in prior work, where good agreement between experimental data and simulation results is shown [5, 6, 7].

I am not able to assess how impressive the experimental results are due to my limited familiarity with the field

See A4.1 for RL baselines. We also wish to highlight that we outperform alternative optimisation methods on the respective problem settings in Sec. 5 with superior fidelities and noise robustness. Moreover, the methods we introduce for pulse-control of physical systems can find application in various fields of physical science such as in Optimising Chemical Reactions [9] and High Energy Physics [8].

References:

[1] Lillicrap et al. 2016, 'Continuous Control with Deep Reinforcement Learning', ICLR 2016

[2] Fujimoto et al. 2018, 'Addressing Function Approximation Error in Actor-Critic Methods', ICML 2018

[3] Bukov, Marin, et al. Physical Review X 8.3 (2018): 031086.

[4] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.

[5] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.

[6] Willsch, Dennis. arXiv preprint arXiv:2008.13490 (2020).

[7] Zhang, X. L. et.al. Physical Review A, 85(4), 042310.

[8] Capuano F. et. al. arXiv preprint arXiv:2503.00499 (2025).

[9] Zhou, Z. et. al. ACS Central Science (Vol. 3, Issue 12, pp. 1337–1344). American Chemical Society (ACS) (2017).

Reviewer 8yQf

We thank the reviewer for recognising our integration of physical constraints into RL, which achieves high-fidelity, noise-resilient solutions.

Weaknesses

"The term "physics-constrained" is used throughout the paper but is not adequately defined or justified."

A3.1: Thank you for raising this point, we refer to A1.3 in the response to sD3o and provide additional detail in A3.7.

"The computational efficiency claims are not convincingly demonstrated. "

A3.2: We acknowledge the lack of timing comparison in our work and provide it here for the results in Table 2 (Sec. 5.1) (with one additional comparison) for the noise-free λ System Optimisation. On a Mac M1 2020 CPU, our highly parallelisable implementation achieves up to 30x faster runtimes for a single run, while also yielding the highest fidelity.

We further show speedups over the baseline time by up to two orders of magnitudes with GPU based parallelisation in Appendix Fig. 11.

Questions

How does the proposed method scale to larger quantum systems with more qubits or higher-dimensional state spaces?

A3.3 Thank you for bringing this up, see response A1.4 to sD3o.

How does the proposed RL approach compare with other existing RL methods for quantum control tasks? [...] "the results are not compared to a wide range of existing methods, especially the existing RL methods for quantum control." [...] "For example, [2], [3]"

A3.4: [2] uses discrete actions, unsuitable for our real-world quantum control problems requiring continuous actions. Hence we cannot benchmark against it, but have added it to our related work section.

Unfortunately, [3] does not provide an open source implementation and the paper does not provide sufficient detail to allow reimplementation, hence, we are unable to benchmark against it.

We want to highlight that we already included [3] in our related work section. We now include a more comprehensive discussion, highlighting its computationally less efficient approach of learning control signal parameters individually at each signal time step.

We are confident that our work includes empirical comparisons to all relevant baselines, and we kindly ask the reviewer to specify if any comparisons were missing. We now also include benchmarks against DDPG and TD3 and refer the reviewer to A4.1 in response to K5by.

How to verify the performance of the proposed approach on real physical systems?

A3.5 Most quantum systems have well established theoretical models. Given the system parameters, learned pulses can be directly applied to real devices; many prior works show good agreement between simulation and experiment [4,5,6].

In future work, we are planning to extend our experiments to real world devices. Unfortunately, IBM has discontinued pulse-level access, so we are pursuing Rigetti [7] as an alternative.

How well does the proposed RL method generalise to other types of quantum systems?

A3.6 Our model-free approach generalises to any quantum system that can be simulated. Our method can also be extended to black box sampling from real physical devices; the feasibility of this has been shown in [1].

How sensitive is the method to the choice of physical constraints such as maximum solver steps? How to choose proper physical constraints?

A3.7 Our ablation study (App. Fig. 12 and 13) shows that larger smoothing kernel standard deviations and smoothing penalties boost fidelity while keeping signals within the bandwidth limits (typically a few hundred MHz) of standard electronics. In the revised manuscript, we will expand on App. Sec. F.3 to explain how we choose physical smoothing constraints that are consistent with electronic bandwidth limitations.

Limiting solver steps improves compute time without affecting optimal dynamics if steps are sufficient. We set N_max based on the adiabatic condition, determined by the maximal effective Rabi frequency (N_max ≳ 1/Ω_eff) and increase it until infidelity significantly drops, as outlined in lines 206–214.

Concluding remarks

We kindly ask the reviewer if we’ve addressed their concerns by (1) adding RL benchmarks and distinguishing our work from existing RL work, (2) including the requested timing comparison, and (3) explaining physics-driven constraints and their effects. With these updates and new results, we politely ask the reviewer to reassess their score or highlight remaining issues.

References:

[1] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.

[2] Niu et al 2019, npj Quantum Information

[3] Ma et al. 2023, IEEE TNNLs

[4] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.

[5] Willsch, Dennis. arXiv:2008.13490 (2020).

[6] Zhang, X. L. et. al. Physical Review A, 85(4), 042310.

[7] Rigetti Computing. (2025). pyQuil 4.16.1.

Reviewer vWkD

Thank you for recognising our rigorous adaptation of RL for quantum control and well reasoned introduction of constraints.

Weaknesses

"Missing discussion of prior RL-based quantum computing research". [...] "... Prior work such as [1-4] should be discussed..."

A2.1: We thank the reviewer for highlighting these works; we will discuss them in the related work of the updated manuscript. We remark that these works do not address the same problem setting but instead focus on circuit-level quantum compilation (in a space of logical quantum gates) which is fundamentally different from the pulse-level control problem (in a space of physical control signals) that our work addresses. Hence, these works do not constitute approaches that we could benchmark against. We mention in the conclusion that expanding our pulse level control to multiple sequential pulses (i.e. a circuit) would be an interesting extension.

"No code provided, reducing reproducibility and practical applicability." [...] "The absence of released code raises concerns about reproducibility and the ability to benchmark against alternative methods.""

A2.2: We acknowledge the reviewers' concern and are committed to reproducibility. To allow for a better review process, we have anonymously published the code under https://anonymous.4open.science/r/RL4qcWpc/README.md. We will release all code publicly upon acceptance, as stated in Section 5.0.1.

"The paper would benefit from additional benchmarking against existing RL-based quantum approaches..."

A2.3: As pointed out in A2.1 above, the highlighted related works [1-4] do not address the same problem setting as our work.

We are confident that our work includes empirical comparisons to all relevant baselines and politely ask the reviewer to specify if there were any missing comparisons.

"It is unclear whether the implementation supports standard RL frameworks, which would facilitate broader testing."

A2.4: Our implementation supports any RL framework, as can be seen in the published code. To emphasise this, and to further motivate the choice of PPO, we have now conducted an ablation study on replacing PPO by DDPG [5] or TD3 [6]. Our implementation of PPO achieves fidelities of over 0.999 for the experimental setup described in Sec. 5.1, whereas both DDPG and TD3 achieve maximum fidelities of 0.995 and 0.994 and take 100x longer to reach >0.99 fidelity on the same GPU. See A4.1 in response to K5by for more details.

"Details on the specific RL algorithm employed and justification for its suitability to this problem are lacking."

A2.5: Thank you for this feedback, we have added motivation for using PPO, an on-policy algorithm chosen for stability (less hyperparameter sensitivity) over DDPG [5] or TD3 [6]. Furthermore, we show that in our quantum control setting, PPO significantly outperforms alternatives, see A2.4 and A4.1 for more details.

"The paper is heavily skewed towards quantum-specific adaptations, limiting its ML relevance for ICML".

A2.6: We believe that the paper's focus on physics-constrained RL for Quantum Control exactly fits the scope of ICML defined in the call for papers. Specifically, ICML calls for “Application-Driven Machine Learning [...] driven by needs of end-users in applications [...] such as physical sciences.” Also, see A4.1 for an ML specific benchmark.

Questions

"What do the authors mean by "interpretable quantum state dynamics" in this context?"

A2.7: It refers to the clear, interpretable time evolution of a quantum system under optimised control (e.g. minimal excited state population in a Lambda system), unlike complex, fast dynamics induced by many unphysical optimal control solutions. A plot contrasting these will be added to the revised manuscript.

"Do the authors plan to release their implementation... will it support standard RL libraries for benchmarking?"

A2.8: As noted in A2.2, the code is now anonymously available and supports standard RL libraries, shown via ablations with DDPG [5] and TD3 [6] vs. PPO (see A2.4).

Concluding remarks

We kindly ask the reviewer if we were able to address their concerns by (1) clarifying the difference to [1-4], (2) making our code available, (3) demonstrating adaptability through comparisons with other RL algorithms. In light of these clarifications and additional results, we politely ask the reviewer to consider updating their score, or to point out further concerns.

References:

[1] van der Linde et al., 2023: RL-based quantum compilation benchmarking

[2] Altmann et al., 2024: Challenges of RL in quantum circuit design

[3] Kölle et al., 2024: RL environment for quantum circuit synthesis

[4] Rietsch et al., 2024: RL for unitary synthesis of Clifford+T circuits

[5] Lillicrap et al. 2016, 'Continuous Control with Deep Reinforcement Learning', ICLR 2016

[6] Fujimoto et al. 2018, 'Addressing Function Approximation Error in Actor-Critic Methods', ICML 2018

Reviewer sD3o

We appreciate your positive feedback on our work and are glad that our research in AI for quantum science was well received.

Weaknesses

"The proposed method requires precise calibration of the target system, and after each calibration the RL algorithm needs to be rerun to obtain high-quality pulses. I wonder, when integrated in a realistic system where calibrations are conducted routinely, how will the proposed RL algorithm perform (in the sense of run time and stability) in real devices."

A1.1: We thank the reviewer for this insightful question regarding the handling of system parameter changes after re-calibration. For small drifts the inherent robustness of our learned policy (achieved via training in noisy environments) can avoid requiring retraining. For large deviations in system parameters, our method can be adapted by training an RL policy that conditions on the current system parameters. This policy is trained by sampling environments with varying system parameters, enabling it to adapt to novel system parameters when deployed.

We would like to point out an additional scenario that we observed in our experiments, where the learned policy can be attributed to symbolic equations (Appendix Section E.1), as is the case for the learned transmon reset pulse. Only requiring four parameters to define, the learned pulse can be easily tuned in a physical experiment and does not require re-training.

"The experiments in the paper are conducted with numerical simulations. For a technique targeting quantum control, it is essential to validate the method on real quantum devices... hard for the authors to access the limited hardware resource, while I still want to know if the proposed method can be applied to accessible commercial quantum devices, i.e., IBM’s devices."

A1.2: Unfortunately, IBM does no longer allow pulse-level access to their devices. For future work, we are investigating alternative hardware providers which offer pulse level device access, including Rigetti [1] and IQM [2] to verify our solutions. Furthermore, we want to highlight that prior work found good agreement between numerical simulations of and experiments with transmon qubits [3,4].

"Following point 2, the constraint design in the framework is not convincingly justified in real device experiments. In this sense, the paper lacks an important discussion on the criteria for selecting these constraints..."

A1.3: Thank you for this remark. We refer to the introduced method as "physics-constrained", as all constraints are derived from real, physical limitations.

First, we constrain signal bandwidth (smoothness), and signal area, reflecting the limited instantaneous bandwidth of electronics and signal components in experiments, and limitations in available signal power and duration. We implement these constraints via the reward function (Eq.2), similar to the Lagrange Multiplier technique introduced in [5].

Second, we constrain the policy to solutions that can be simulated within a predefined number of maximum solver steps. This constraint incorporates priors about the physical solution time scales into the algorithm (see lines 209-13), which incentivises adiabatic quantum state dynamics, yielding robust and interpretable solutions. Additionally, this constraint incentivises smoothness (requiring lower bandwidth) of signals, as smooth signals generally require fewer solver steps. Furthermore, the constraint on maximum solver steps significantly reduces computational demand.

We will include this explanation in Sec. 4.1 in our revised manuscript to make it clearer for the reader. We will also add a figure in Appendix Sec. F.3 which shows how smoothing constraints translate to real signal bandwidth limitations using an FFT analysis.

"Since the RL algorithm relies on a full simulation of the system, it is hard in its current form to scale up, i.e. to more than 10 qubits."

A1.4: We agree with this observation. However, one could replace a full system simulation with an ML based emulator, or direct physical device sampling [6]. Given such improvements, our proposed method could scale to larger systems. We also want to highlight that control of smaller dimensional systems is an important and relevant research direction for advancing quantum technologies.

Minor comment

Thanks for spotting this, the third term intentionally has the same structure but there is a typo $S(\Omega_i)$ should read $S(\Delta_i)$.

References:

[1] Rigetti Computing. (2025). Pulses and waveforms. pyQuil 4.16.1. https://pyquil-docs.rigetti.com/en/stable/quilt_waveforms.html

[2] IQM. (2025). IQM Pulla 6.15. https://docs.meetiqm.com/iqm-pulla/

[3] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.

[4] Willsch, Dennis. arXiv preprint arXiv:2008.13490 (2020).

[5] Bhatnagar, S., Lakshmanan, K. J Optim Theory Appl 153, 688–708 (2012).

[6] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.