QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Multi-ranged Context Learning Baseline

We would like to thank the reviewer for the insightful comments and inquiries, and the positive evaluation of our work. We have summarized the newly added Figs and Tabs at this link.

Below are our responses.

1. Real devices other than IBM Kyiv and traditional QEM techniques like ZNE on real devices.

We apologize for the earlier oversight. We have now incorporated the following enhancements:

• Two datasets derived from 63-qubit Trotter circuits executed on IBM Brisbane have been included—one with extreme outlier filtering (Brisbane Pre) and one without (Brisbane Raw).
  • Dataset statistics are detailed in Tab. 1.
  • The performance of various QEM techniques is provided in Tab. 2 and Fig. 1.
• We have evaluated ZNE across the datasets from Kyiv (Pre and Raw) and Brisbane (Pre and Raw), as detailed in Fig. 1-2 and Tab. 2-3.

We would like to note that the CDR approach entails significant time costs due to the need to construct training sets for each circuit individually. Consequently, we plan to include its results in future revisions.

2. Ablation studies of QEMFormer

We now conduct two sets of analyses.

• Tab. 4 compares the performance impact of the MLP and Graph Transformer modules.
• Tab. 5 evaluates our multi-ranged feature extractor.

3. RMSE as the sole evaluation metric?

We apologize for any potential confusion. In the original manuscript, we report RMSE (Tabs. 1 & 2), MAE (Tabs. 5 & 6), and standard deviation (Tabs. 1 & 2) to provide a comprehensive evaluation. RMSE emphasizes larger deviations and is sensitive to outliers, while MAE directly measures the average error magnitude. Additionally, violin plots (Figs. 3 & 6 of the original manuscript) depict the full error distribution along with the STD.

4. About prior QEM benchmarks.

To the best of our knowledge, there are currently no standardized benchmarks for the ML-QEM task. This gap, as also noted by reviewers Qjn1 and 6Drn, has been a primary motivation for this work.

5. Dataset curation process and whether it reflects real-world noise

We detail the dataset curation process in Tab. 6. Our design intentionally reflects real-world noise characteristics in two key ways:

• Diverse Circuits: A single type of noise impacts various circuits differently. By including types of structured circuits and random unstructured circuits, QEM-Bench aims to gather a comprehensive depiction of real-world conditions.
• Broad Noise Modeling: We incorporate multiple realistic noise sources, including data from real devices (Kyiv), fake providers (published by IBM) resembling the real devices, and manually set configurations based on statistics from representative real devices (e.g., Sycamore [1] for incoherent settings).

This design aims to let the QEM-Bench effectively mirror the diverse noise characteristics encountered in practice and bridge the current gap for further research.

6. Hyperparameter tuning details.

We include an analysis of how the number of layers and hidden dimensions in the MLP and Graph Transformer modules affect QEMFormer's performance (see Fig. 3). Our findings indicate that increasing the model size initially enhances model capability; however, excessively large models can lead to overfitting and degraded performance. We also politely noted that hyperparameter settings are provided in Tab. 7 of the original manuscript.

7. Formal theoretical analysis.

We appreciate the reviewer's suggestion. Our primary focus in this work is to introduce a comprehensive benchmark dataset that spans a variety of circuits and noise configurations, addressing a significant gap in ML-QEM studies.

On the method side, we specifically design a 2-branch neural network architecture in QEM tasks. The idea is that the multi-range context and dependency as frequently occurred in the quantum system can be better captured, and the effectiveness is also verified by our extensive empirical results. We plan to incorporate further qualitative analysis in the revised manuscript.

Due to the inherent complexity of quantum systems and noise profiles, we leave a rigorous theoretical analysis of QEMFormer’s effectiveness for future work.

8. Discussion about hardware-specific noise mitigation and circuit optimization techniques.

We will expand the related works section to include discussions on hardware-specific noise mitigation techniques, hybrid classical-quantum optimization strategies, and ML-based circuit optimization methods. We will clarify the differences and relationships between these approaches and ML-QEM techniques, especially QEMFormer.

We hope the reply eases your concern. If you have any further questions, we would be pleased to respond.

References:

[1] Quantum supremacy using a programmable superconducting processor, Nature volume 574, 2019.

We appreciate your constructive feedback and positive evaluation of our work. We acknowledge that quantum error correction is aimed to be solved on the hardware side ultimately. However, the significant qubit overhead associated with QEC renders it less feasible in the near term, especially for large-scale circuits, as illustrated in [1] and [2]. Therefore, during the NISQ era, ML-QEM methods offer an effective and efficient interim approach, and we hope our work lays a foundation for future studies.

We will revise the manuscript to clearly reflect this nuanced perspective and incorporate your valuable suggestions.

References:

[1] Quantum Error Mitigation, Reviews of Modern Physics, American Physical Society, 2022

[2] Near-term quantum computing techniques: Variational quantum algorithms, error mitigation, circuit compilation, benchmarking and classical simulation, Science China Physics, Mechanics & Astronomy, 2023.

We would like to thank the reviewer for the insightful questions and positive evaluation of our work. We summarize the Tabs. and Figs. for newly added experiments at this link.

Below are our responses to each question.

Q1: Why does the paper evaluate IBM Kyiv only? There are other quantum computers in the IBM Quantum system.

Initially, we evaluated only IBM Kyiv due to the significant financial and time costs associated with executing large-scale quantum circuits on real quantum devices. Although multiple IBM devices are available, these constraints still limited our testing.

To improve the diversity of QEM-Bench and strengthen your confidence in our work, we have expanded our evaluation to include two datasets from the IBM Brisbane device: one with extreme outlier filtering (Brisbane Pre) and another with raw, unfiltered data (Brisbane Raw).

• Detailed statistics for the two datasets are provided in Tab. 1
• The performance of various QEM techniques is shown in Tab. 2 and Fig. 1.

Notably, even with the more severe noise effects on the Brisbane device (compared to Kyiv), QEMFormer consistently outperforms the baseline methods.

We shall include these datasets and results in the revised manuscript.

Q2: Is the evaluation on a 50-qubit system good enough? Why? If the circuits are deployed on a 127-qubit system, will the performance stay the same?

We would like to clarify that the construction of our datasets on 50-qubit systems is limited by the prohibitive computational cost of obtaining ideal EVs for circuits with over 100 qubits. Although devices like Kyiv and Brisbane support up to 127 qubits, their outputs are inherently noisy. Hence, ideal EVs, which serve as the dataset labels, have to be obtained through classical simulation. Yet, to the best of our knowledge, the IBM simulators that do not restrict circuit structure to provide ideal simulation are limited to 63 qubits (namely, Aer matrix_product_state simulator), making 100-qubit simulations difficult under the current implementation structure of QEM-Bench.

Furthermore, to the best of our knowledge, the only ML-QEM method exploring circuits beyond 100 qubits, [1], also demonstrates such difficulty in ideal EV obtaining in their work and thus uses ZNE-mitigated results as training labels. However, our experiments applying IBM’s built-in ZNE to both 50-qubit and 63-qubit circuits (on the Kyiv and Brisbane devices) show only marginal improvements over noisy outcomes, with significant residual errors (see Fig. 1–2 and Tab. 2–3). This suggests that using ZNE outcomes as labels may not provide a fair or reliable benchmark for larger systems.

Accordingly, due to the current time constraint, we plan to extend the inclusion of systems exceeding 100 qubits to future work. Nevertheless, to enhance your confidence and further evaluate the scalability of various ML-QEM methods, we have developed the Brisbane Pre and Brisbane Raw datasets on 63-qubit systems (results summarized in Tab. 2 and Fig. 1).

We hope the reply eases your concern. If you have any additional questions, we would be pleased to provide further responses.

References:

[1] Machine learning for practical quantum error mitigation, Nature Machine Intelligence, 2024

Thank you for your thoughtful comments and inquiries. We summarized all Tabs. and Figs. of the newly added experiments at this link. Below are our responses.

1: RF [1] and GTraQEM [2] on QEM-Bench.

We apologize for the oversight and now implement the RF in [1] and GTraQEM in [2] and evaluate them on 20 datasets from QEM-Bench (Figs. 1&2, Tabs. 1-3).

Although [1] reported that the RF outperformed the MLP and GNN on a simple 4-qubit random circuits dataset, we consider it not convincing to state that "the RF routinely beats the MLP and GNN in [1]" as no further comparisons are conducted in other settings in [1]. Our comprehensive evaluation shows that RF's performance degrades in advanced settings such as trotter zero-shot.

Also, though GTraQEM shows competitive performance in certain settings, its non-message passing aggregation incurs high computational costs with increasing circuit depth, as constructing its structural matrix has an $O(n^3)$ complexity.

2. How to obtain the ideal EV on the Kyiv device.

The ideal EVs were computed solely based on the circuit, independent of any quantum device. For 50-qubit circuits, we use the IBM Aer simulator for ideal simulation, specifically with:

simulator_ideal = AerSimulator(method='matrix_product_state')
estimator = Estimator(mode=simulator_ideal)
job = estimator.run([(circs, observables)])

3. Large Error Bars?

There may be confusion in our previous writing. To clarify, the reported error bars (in Tab. 2 & 3 of the original manuscript) refer to

$$\sigma_{\text{dataset}} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \left(y^{\text{miti}}_i - y^{\text{ideal}}_i\right)^2},$$

where $N$ is the number of test data points. This metric quantifies the deviations of noisy (or mitigated) results from the ideal values across the dataset. It is not a measure of model reproducibility across different random seed runs, computed by:

$$\sigma_{\text{stability}} = \sqrt{\frac{1}{K} \sum_{k=1}^{K} \left(\text{MAE}_k\right)^2} \quad \text{(or similarly for RMSE)},$$

with $K$ representing the number of runs.

To show this distinction, we executed QEMFormer under five different random seeds, summarized in Tab. 4.

We will clarify it in our main paper:

1. The models show a small $\sigma_{\text{stability}}$, indicating that the mean performance is reliable for comparison.
2. The relatively large $\sigma_{\text{dataset}}$ reflects the variable impact of noise on different circuits, leading to diverse deviations from the ideal outcomes.

4. Difference between GNN in [1] and QEMFormer

We would like to note that QEMFormer is an integrated architecture rather than a mere refinement of the GNN in [1]. A comparison is shown in Tab. 5. Importantly, QEMFormer experimentally outperforms the GNN in [1] among most settings, demonstrating its two-branch design's inherent well-suitedness for the quantum system.

5. How QEM-Bench came up with?

We apologize for the unclear expression. Regarding the design of the QEM-Bench:

1. QEM-Bench is built on insights from existing QEM and quantum computing research to ensure it includes the key concerns of the community.

2. Key Enhancements:

   • Structural Diversity: Incorporates representative QAOA circuits.
   • Circuit Complexity: Enriches gate types and parameter selection in random circuits.
   • Noise Characterization: Include coherent noise.
   • Evaluation Scope: Expands zero-shot settings to test generalization ability.
   • Real-World Data: Construct large-scale circuit datasets executed on quantum devices with ideal EVs as labels.

By integrating these enhancements with established research, QEM-Bench aims to address the need for a standardized benchmark evaluation dataset for ML-QEM techniques.

6. Are noise parameters fixed across error models?

We politely clarify that QEM-Bench does incorporate different noise strengths across error models. Incoherent noise parameters are derived from the real device, the Sycamore [3], and the parameters of real devices and simulators provided by IBM are not manually set, thereby capturing a realistic and diverse range of noise regimes.

7. How far do the noisy EVs differ from the ideal EVs?

We respectfully note that the error distributions, MAE, and RMSE of raw data are detailed in Tabs 2-4 and Figs 3, 4 & 6 of the original manuscript.

We hope the reply eases your concern. Should you have any further inquiries, we would be pleased to offer responses.

References:

• [1] Machine learning for practical quantum error mitigation. Nature Machine Intelligence, 2024
• [2] Beyond circuit connections: a non-message passing graph transformer approach for quantum error mitigation. ICLR 2025.
• [3] Quantum supremacy using a programmable superconducting processor, Nature 574, 2019

    t_stat, p_value = stats.ttest_rel(baseline_err_array, ours_err_array,  alternative='greater'),