End-to-End Learning Framework for Solving Non-Markovian Optimal Control

Thank you for your careful reading and constructive suggestions. Below, we provide a detailed response regarding your concerns and questions.

Baselines: As detailed in our response to Reviewer cBiX, we have conducted extensive comparisons between FOLOC and several widely used methods, including MBRL, CMA-ES, MPC, PID, and iLQR.

Why learning based LQR? (i) FOLOC generalizes across different FOLTI systems (Appendix J.2), predicting system parameters and optimal controls (OCs) directly from trajectories. System parameters are not used during testing on synthetic data, and are not used entirely during both training and testing on real-world dynamics. (ii) It enables fast and accurate real-time control. Although closed-form solutions (Eqs. 16 and 17) exist given full system knowledge, they involve large-scale matrix inversions and become impractical for long horizons or high-dimensional systems. MPC, though an alternative, also struggles with the memory effects in fractional-order systems. FOLOC overcomes these challenges by learning a direct mapping from trajectories and cost matrices to OCs.

Figures: We modified our manuscript to address your concerns about figures: we (i) clarified in Fig. 2’s caption that the sample complexity refers to the Lagrange multiplier-based solution, (ii) updated Fig. 4 by removing the secondary y-axis and aligning both error metrics on a shared axis, and (iii) revised Fig. 1’s notation in alignment with Eq. (29).

Joint optimization pipeline: The sys-id loss is a physical constraint for our model that encourages the model to accurately identify the system, further ensuring accurate control, the ultimate goal of our method. During the training period, we use the sys-id label as guidance to ensure accurate sys-id, and we do not need these labels in the test period. Some methods, such as PINN [1], couple multiple loss terms by incorporating physical constraint losses. This helps the system learn representations that are both physically grounded and task-relevant, reducing error propagation and avoiding reliance on purely minimizing prediction error. For real-world datasets where the system parameters are unknown, we can first pretrain the model on synthetic datasets with known parameters($A$, $B$, $\alpha$). Then, we do not need sys-id labels(set sys-id loss weight $λ=0$) and finetune the model on the real-world dataset. We'll provide results for $λ=0$ for comparison in Appendix H.1.

Notations $p$ and $N$: We have clarified in the revised version that $p$ denotes the number of initial conditions, and $N$ denotes the number of trials adding Gaussian noise to fractional-order dynamics from these $p$ initial conditions.

Assumption: This is a strong assumption and represents a limitation of the theoretical sys-id approach. It serves a similar role to assumptions commonly made in classical sys-id, where the exact (integer) order of the system is often presumed known in advance. In the context of fractional-order systems, estimating the system order—i.e., the fractional derivatives—is substantially more challenging, and to the best of our knowledge, no existing method provides theoretical guarantees in a general setting. To address this limitation, we propose FOLOC framework, which can give accurate OC solutions without this assumption.

Performance decrease: The decrease of the performance may be caused by the Joint training loss trade-off and Stochastic training dynamics and over-regularization, Despite this the variation in MSE/MAE across training sizes is small (e.g., 5.18 → 5.22 × 10⁻³ in MSE) indicate that the model exhibits both robustness and stability across different settings even with a few sample data The results are as follows, which show a significant drop in performance using fewer training samples.(i.e., 500) (All reported values are in units of 10⁻³.)

Designing of model architecture: Align with theoretical derivation in Eqs. (13)–(15), The system parameters($A, B, \alpha$) in physical equations of the fractional-order system reveal a recursive relationship within the sequence. We want to use RNN-based to capture the recursive dependency across time steps. In Eq. (13), the sequence $A_j$ is derived from the system parameters A and $\alpha$, and we employ a sequential model to estimate this computation. We also experimented with replacing the RNN with MLP to predict sys-id parameters. The performance degradation highlights the necessity of using RNN-based models. (All reported values are in units of 10⁻³.)

[1] Raissi, et al. "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations."

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are especially grateful for the recognition of our theoretical contributions and the clarity of our proofs.

Complex benchmark systems: Thank you for your thoughtful comment regarding the applicability of our method to real-world systems. While our current experiments focus on benchmark systems such as the cart-pole and quadrotor for clarity and reproducibility, our approach is broadly motivated by the need to model complex real-world phenomena that have memory effects, non-Markovian behavior, and other complex nonlinearities. Fractional-order dynamical systems (FOLTI systems) are well-suited for capturing such dynamics and have been shown to outperform integer-order models in various application domains. We provide the case study for turbulence control [1,2] derived from the incompressible Navier–Stokes equations as follows:

Prior work has also demonstrated the effectiveness of fractional-order models in: (i) EEG signal [3], where non-Markovian dependencies are critical for accurate representation, (ii) Power system dynamics [4], where Phasor Measurement Unit (PMU) data exhibit long-range temporal correlations better captured by ARFIMA models with non-integer differencing parameters, (iii) Chronic disease modeling, such as chronic obtrusive pulmonary disease (COPD) progression prediction [6], where fractional-order deep learning frameworks improve predictive accuracy, (iv) Biological control systems, including artificial pancreas control and pacemaker regulation of heart rate [5,7], where fractional-order models provide a more faithful representation of physiological dynamics. These examples reflect the broad relevance of fractional-order modeling in scenarios where the underlying physics is inherently complex (memory effects). Our method is the first end-to-end, deep learning-based approach designed to solve optimal control problems in fractional-order dynamical systems, and it can be directly applied to a wide range of real-world applications. We also evaluate our method on very high-dimensional synthetic data and demonstrate its scalability, due to space constraints, we kindly refer the reviewer to our detailed response to Reviewer cBiX for evaluation results.

Baselines: Thank you for highlighting the importance of baseline comparisons. We initially excluded them because standard baselines require system parameters, which our method does not. However, we agree that including them strengthens the evaluation. We have conducted extensive baseline comparisons on the cart-pole and quadrotor systems, detailed in our response to Reviewer cBiX. We hope these results clarify the advantages of our approach.

[1] Holmes, Philip. Turbulence, coherent structures, dynamical systems and symmetry. Cambridge university press, 2012.
[2] Li, Zhijin, et al. "Optimal control of cylinder wakes via suction and blowing." Computers & Fluids 32.2 (2003): 149-171.
[3] Besançon, et al. "Fractional-order modeling and identification for a phantom EEG system." IEEE Transactions on Control Systems Technology 28.1 (2019): 130-138.
[4] Saadatmand, et al. "PMU-based FOPID controller of large-scale wind-PV farms for LFO damping in smart grid." IEEE Access 9 (2021): 94953-94969.
[5] Gupta, Gaurav, et al. "Non-markovian reinforcement learning using fractional dynamics." 2021 60th IEEE Conference on Decision and Control (CDC). IEEE, 2021.
[6] Yin, Chenzhong, et al. "Fractional dynamics foster deep learning of COPD stage prediction." Advanced Science 10.12 (2023): 2203485.
[7] Bogdan, Paul, et al. "Pacemaker control of heart rate variability: A cyber physical system perspective." ACM Transactions on Embedded Computing Systems (TECS) 12.1s (2013): 1-22.

Thank you very much for the recognition of our novel theoretical contributions, the FOLOC framework, and the experimental design, as well as highlighting its effectiveness and robustness. We also thank the reviewer for pointing out areas where additional empirical comparisons and runtime benchmarks could further strengthen our work. We will address these points in detail below and incorporate the suggestions in the revised version.

High-dimensional settings: Thank you for your valuable comment regarding the scalability of our FOLOC framework in high-dimensional settings. In response, we have added a new experiment to specifically evaluate performance in such scenarios. The results, summarized in the table below, demonstrate that our method remains efficient and robust as the dimensionality increases. (All reported values are in units of $10^{-3}$.)

Table 1: Experiments on high dimensional datasets

Baselines: Thank you for your thoughtful comment on runtime comparisons. We conducted detailed benchmarks comparing FOLOC with several popular baselines, including MBRL, CMA-ES, MPC, PID, and iLQR. For each method, we report both performance and runtime. FOLOC consistently achieves the highest accuracy and lowest computation time on both cart-pole and quadrotor tasks. We found that FOLOC achieves superior performance on cart-pole dynamics, with MSE reduced by up to 99.99% and MAE by up to 99.32%. All experiment code is available at the link provided in Appendix A (Page 12).

Table 2: Baseline Model Evaluation on cart-pole dynamics

---

Table 3: Baseline Model Evaluation on quadrotor dynamics

Header: We appreciate the reviewer for pointing out the heading issue, we will ensure to correct it in the revised version.