Learning System Dynamics from Sensory Input under Optimal Control Principles

Major:

• The limitation acknowledged in section 5 ("training our model requires having optimal trajectories") is very strong. It undermines quite a bit of the setup and background about learning from data being general, having a ton of applications, etc. Moreover, I don't believe this is even mentioned until the experiments section--it should be in the abstract.
• The experiments section is unconvincing and pretty weak. For how many thousands of learning algorithms have been thrown at the cartpole and inverted pendulum over the last 30+ years, comparing to a single algorithm is not an adequate baseline. There are many model-based and model-free approaches that have worked well on those two toy-ish problems; I'm not convinced that the authors' algorithm is required or SOTA here.
• A few experimental/implementation details have been swept under the rug -- the most prominent (unless I missed it) being the role of $T$ and $M$ -- how big do they need to be in theory, in practice, how do you set them.
• No code was provided (as far as I can tell in the reviewer console).

Minor:

• Table 1 is not a good way to present this data (which isn't compared to the other alg.), the tolerances seem pretty loose as well.
• I have no idea what Table 2 and the related section are talking about. It seems like it is somehow implementing an MPC-style algorithm, but it's really unclear.
• The related work doesn't acknowledge almost anything written from about 1965 to 2005, and only slightly more pre-2015. The words "system identification" do not appear in the paper at all, which is concerning.

1. The impact of this approach may be significant, but the scope of the demonstrations in the paper is so narrow that I am unconvinced. Since they only look at 2 very simple systems (pendulum and cartpole), and rely on "demonstrations" from trajectory optimization, there is no evidence that this approach will work well for other systems. Even having 2-3 more toy problems would have greatly improved the argument for generalizability. I feel that the most exciting part of this paper is the ability to deal with complex robotic systems at the image level, but it was only shown to work on simple toy problems with many readily available solution methods.
2. The results are compared to Bounou et al. 2021, which makes sense because the approaches are so similar. However, since you are only considering deterministic systems with computer generated images trained on a dataset of optimized trajectories, a comparison to a simple supervised learning regression based controller would have been very useful. This comparison would (hopefully) have shown that a simple supervised learning approach was ineffective.

I've had the opportunity to delve into Koopman operator theory (KOT) in my past research so I am very well aware of the nuances in this paper. The weakness of the paper are as follows:

1. It must be noted that that control-affine systems transform to a bilinear control Koopman based system (of Koopman based linear system) in the lifted space under particular conditions, as exemplified by Theorem II.1 in the paper [1]. It's also worth mentioning that these bilinear/linear forms may not be universally applicable to all general nonlinear systems, especially those that do not adopt the control affine form. Some recent works on KOT have also showcased that a general control-affine nonlinear system can be transitioned to a more generalized input-separable Koopman system, with bilinear and linear forms being special instances of these separable Koopman forms.

2. Note that KOT based approximation is only valid for control affine systems and not general nonlinear systems of form (1) given in the paper.

3. The problem of control of dynamical systems using Koopman operator theory using MPC [4], LQR[5], Lyapunov based[6,7] methods has already been explored before. Furthermore, [5] considers more complicated nonlinear systems such as lunar lander in their simulations. There are more subsequent works on Koopman based control applied to real world applications such as soft robots [8].

4. On the topic of stability in Koopman-based learned matrices you use the proximal method of multipliers as in eqn (6) of your paper, however there are several contributions, such as [3], and its subsequent related studies that have already addressed this issues. However, you have not cited them in the literature. I believe you are just using another approach to promote stability.

5. Considering the above points, a more comprehensive literature review on the Koopman operator for controlled/non-controlled dynamical systems might enhance the paper's breadth. Furthermore, you should consider examples such as quadrotors, vehicle dynamics based on bicycle model etc.which are more nonlinear and applicable to real world.

6. The training objective is very standard in KOT literature and does not solve any new challenges. Furthermore, I feel that other linearization based methods such as the iterative LQR (iLQR) would work better or equivalent to the proposed Koopman based LQR approach.

7. Extensive numerical comparisons with well known and established control techiques are missing in their paper such as Nonlinear MPC, iLQR, PID and recent methods like MPPI and so on.

[1] Bruder, Daniel, Xun Fu, and Ram Vasudevan. "Advantages of bilinear Koopman realizations for the modeling and control of systems with unknown dynamics." IEEE Robotics and Automation Letters 6, no. 3 (2021): 4369-4376.

[2] Lusch, Bethany, J. Nathan Kutz, and Steven L. Brunton. "Deep learning for universal linear embeddings of nonlinear dynamics." Nature communications 9, no. 1 (2018): 4950.

[3] Fan, Fletcher, Bowen Yi, David Rye, Guodong Shi, and Ian R. Manchester. "Learning stable Koopman embeddings." In 2022 American Control Conference (ACC), pp. 2742-2747. IEEE, 2022.

[4] Korda, Milan, and Igor Mezić. "Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control." Automatica 93 (2018): 149-160.

[5] Han, Yiqiang, Wenjian Hao, and Umesh Vaidya. "Deep learning of Koopman representation for control." In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 1890-1895. IEEE, 2020.

[6] Zinage, Vrushabh, and Efstathios Bakolas. "Neural koopman lyapunov control." Neurocomputing 527 (2023): 174-183.

[7] Huang, Bowen, Xu Ma, and Umesh Vaidya. "Feedback stabilization using Koopman operator." In 2018 IEEE Conference on Decision and Control (CDC), pp. 6434-6439. IEEE, 2018.

[8] Bruder, Daniel, Xun Fu, R. Brent Gillespie, C. David Remy, and Ram Vasudevan. "Data-driven control of soft robots using Koopman operator theory." IEEE Transactions on Robotics 37, no. 3 (2020): 948-961.

The computational difficulty is the solution of the LQR, which may become too expensive for high-dimensional systems. Also, the proposed method is not explicitly compared with other existing methods.