DEQ-MPC : Deep Equilibrium Model Predictive Control

We thank the reviewer for their detailed feedback. We’re glad the reviewer appreciated the novelties in the method. We discuss some of the criticisms and questions pointed out by the reviewer as follows :

1. Theoretical Contribution :
   • Note on Convergence : We have added a note around convergence analysis in the appendix when discussing the solver details. Our treatment of the system as a DEQ allows us to borrow results from [1][2] to ensure convergence of the fixed point iteration. Specifically, if we assume the joint Jacobian of the fixed point iterations we describe are strongly monotone with smoothness parameter $m$ and Lipschitz constant $L$, then by standard arguments (see e.g., Section 5.1 of [3]), the fixed point iteration with step size $\alpha<m/L^2$ will converge. However, going from the strong monotonicity assumption on the joint fixed point iterations to specific assumptions on the network or the optimization problem is less straightforward. But, in practice a wide suite of techniques have been used to ensure that such fixed points exist and can be found using relatively few fixed point iterations. In fact, for all of our experiments, we converge within 6 ADMM iterations once trained.
   • Representational Benefits : Perhaps a more central claim in the paper is around the representational benefits of our approach over vanilla differentiable MPC. However, given the non-linear nature of the networks and optimization problems we consider, it’s unclear to us how we could theoretically demonstrate the representational benefits of the approach beyond the intuitive reasoning we provide in the paper. We believe this would be a significant theoretical undertaking and hope that future work will address these representational benefits more rigorously. Broadly, we view the main contribution of our paper as providing the algorithmic tools and empirical results and ablations to leverage these representational benefits when combining optimization layers within deep learning pipelines in a principled way.
2. Computational expense : For most of the experiments, we compare the methods in the streaming setup with warm-starting, i.e 1 ADMM iteration for DEQ-MPC methods vs a network inference + 2 AL iterations for the Diff-MPC methods. In this setting, the two methods have almost exactly the same runtime (most of it consumed in the optimization iterations) : 0.121s vs 0.120s. On the other hand, when used without warm-starting, the run-time differences between the two methods are non-insignificant : runtimes of 1.12s and 1.18s for DEQ-MPC-NN and DIFF-MPC-NN respectively without any warm-starting. This is despite the fact that we run more network inference steps (6 vs 1) and more total AL iterations (12 vs 10) for DEQ-MPC. We observe that DEQ-MPC requires fewer line-searches compared to Diff-MPC and each inner AL solve converges faster given the network adapts the references across iterations. This was interesting given this was never explicitly incentivized in the network learning objectives and was learnt automatically by DEQ-MPC. However, we do not report these runtime results in the paper given the optimization solver we implemented is in pytorch and hence not optimized for real time usage. Thus, quantitative runtime comparisons are somewhat misleading.
3. Writing Clarity : Pardon us for the confusion. We try to ground these variables more concretely in section 4.1 in the context of trajectory tracking. The optimization parameter in that context for example would represent the waypoints we expect the robot to follow and the final optimized trajectories are represented as $\tau = [x,u]$. Do let us know if the description there was unclear.
4. Task specific priors : Pardon us again for the confusion here. But in the context of Differentiable-MPC and DEQ-MPC, we refer to the incorporation of the optimization problem as part of the prediction problem itself as the priors as opposed to predicting the trajectories directly as outputs of a network. Thus, we are able to explicitly incorporate objectives and constraints (eg. dynamics, control limits, obstacle avoidance constraints) to bake in specific properties of the system we might know as part of the end-to-end prediction problem. We show this across examples.

[1] Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems, 2020.

[2] Shaojie Bai, Vladlen Koltun, and Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In Proceedings of the International Conference on Machine Learning, 2021.

[3] Ernest K Ryu and Stephen Boyd. Primer on monotone operator methods. Appl. Comput. Math, 2016.

We thank the reviewer for their thoughtful feedback. We’re glad the reviewer appreciated the novelties in the method, the diverse ablations, empirical results and clear writing. We acknowledge the missing error bars in the original submission. We have now revised the paper with the provided feedback. We have included mean and variance of the normalized returns on all the experiments in the table and have also provided the error bars for the ablation experiments. We thank the reviewer for their feedback!

We thank the reviewer for their detailed feedback. We’re glad the reviewer appreciated the importance of the problem, the novelties in the method, the diverse ablations, empirical results and clear writing. We address the specific questions and weaknesses pointed out by the reviewer.

1. End-to-end neural net baselines : The end-to-end neural net baselines in the ablations represent simply measuring the validation error on the waypoints predicted by the network. We do not include the end-to-end neural net baselines in the table given that tracking those waypoints post-hoc using the mpc solver in a warm-started setting typically just barfs and gets unstable in a number of cases.
2. Gradient smoothness/alignment : We believe the discussion of global/desired update direction was a bit of an overreach and have edited it in the draft to focus on smoothness instead.
3. Figure 1 corrections : We acknowledge that adding an iterative component to the network architecture in Figure 1 might make it more informative. However, we believe that it somewhat distracts from the main point of the paper which is that DEQ-MPC layers benefit representationally from conditioning the network on the optimization iterates. The additional occasional stability/representational benefits from making the network itself iterative (i.e DEQ) was almost an afterthought from the perspective of the paper. Thus we chose to not make it explicit in the figure either.
4. Predicting dynamics/constraint parameters : We acknowledge that in our current experiments we formulate the NN output / MPC parameters as waypoints and the MPC objective as waypoint tracking. We agree that our work can be strengthened by wiring those modules differently such as learning the dynamics or constraints (obstacle positions). Indeed we hope to explore those and other tasks in our future work. However, we focus specifically on the trajectory tracking task in this paper given its wide ranging applications. That helps us ground the method in an important task while keeping the formulation consistent and simple across experiments. This helps us focus specifically on the modelling and methodological choices of our work.
5. Effect of number of warm-starting steps : We show some ablation experiments for long-horizon evaluations in the appendix Figure 8. Regarding the warm-starting steps, for our experiments, we tested with 2, 4 and 6 warm-starting AL iterations and 1, 2, 3 ADMM iterations respectively in Figure 7. We found that even with just 1 ADMM iteration, DEQ-MPC performs pretty well. In fact, as we increase to 6 warm-started AL iterations, even the Diff-MPC variants start performing competitively.
6. Convergence criterion for DEQ : Pardon us for the missing detail. We have added it to the architectural details in the appendix. For the DEQ network, we solve it to convergence using a fixed point solver (to a tolerance of 1e-2). For the outer DEQ, we use 6 ADMM iterations across experiments when not warm-starting.
7. Intermediate gradients for Diff-MPC : Using intermediate gradients for Diff-MPC biases the network gradients with the earlier iteration gradients. Given that the network in this case can’t distinguish between ite rations (due to the lack of iterate conditioning), this results in worse learning. Whereas, given that the network in DEQ-MPC is conditioned on the optimization iterate, using intermediate iteration gradients for DEQ-MPC does not lead to any biases. In the trajectory tracking problem we study, we learn the waypoints for the optimization problem, which appear as cost parameters. Other cost and dynamics parameters are indeed set manually currently. While they could conceivably be learnt jointly using gradient descent, we believe it might be unstable to train especially when the parameters are wildly off. Thus, we believe that, in such a scenario, it might be a good idea to first fit the cost and dynamics parameters independently using a proxy task. Example, for the dynamics, one could just fit the dynamics parameters by minimizing the dynamics residuals on the data provided or directly on the trajectory tracking problem using IFT gradients. Likewise, the cost terms could be learnt by using perturbed tracking trajectories and then learning the cost terms on the corresponding end-to-end tracking problem using IFT gradients.

We thank the reviewer for their detailed feedback. We’re glad the reviewer appreciates the importance of the underlying problem we seek to solve and the nuances of the methodological contributions. We address the specific questions and weaknesses pointed out by the reviewer.

1. Complicated approach : At a fundamental level, the methodology is motivated by the fact that for representational effectiveness of learning+optimization layers, we desire a coupled inference problem where the network inference is a function of the optimization iterates (as observed in our representational ablations). We believe that’s a fundamental representational unlock needed to exploit the implicit/iterative nature of optimization problems which hasn’t been explored in previous work. The specific algorithmic contributions around the scheduled alternating optimization etc, are simply a means to achieve this. They allow us to solve the alternating problem in an efficient manner. But we believe those are important details towards making these approaches fast enough for real-time applications!

2. Weak evaluation : As pointed out by the reviewer we indeed use pendulum/cartpole for analysis and debugging purposes (e.g gradient analysis). But we also include extensive experiments on more complicated problems such as

   (a) Quadrotor goal reaching task : We use the full nonlinear dynamics of the quadrotor with motor thrust actions (considering the agile low-level dynamics), unlike many other work which output thrust and velocities, abstracting what we consider in our work.

   (b) Quadrotor-pole : where a pole is attached to the quadrotor and the task is to swing up the pole (using its inertia) while reaching the goal. This task is dynamic and challenging, highlighting the importance of incorporating the MPC layer.

   (c) Quadrotor-pole with obstacles : which involves solving the above problem while avoiding obstacles in the path. This task is not just a simple 3D navigation problem as it features high complexity and nonconvexity of the dynamics and obstacle avoidance. This task is also considered safety-critical, as the policy must reason about both hard input and state constraints. Moreover, we run all the ablation experiments on (b) and (c) which we consider fairly complicated environments as observed from the results. Through these ablations, we tease out the specific representational benefits obtained from this approach. While we acknowledge that more complicated tasks could also have been tried, we believe these experiments demonstrate the representational benefits that could be gained from our method in fairly complicated situations as well.

3. Architectural and solver details : We provide some of the solver and architecture details in the appendix in section A.3 and A.4 and provide the code for any additional nuances we may have missed in our description. We would be happy to provide any additional details if the reviewer has specific questions in that regard!

4. Computational expense : For most of the experiments, we compare the methods in the streaming setup with warm-starting, i.e 1 ADMM iteration for DEQ-MPC methods vs a network inference + 2 AL iterations for the Diff-MPC methods. In this setting, the two methods have almost exactly the same runtime (most of it consumed in the optimization iterations) : 0.121s vs 0.120s. On the other hand, when used without warm-starting, the run-time differences between the two methods are non-insignificant : runtimes of 1.12s and 1.18s for DEQ-MPC-NN and DIFF-MPC-NN respectively without any warm-starting. This is despite the fact that we run more network inference steps (6 vs 1) and more total AL iterations (12 vs 10) for DEQ-MPC. We observe that DEQ-MPC requires fewer line-searches compared to Diff-MPC and each inner AL solve converges faster given the network adapts the references across iterations. This was interesting given this was never explicitly incentivized in the network learning objectives and was learnt automatically by DEQ-MPC. However, we do not report these runtime results in the paper given the optimization solver we implemented is in pytorch and hence not optimized for real time usage. Thus, quantitative runtime comparisons are somewhat misleading.

We thank the reviewer for their detailed feedback. We’re glad the reviewer liked the diverse ablations, empirical results and clear writing. We address the specific questions and weaknesses pointed out by the reviewer.

1. Incremental methodology+complicated approach : At a fundamental level, the methodology is motivated by the fact that for representational effectiveness of learning+optimization layers, we desire a coupled inference problem where the network inference is a function of the optimization iterates (as observed in our representational ablations). We believe that’s a fundamental representational unlock needed to exploit the implicit/iterative nature of optimization problems. As shown in our experiments, this improves the generalization and scaling abilities of the model. But more importantly it allows us to utilize warm-starting more effectively which was not possible with traditional differentiable mpc methods. The specific algorithmic contributions around the scheduled alternating optimization etc, are simply a means to achieve this. They allow us to solve the alternating problem in an efficient manner.
2. Reporting error bars : We updated all our experiments/ablations as suggested by the reviewer and ran multiple runs with different dataset partitions and reported the mean and variance for all the results and ablations.
3. Theoretical insights : Our central claim in the paper is around the representational benefits of our approach over vanilla differentiable MPC. However, given the non-linear nature of the networks and optimization problems we consider, it’s unclear to us how we could theoretically demonstrate the representational benefits of the approach beyond the intuitive reasoning we provide above. We believe this would be a significant theoretical undertaking and hope that future work will address these representational benefits more rigorously.
4. Convergence criteria : We run the methods in a non-streaming setup. For the DEQ-MPC variants we run the solve the joint problem for 6 ADMM iterations whereas for the Diff-MPC variants, we run the Augmented Lagrangian solver for 10 AL iterations. We observe that the solver in both cases converges to <1e-3 tolerance on the steps.
5. Computational costs : For most of the experiments, we compare the methods in the streaming setup with warm-starting, i.e 1 ADMM iteration for DEQ-MPC methods vs a network inference + 2 AL iterations for the Diff-MPC methods. In this setting, the two methods have almost exactly the same runtime (most of it consumed in the optimization iterations) : 0.121s vs 0.120s. On the other hand, when used without warm-starting, the run-time differences between the two methods are non-insignificant : runtimes of 1.12s and 1.18s for DEQ-MPC-NN and DIFF-MPC-NN respectively without any warm-starting. This is despite the fact that we run more network inference steps (6 vs 1) and more total AL iterations (12 vs 10) for DEQ-MPC, we observe that DEQ-MPC requires fewer line-searches compared to Diff-MPC and each inner AL solve converges faster given the network adapts the references across iterations. This was interesting given this was never explicitly incentivized in the network learning objectives and was learnt automatically by DEQ-MPC. However, we do not report these runtime results in the paper given the optimization solver we implemented is in pytorch and hence not optimized for real time usage. Thus, quantitative runtime comparisons are somewhat misleading.
6. Utility for RL/high-dimensional tasks : As discussed above, the main benefits of DEQ-MPC layers over DIFF-MPC layers is representational. Thus, we believe these methods would directly transfer to most other problems where differentiable-mpc layers have already been shown to be effective. Eg : [1] for RL or [2] in high dimensional partially observed environments.
7. Figure 1 description : Pardon us for the lack of clarity here. We have modified the figure description for additional clarity. The drone inputs were meant to show observations from the robot and the bounded curves are the waypoints with bounds representing constraints which are violated by \theta but satisfied by \tau.

[1] Romero, Angel, Yunlong Song, and Davide Scaramuzza. "Actor-critic model predictive control.", IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024

[2] Acerbo, Flavia Sofia, et al. "Learning from Visual Demonstrations through Differentiable Nonlinear MPC for Personalized Autonomous Driving." arXiv preprint arXiv:2403.15102 (2024).