DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation

We sincerely appreciate the reviewers' insightful comments and constructive feedback, which have helped improve our paper. Below we provide detailed responses to each point.

Q1: No Theorem Proving Speedup

We thank the reviewer for this important question. Our work is situated in the field of differentiable control, which aims to expose model structure to learning and improve sample and computational efficiency. In this area, it is commonly understood—though not typically accompanied by formal theorems—that incorporating control priors and structured solvers leads to faster learning. Foundational works such as:

• Revisiting Implicit Differentiation for Learning Problems in Optimal Control (NeurIPS 2023)
• Pontryagin Differentiable Programming (NeurIPS 2020)
• Infinite-Horizon Differentiable MPC (ICLR 2020)
• Differentiable MPC for End-to-End Planning and Control (NeurIPS 2018)

share a similar focus on demonstrating empirical improvements, without formal speedup theorems.

Such approaches have seen success in robotics, e.g.,

• Reaching the Limit in Autonomous Racing: Optimal Control vs Reinforcement Learning (Science Robotics 2023)
• Actor-Critic Model Predictive Control (ICRA 2024)
• Guiding RL with Incomplete Model Information (IROS 2024)
  demonstrating practical benefits in learning efficiency.

In addition, our method improves computational efficiency: by leveraging analytic gradients, we avoid differentiating through rollouts, reducing complexity to $\mathcal{O}(1)$ per iteration—a property determined by algorithm design.

Q2: Experiments on More Complicated Tasks

As noted in our discussion section, many prior works (Amos et al., 2018; Watter et al., 2015; Xu et al., 2024a; Jin et al., 2020) also use simplified environments to illustrate core ideas. Differentiable control methods are built upon model-based control and therefore require access to structural priors, which makes them less "plug-and-play" than general-purpose RL. However, the benefit is that such structure greatly reduces the sample complexity.

To go beyond toy tasks, we further evaluate our method on a challenging rocket control task with a 13-dimensional state space—the most complex benchmark among recent analytical differentiable control works. Videos are available on our project page:
https://sites.google.com/view/dilqr/ . Additional results are currently being prepared and will be shared shortly. The rocket model and controller code are also open-sourced in our codebase.

Q3: Sudden Drop and High Standard Deviation for Model Loss

The sudden drop corresponds to learning progress in reducing model loss.
The observed standard deviation (~0.17) is reasonable, especially considering the mean gap between methods is 0.47, indicating that the improvement is both significant and consistent.

Q4: Typographical Errors

We sincerely thank the reviewer for catching these formatting issues. We will carefully review the entire manuscript and correct all typographical/formatting errors in the final version.

We sincerely thank the reviewer for the insightful comments and constructive feedback. Below we provide point-by-point responses to the raised questions.

Q1: Comparison with DiffTORI

We thank the reviewer for raising this point. DiffTORI ([Wan et al., NeurIPS 2024]) is a pioneering work that bridges differentiable control and RL, especially in high-dimensional settings. We fully agree its generality (e.g., model-agnostic design, multi-modal rewards) offers broader applicability for perception-driven tasks.

Our work focuses on structured control domains where iLQR is the canonical solver. By deriving exact gradients for iLQR, we achieve:

1. Data efficiency: Sub-2000 environment steps suffice to achieve the results in our paper, leveraging prior knowledge of dynamics.
2. O(1) backward complexity (vs. autodiff's linear cost, Fig. 2), enabled by analytic gradient computation.

This aligns with the PDP/DiffMPC lineage ([Jin et al., NeurIPS 2020; Amos et al., NeurIPS 2018]), emphasizing precision for structured control. Crucially, our method and DiffTORI address non-overlapping challenges:

• DiffTORI: General-purpose RL-control synergy with perception.
• Ours: Analytic acceleration for iLQR-based pipelines with structured dynamics.

We will add more discussion and clarify this complementary in the final version.

Q2: Differences with Theseus

Our method provides exact gradients for iLQR, which Theseus cannot compute automatically. While Theseus (similarly JAXopt) can differentiate through fixed-point equations ($x^*=f(x^*)$), iLQR's unique challenge is that its dynamics (D) and cost (C) depend on x* itself, creating a recursive relationship $x^*=f_{x^*}(x^*)$.

This cannot be resolved through auto-implicit-differentiation alone - it requires reformulating LQR as a pure fixed-point problem, which our paper implements theoretically (not just at the coding level). This is not a small step.

Q3: Relation to DiffMPC

We clarify that DiffMPC approximates gradients by treating D/C as independent of x* (i.e. x*≈f(x*)), while our method properly accounts for their interdependence through implicit differentiation.

Our key innovation lies in embedding Amos's results into a correct fixed-point formulation that yields exact analytical gradients, representing a high-level advancement beyond prior work.

Q4: Comparison with (Safe)PDP and IDOC

As noted in our related work (L147), these prior methods compute gradients under the assumption of optimal forward solutions, which creates mathematical inconsistencies when the forward pass is suboptimal. In contrast, our approach maintains alignment between forward and backward passes throughout iLQR optimization.

This key difference is evidenced in our cartpole experiments:

• IDOC: 4×10⁻² error
• PDP: 2×10⁻² error
• Ours: 1×10⁻⁴ error

Additional results will be available on our project page.

We acknowledge that (Safe)PDP and IDOC offer greater generality in handling diverse solvers and constraints - an important contribution to the field.

Q4: Losses for SysID

The SysID approach follows a two-stage process:

1. Estimates model parameters from data
2. Uses the estimated model in MPC for trajectory prediction

Compared to end-to-end learning, this represents a modular strategy that decouples system identification from control.

Q5: NN Baseline Comparison

This is exactly the key point we want to demonstrate! Our method essentially transforms the neural network's black-box approach into a white-box solution, achieving significant improvements in sample efficiency through structured prior knowledge.

Q6: Visual Control Task

We employ a basic autoencoder (AE) framework, which may exhibit sensitivity limitations. This work is primarily conceptual, with the visual control section demonstrating that:

• iLQR can function as a differentiable controller
• It successfully integrates with vision modules

We believe that combining our method with DiffTORI could lead to significant improvements in handling multi-modal inputs.

We sincerely thank the reviewer for the valuable feedback. Below we provide responses to each question.

Q1: Application to Higher Dimensional RL Tasks

The field of differentiable control remains relatively young. Current real-world applications primarily focus on autonomous vehicles and drones, as evidenced by recent works:

[1] Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning (Science Robotics 2023)
[2] Actor-Critic Model Predictive Control (ICRA 2024)
[3] Guiding RL with Incomplete Model Information (IROS 2024)

In simulation environments, our 6-DoF rocket control task (along with quadrotor control) represents the most complex benchmark among recent works in this field.

Key Challenges:
The primary difficulty lies not in computation, but in handling locomotion tasks where contact forces violate Lipschitz continuity and complicate gradient computation. Potential solutions include:

• Soft contact models
• Neural contact approximations

Q2: Extension Beyond Toy Examples

Our rocket control task with 13-dimensional state space currently represents the most challenging benchmark in analytical differentiable control, based on peer-reviewed literature:

[1] Revisiting Implicit Differentiation for Learning Problems in Optimal Control (NeurIPS 2023)
[2] Pontryagin Differentiable Programming (NeurIPS 2020)

Initial results are available on our project website, with additional experiments underway. Rocket model and control file can also be found in the codebase.

Q3: DiffMPC Comparison in Figure 3

While DiffMPC remains an excellent benchmark in the field, our method demonstrates approximately 5-7% improvement over it. This difference appears modest compared to the 1e4 magnitude improvement over SysID and NN baselines, making visual comparison challenging in the same plot.

However, our approach provides superior parameter estimation accuracy - a crucial factor for industrial applications where physical interpretability matters, even when iLQR's robustness can compensate for parameter inaccuracies.

Q4: Determining Rolling Direction

As detailed in Page 8, Line 414:
"We stack four compressed images as input channels to capture velocity information and determine the pendulum's swing direction."

This temporal stacking approach enables the system to infer motion dynamics from visual input.

Q5: Typographical Errors

We sincerely appreciate the reviewer's careful reading and for bringing these typographical issues to our attention. We have carefully reviewed the manuscript and corrected all identified formatting and typographical errors in the final version. These corrections include:

1. Fixed notation inconsistencies in equations (e.g., θX⋆ → ∂X⋆/∂θ)
2. Corrected the Appendix A title
3. Addressed all minor formatting issues throughout the text

We sincerely appreciate the positive evaluation and constructive suggestions. Below we address the specific questions raised:

1. Computational Complexity Estimates (Q1):
We clarify the computational costs with corrected exponents and iteration factors:

• Forward pass (iLQR):
  O(IT(n² + m²)) for T timesteps and I iterations (standard LQR complexity)

• Backward pass (diLQR.dx/cost):
  O(T(n² + m²)) per parameter dimension (Theorem 3.1)
  No dependence on I due to analytic formulation

• Autodiff baseline:
  O(IT(n² + m²)) where I = unrolled iterations

This explains Figure 2's constant backward cost vs. autodiff's linear scaling with I. We will add a FLOP count table in revision.

2. Scaling Barriers (Q2):
The primary challenge lies in obtaining second-order matrices in Section 4.6:

• Our codebase dedicates ~500 lines specifically to obtain these matrices element-by-element
• Current engineering overhead includes: • Hessian calculations for dynamics/cost
  • KKT matrix constructions
• While substantial, we believe future code optimizations can reduce this overhead ( such as pre-calculating and storing analytical solutions for key matrix operations, and developing automated routines to substitute corresponding values)

We have also implemented our method on a challenging rocket control task with 13-dimensional state space, which currently represents the most advanced benchmark in analytical differentiable control literature ( from PDP and IDOC paper). Demo and code are available on our project page:
Project Website

3. Typos/Clarifications:
We will fix:

• Appendix A title
• Line 228/229 notation (θX⋆ → ∂X⋆/∂θ)
• Line 264/265 clarification of Eq (14) dependencies

All corrections will be highlighted in the camera-ready version.

Acknowledgements
We thank the reviewer for catching these important technical nuances, which have improved our paper's precision.