DiLQR: Differentiable Iterative Linear Quadratic Regulator

We appreciate the reviewer for the serious and careful feedback . For this discussion, let’s use [BKK] as an abbreviation for “Bai et al. (2019)”.

[BKK] implemented our idea (analytically deriving derivatives for a fixed-point) before

In [BKK], fixed-point theory is used to handle the gradients in various kinds of DNN (transformers, RNNs, etc.), which are used for language modeling. By contrast, we apply fixed-point theory to iLQR, a nonlinear controller. This context is quite different from that of DNN. Different problems call for different technical contributions. The novelty of our method comes from: 1. observing that the structure of iLQR fits the fixed-point methodology, and 2. developing the details that make the idea work for iLQR (sections 4.2-4.6). We do not claim to be the inventors of implicit differentiation or fixed-point theory.

Other frameworks like JAX could provide invaluable insights.

We agree JAX has special properties in comparison with Pytorch. But our contribution identifies an approach that saves calculation steps that any autodiff approach would have to do. Our innovation is theoretical, and has nothing to do with what programming language and library we are using. Since our contribution is primarily conceptual, we think it’s reasonable to demonstrate proof-of-concept on just one widely-endorsed platform. Using Pytorch suffices to demonstrate the difference in work just described.

Visual experiment lacks discussion and analysis.

Our visual experiments aim to show visual control with our modular could predict a series of images (with high accuracy) instead of a single image. We will provide some numerical analysis shortly.

"the work is not compared to more established methods on advanced tasks, and it suggests potential for SOTA performance"

We are unclear about the definition of "SOTA" and "performance." The two cited papers also seem to lack explicit definitions. Our paper focuses on proof of concept, showing that our theoretical framework works effectively. In the future, we may combine more powerful vision models such as large vision-language foundation models combined with MPC for more complex tasks. We also have a discussion in the common comment.

Our method lacks exposition and strong theoretical guarantees compared with our references.

We are unsure about the specific aspects referred to as "exposition" and "strong theoretical guarantee". We have added a theoretical section in appendix A.4 to give more support of our method. The reviewer could provide further clarification or examples.

Limitations absent

Shortly speaking, our method assumes the iLQR can find a fixed-point, and also the requirement of first-order and second-order derivatives of the dynamics. These assumptions may restrict the applications of our methods. We will revise our submission to include them.

Inconsistent notations

We have made efforts to maintain consistent notations and ensure that each notation is properly explained. But, as mentioned in the common comments, we have updated the notations to make them more readable.

Questions from the reviewer

Q1.1 In section 4.4, is the L referenced there still referring to the loss function from line 208?

The L in section 4.4 can refer to the loss in line 208, however, that is not how we use it in section 4.4. Here, we treat H as L, and we rely on derivatives provided in (Amos 2018) (e.g. $\frac{\partial L}{\partial D}$) to calculate $\frac{\partial H}{\partial D}$.

Q1.2 Is the binary loss approach described in that section approximate?

No, it is not approximated, as we enumerate all the elements in H to create multiple binary losses to obtain $\frac{\partial H}{\partial D}$.

Q2. If the first paragraph in section 4.6 addressed and implemented in our code

Yes, it is. Please refer to line 703 in lqr_step_explicit.py in our code, where true_dynamics.grad_input performs the operation described in Section 4.6. The function is defined in the model file. In line 711 of lqr_step_explicit.py, we return dtheta, which indicates that we compute the analytical gradient of $\theta$ and pass the customized gradient to $\theta$, rather than allowing PyTorch to traverse the graph using create_graph=True, as noted by the reviewer in the mpc_explicit.py file. We enabled create_graph only for testing auto-differentiation.

Q3.Does $\frac{\partial D_t}{\partial \theta}$ refer to term in Eq. 13?

Yes, it refers to the $\frac{\partial D}{\partial \theta}$ in equation (13). We refer to equation (9) because we want to trace back to the root equation. We have revised this point to make the section more readable.

Typos

We thank the reviewer for pointing out the typos. We have corrected them accordingly.

We sincerely thank the reviewer for his/her valuable feedback and the time and effort dedicated to evaluating our work. We’ve decided to withdraw the paper at this stage to further improve it before resubmitting. This will save the Area Chair and reviewers’ time.

We thank the reviewer for pointing out two articles that we did not cite. For the current discussion, let’s refer to these works as

[DM+] Differentiable Optimal Control via Differential Dynamic Programming;

[BPC] Leveraging Proximal Optimization for Differentiating Optimal Control Solvers.

We agree that they are relevant, but we do not agree that the derivatives of [DM+] and ours are similar. We believe that our innovations differ from [DM+, BPC]. They are not in conflict with ours, but rather complementary.

Differences between our method and [DM+,BPC]

1. In our notation, the linearized model involves coefficients D and d. [DM+] essentially details a more accurate calculation of just one term (equation (19) in [DM+], namely, the one we denote $\frac{\partial H}{\partial D}$) among the several terms on the right-hand side of our Equation (13). We currently use the results from (Amos et al.2018) for this term. Similarly, [BPC] provides a more accurate calculation of the same term, $\frac{\partial X}{\partial D}$, under constraints. Both cited results omit the other terms we show on the right side of Equation (13) as well as the fixed-point finding process in Equation (9). Thus our fixed-point perspective leads to materially different calculations from these works, and our experiments confirm that the difference is indeed an improvement.

2. Why does the fixed-point method make a difference? We have added a comparison between the fixed-point and non-fixed-point methods in the appendix A.4. Please refer to section 4.7 and the additional appendix A.4 in the revised PDF for details.

3. In fact, it could be useful to combine the refinements of [DM+,BPC] with the structure suggested in our manuscript to generate a new method with all the advantages of both. However, [DM+,BPC] didn’t release their code, so we cannot complete this project during the current discussion period.

“Section A of [DM+] states that they use any method (including either iLQR or DDP) for solving optimal control problems."

As noted in the introduction of our paper, we focus on the exact gradient of iLQR. This includes aligning the forward solution with the backward pass gradient. Such alignment can be important, especially when the forward pass finds a sub-optimal path but the backward pass obtains the gradient of the path assuming the path is optimal.

[BPC] can also solve the OC problem with iLQR

[BPC] briefly suggests a way to extend their result to iLQR; however, it remains a non-fixed-point method that simply uses the chain rule to connect each derivative (same as Amos (2018)). This has been addressed in Points 1 and 2 in the reply above.

Simple example

As mentioned in our discussion section, 'Many prior works, such as Amos et al. (2018), Watter et al. (2015), Xu et al. (2024a), and Jin et al. (2020), also rely on such toy examples to demonstrate foundational concepts.' The [BPC] paper referenced by the reviewer similarly uses these exact examples. We have provided further discussion on this point in the common comments.

We sincerely thank the reviewer for his/her valuable feedback and the time and effort dedicated to evaluating our work. We’ve decided to withdraw the paper at this stage to further improve it before resubmitting. This will save the Area Chair and reviewers’ time.

We are grateful for the reviewers' feedback and the effort the reviewer has put into improving our work.

Relations between each component

Section 4.1 provides an overview of the pipeline for differentiable control. Section 4.2 focuses on our main method, specifically for $\frac{\partial \tau}{\partial \theta}$ in section 4.1. Section 4.3 addresses the computation of $$F_X$,$F_U$,$G_X$,$G_U$$ as referenced in Section 4.2. Section 4.4 covers $$\frac{\partial H}{\partial D}$$ (where $H = F$ or $H = G$) as used in Section 4.3. Section 4.5 deals with $$\frac{\partial D}{\partial X}$$ from Section 4.3, while Section 4.6 explains $$\frac{\partial D}{\partial \theta}$$ from Section 4.3. We can incorporate this explanation into either a pseudo-code block or an additional subsection for improved clarity.

Contribution of each component

Section 4.4 demonstrates at least a 2x acceleration when the data size is substantial(>1500 data pairs). Section 4.5 highlights significant memory cost benefits. Section 4.6 is a critical component specifically developed for our overall method. Without this module, the fixed-point implementation would not achieve the expected time advantage. For example, with 1,500 input data pairs, the computation time for gradients (line 704 in lqr_step_explicit.py) is reduced from 5 seconds to 0.02 seconds. Additional numerical illustrations will be provided shortly. The contributions of these components are independent and can be evaluated separately without the need for an ablation study.

Notation issue

Please see the common comments for more information. we have updated the $D_\theta$ operator to $\nabla_\theta​$.

We sincerely thank the reviewer for his/her valuable feedback and the time and effort dedicated to evaluating our work. We’ve decided to withdraw the paper at this stage to further improve it before resubmitting. This will save the Area Chair and reviewers’ time.