Disentangling Linear Quadratic Control with Untrusted ML Predictions

Question 2. Implementation complexity:

We thank Reviewer 6dfr for this question and we argue that the additional computational overhead of DISC is minimal.

Like MPC, DISC requires the prediction of future disturbances at each time step for decision-making. Given this prediction, DISC's overhead primarily stems from two sources: the disentanglement algorithm and the online learning procedure for the confidence parameter $\lambda$. The latter mainly involves computing the gradient of $\zeta^{\top} H \zeta$ w.r.t. $\lambda$.

ICA theory requires that the latent dimension be smaller than the observation dimension for identification. The computational cost of disentanglement varies by algorithm but is generally efficient. For instance, in the setting of our experiments, the FastICA algorithm completes in under a second.

For the online learning procedure, when under linear mixing, the gradient computation involves several matrix multiplications with the dimension of the matrix dimensions upper bound by the observation dimension. For nonlinear mixing, one can employ a neural network to represent the mixing function and use inference and the Jacobian computation of this neural network to calculate the gradient. Both can be efficiently completed within seconds for observation dimensions typical in control tasks. In our experiments with linear mixing, the online learning of the $\lambda$ procedure takes less than a second.

Overall, DISC's computational requirements remain very manageable for most practical applications.

Besides the responses in the formal rebuttal (will be revealed after the rebuttal phase), due to space limitations, in this official comment we include our additional responses to the limitations mentioned at the end of the review:

Limitation 1: While our study focuses on applications like drone navigation and voltage control, the underlying principles and methods of DISC are designed to be broadly applicable. We acknowledge that real-world deployment may vary across different domains, and future work will aim to test DISC in a wider range of applications to better understand its generalizability. We are also exploring how the methodology can be adapted to other fields. In future work, we plan to evaluate DISC on a broader range of datasets and more diverse scenarios.

Limitation 2: We'd like to highlight that the dependency on prediction error in the derived competitive ratio bound offers the first bound of this nature, grounded in a term as a special function of the prediction error, without restrictive assumptions on the errors $(\overline{\varepsilon}(1),\ldots,\overline{\varepsilon}(k))$.

For reference, the competitive ratio bound is outlined informally as follows: $\mathbb{E}\left[\mathsf{CR}(\text{DISC})\right]\leq 1+o(1)+ O\left(\rho^{2w}\right) + \underbrace{O\left(\sum_{i=1}^{k}\frac{\overline{\varepsilon}(i)}{\Omega(T/w)+\overline{\varepsilon}(i)}\right)}_{\text{*Best-of-both-worlds utilization*}},$ where the $o(1)$ term hides quantities that vanish when the total number of steps $T$ increases; $k$ is the number of latent variables generating the perturbations; $\rho\in (0,1)$; $w$ denotes the prediction window size and each $\overline{\varepsilon}(i)$ (for $i=1,\ldots,k$) denotes the prediction error corresponding to each latent component.

The last term in this result depends on the prediction error, but it highlights the desired performance guarantee:

Consistency: When a component-wise prediction error $\overline{\varepsilon}(i)$ is small, the individual term ${\overline{\varepsilon}(i)}/\left({\Omega(T/w)+\overline{\varepsilon}(i)}\right)$ will be negligible. Precisely, if $\overline{\varepsilon}(i)=0$, the resulted $i$-term becomes zero as well. This actually yields a smaller competitive ratio compared to traditional robust control policies that do not utilize disentangled predictions.

Robustness: Otherwise, it always holds that ${\overline{\varepsilon}(i)}/\left({\Omega(T/w)+\overline{\varepsilon}(i)}\right)\leq 1$, regardless of how high the prediction error becomes. The robustness can be inferred from this dependency on the prediction error such that even if $\overline{\varepsilon}(i)$ becomes large, the last term is still bounded since $\overline{\varepsilon}(i)$ appears in both the numerator and the denominator. It's worth mentioning that such a bounded competitive ratio is not achievable by traditional control policies such as MPC.

Thank you for all the comments and we appreciate your positive feedback on our work!

Regarding the issues pointed out in the weakness part, here are our explanations and future plans:

Assumptions: We acknowledge the limitations of assuming the continuity and bijectivity of the mixing function $f$. It is worth highlighting that those assumptions are standard in nonlinear independent component analysis (ICA) models [Hyvarinen 2016, Khemakhem 2020, Yang 2022, Zheng 2022]. These advances leverage additional assumptions on the latent variables in $s_t$ and the mixing function $f$ to make the model identifiable. Below we summarize those common assumptions (besides the standard assumptions of mutual independence between latent variables and at most one latent variable is Gaussian) made in those models as examples. A similar table can be found in our Appendix B.1.

Standard Assumptions on $f$ and $s$ for a Subset of Nonlinear ICA Models

Datasets: Thank you for the comment. In future work, we plan to evaluate DISC on a broader range of datasets and more diverse scenarios.

Implementation Complexity: We discuss the practicality of DISC in the following 1-1 responses to the questions raised.

Question 1. The core step in DISC is the computation of the action through the following $\lambda$-CON policy:

$u_t =-Kx_t- Y\sum_{\tau=t}^{\overline{t}}\left(F^\top\right)^{\tau-t} P f\left(\lambda\circ\widetilde{s}_{\tau|t};\widetilde{\theta}_t\right),$ where $Y:= (R+B^{\top}PB)^{-1}B^{\top}$; $\\widetilde{s}\_{\\tau|t}$ and $\\widetilde{\\theta}\_t$ denote the predicted latent variable and mixing parameter at time $t$. When the number of latent variables $k$ increases, the complexity comes from the computation of the Hadamard product $\\lambda\\circ\\widetilde{s}\_{\\tau|t}$ ($k$ operations). The learning of $\lambda\in [0,1]^k$ involves solving an online optimization whose variable is in $[0,1]^k$. In practice, the state dimension $n$ and action dimension $m$ are often much larger than $k$ and the number of latent variables is not a bottleneck in all these key steps. For example, in our drone navigation example, external disturbances are caused by two latent variables, i.e., the wind speed and rain intensity. Similarly, in the second voltage control example, $k=3$ corresponding to PV integration, wind generation, and residential consumption.

Question 2. Due to space limitations, we provide a detailed discussion in the following official comment.

Question 3. This is a great question. Dealing with nonlinear dynamics is challenging, since there may not exist an explicit expression of the total regret $J(\pi(\lambda)) - J^{\star} = \sum_{\ell=0}^{T-1}\psi_{\ell,T}^{\top}(\lambda) H \psi_{\ell,T}(\lambda)$ as a function of some trust parameter $\lambda$. It is therefore nontrivial if optimizing $\lambda$ online is still possible. We have been thinking about this issue by constructing surrogate functions and would like share more insights if you are interested.

Question 4. We have compared DISC with vanilla MPC and LQR in our experiments. We will add more experiments that compare DISC with robust control methods such as the $\mathcal{H}_{\\infty}$ controller to compare the consistency and robustness.

Thank you once again for your constructive feedback. We will incorporate all your suggestions to enhance the clarity and applicability of our work. Due to space limitations, we defer our responses to the remaining issues in the official comment.

REFERENCES

[Hyvarinen 2016] Aapo Hyvarinen, and Hiroshi Morioka. "Unsupervised feature extraction by time-contrastive learning and nonlinear ica." Advances in neural information processing systems 29 (2016).

[Khemakhem 2020] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. "Variational autoencoders and nonlinear ica: A unifying framework." In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020.

[Yang 2022] Xiaojiang Yang, Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and Junchi Yan. "Nonlinear ICA using volume-preserving transformations." In International Conference on Learning Representations. 2022.

[Zheng 2022] Yujia Zheng, Ignavier Ng, and Kun Zhang. "On the identifiability of nonlinear ICA: Sparsity and beyond." Advances in neural information processing systems 35 (2022): 16411-16422.

Thank you for your review. We appreciate your comments on the strengths and acknowledge the concerns regarding the clarity of our writing and presentation. Please find our clarification below.

1. Clarification of Basic Concepts in Abstract The detailed meanings of "best-of-both-worlds" and "consistency and robustness" in our abstract are explained below. FYI, the original sentence reads

"Our results highlight a first-of-its-kind “best-of-both-worlds” integration of machine-learned predictions, thus lead to a near-optimal consistency and robustness tradeoff"

In our study, the predicted latent variable time series $(\widetilde{s}_{\tau|t} : t \leq \tau \leq \overline{t})$ and the mixing parameters $(\widetilde{\theta}_t : t \in [T])$ can vary in accuracy, potentially being either precise or erroneous. The term “best-of-both-worlds” highlights that our method's ability to ensure reliable performance (a feature that we refer to as robustness) for worst-case scenarios. Mathematically, this corresponds to a bounded competitive ratio even if the prediction errors are significant. Conversely, if the predictions are near-optimal, our method demonstrates adaptability, and behaves similarly to an optimal controller (a.k.a consistency). This dual capability is encapsulated in our main result, which establishes a bound on the (expected) competitive ratio, illustrating our method's effectiveness across varying levels of prediction accuracy. Note that the control policy is not given the prediction accuracy beforehand.

For further context, this can be seen in our main result that bounds the (expected) competitive ratio:

$\mathbb{E}\left[\mathsf{CR}(\mathsf{DISC})\right]\leq 1+o(1)+ O\left(\rho^{2w}\right) + \underbrace{O\left(\sum_{i=1}^{k}\frac{{\color{blue}\overline{\varepsilon}(i)}}{\Omega(T/w)+{\color{blue}\overline{\varepsilon}(i)}}\right)}_{\text{*Best-of-both-worlds utilization*}},$ where the $o(1)$ term hides quantities that vanish when the total number of steps $T$ increases; $k$ is the number of latent variables generating the perturbations; $\rho\in (0,1)$; $w$ denotes the prediction window size and each $\overline{\varepsilon}(i)$ (for $i=1,\ldots,k$) denotes the prediction error corresponding to each latent component. The expectation is over the randomness of the control policy if $f$ is a general mixing function (note that the control policy becomes deterministic for the linear mixing case).

The meaning of Best-of-both-worlds utilization in the last term reflects the following guarantees on robustness and consistency:

Robustness The robustness can be inferred from the bound above that even if $\overline{\varepsilon}(i)$ becomes large, the last term is still bounded since $\overline{\varepsilon}(i)$ appears in both the numerator and the denominator. It's worth mentioning that such a bounded competitive ratio is not achievable by traditional control policies such as MPC.

Consistency On the other hand, if $\overline{\varepsilon}(i)=0$, the resulted $i$-term becomes zero as well. This yields a smaller competitive ratio compared to traditional robust control policies that do not utilize disentangled predictions.

We hope the above explanations make sense. We will revise our abstract and introduction based on the concern and please let us know if there are further questions. Thank you again for the comment.

2. Technical Details

The formal definitions of $(1+o(1))$-consistent and $\omega(1)$-consistent are provided in Definition 2.2 (Section 2). Let $\mathsf{CR}(\pi;\varepsilon)$ denote the competitive ratio of a control policy $\pi$ with a fixed prediction error $\varepsilon$. Specifically, A policy $\pi$ is $\gamma$-consistent if its competitive ratio satisfies $\mathsf{CR}(\pi;\varepsilon)\leq \gamma$ for $\varepsilon=0$ and $\kappa$-robust if $\mathsf{CR}(\pi;\varepsilon)\leq \kappa$ for all $\varepsilon$.

We will add a pointer to clarify in our introduction section. Thank you for pointing out this issue.

3. Update Rule

In fact, the update rules of $\lambda_t$ are provided formally in later sections for both the linear mixing and general mixing cases correspondingly. Section 3.2 briefly overviews our control policy, without specifying the ONLINE-PROCEDURE. The main reason is because we use different update rules for linear mixing and general mixing settings, so we chose to specify them in later sections, together with the theoretical results.

Update Rule for Linear Mixing In Section 4.2, Eq. (11), we have detailed the FTRL procedure for learning $\lambda_t$ in our context:

Thank you for the great question. In the implementation of the algorithm, we solve the follow-the-regularized-leader (FTRL) optimization in Eq. (12) by the following online mirror descent (OMD) for the linear mixing setting:

${\tau_{t} = \tau_{t-1}-\frac{\beta}{2}\nabla_\lambda\left(\zeta_{t}^\top H \zeta_{t}\right), \ \ \lambda_t = \mathsf{Proj}_{\mathcal{I}}(\tau_t)},$ where $\\tau\_t$ represents our internal parameter updates, $\\nabla\_\\lambda$ denotes the gradient with respect to $\\lambda$, and $\\mathsf{Proj}\_{\\mathcal{I}}$ is the projection onto the feasible set $\\mathcal{I}$ of $\lambda$.

The OMD above is shown in [Hazan 2010] to be equivalent to the FTRL procedure in Eq. (12).

More precisely, the gradient of the cost gap at time $t\in [T]$ is given by \\nabla\_{\\lambda}\\left(\zeta\_t^\\top H \\zeta\_t \\right) =\\mathsf{J}^{\\top}(\\zeta\_t)\\left(H+ H^\\top\\right)\\zeta\_t = 2 \\mathsf{J}^\\top\\left(\\zeta\_t\\right)H\\sum\_{\\tau=\\underline{t}}^{t}\\left(F^\top\\right)^{t-\\tau} P \\left(\\theta s\_t - \\widetilde{\\theta}\_\\tau\\left(\lambda\\circ\\widetilde{s}\_{t|\\tau}\\right)\\right), where we have used the fact that $H\coloneqq B(R+B^\top P B)^{-1} B^\top$ is symmetric. Moreover,

Dear Reviewer 4nx1,

We hope our updated clarification of the Update Rule for Linear Mixing is helpful. Please don't hesitate to reach out if you have any further questions or need additional information. Thank you.

Further responses to the major concerns, including those regarding technical novelty are included in the formal rebuttal and will be revealed on August 6, 11:59 PM AoE, after the rebuttal period.

Question 1: Learning $A$ and $B$

Yes, in our work, we focus on learning the latent variable time series and the mixing parameters, and designing a corresponding control policy. For the ease of presentation, we suppose $A$ and $B$ are given. Standard system identification methods can be used to estimate $A$ and $B$ offline. If $A$ and $B$ are unknown before applying the control policy, a new online control policy needs to be designed and this is nontrivial for the dynamics considered in this work. For instance, it is not straightforward if learning $A$ and $B$ online would lead to an additional logarithmic regret as in [Agarwal 2019], while simultaneously guaranteeing the near-optimal consistency and robustness tradeoff presented in our work, as well as the competitive ratio guarantee. This suggests an interesting future direction, and we will add a discussion in the revised manuscript.

Question 2: Stronger statement in Theorem 4.1

This is a great question. In fact, the statement in Theorem 4.1 can be tightened to the following:

"There exists a constant $C>0$ such that if the $\lambda$-confident policy $\lambda$-CON is $(1+C)$-consistent, then it cannot be $o(T)$-robust, even if the mixing parameter estimate is perfect, i.e., $\overline{\eta}=0$."

In the proof of Theorem 4.1, Appendix G, assuming $\lambda$-CON is $(1+o(1))$-consistent implies $\lambda$ has to be $\mathbf{1}_k$. Setting $\mu=\log T$ directly implies this stronger claim. Indeed, the adversary can choose any $\mu>0$ to make the total policy cost as large as possible if $\lambda$ is fixed. Moreover, the assumption on the consistency can also be tightened from $(1+o(1))$ to $(1+O(1))$, i.e., $\lambda$-CON is $(1+C)$-consistent for some $C$ satisfying

C\\leq \\frac{1}{J^{\star}} \\lambda\_{\min} \\left((R+B^\\top P B)^{-1}\\right) \\sum\_{t=0}^{T-1} \\left\\|B^{\\top} \\sum\_{\\tau=\\ell}^{\\overline{T}}\\left(F^\\top\\right)^{\tau-\\ell} P \\theta\_{\\tau} \\left( \\left(\\mathbf{\frac{1}{2}}_{k}\\right)\\circ s\_{\\tau|\\ell} \\right)\\right\\|^2.

We can construct matrices $A,B,Q,R$ and latent variables to make $C$ a positive constant. Then, applying the inequality (33) in Appendix G, we know $\lambda$ can not be the all-zero vector. The same argument in Lines 818-821 implies that $\lambda$-CON cannot be $o(T)$-robust.

In our original presentation, we used $\omega(1)$ to indicate that learning $\lambda$ yields a fundamentally better tradeoff. We thank the reviewer once again for the constructive feedback, and we will add additional remarks in our manuscript to further clarify the points discussed above.

REFERENCES

[Agarwal 2019] Naman Agarwal, Elad Hazan, and Karan Singh. "Logarithmic regret for online control." Advances in Neural Information Processing Systems 32 (2019).

Thank you for recognizing the novel aspects and strengths of our work, especially the dual focus on consistency and robustness in the use of machine learning predictions within control systems. We also appreciate your acknowledgment of the importance of the problem we are addressing and the novel contribution of Theorem 4.1, which highlights the limitations of the fixed $\lambda$.

Regarding the concern about the novelty of the techniques used in our research:

1. Technical Novelty While it is true that the foundational benchmarks such as competitive ratios [Goel 2022a, Sabag 2022], and the basic idea of learning trust parameters online [Li 2022] we employed are established within the field, our work is still technically novel in the following aspects.

a. $k$ latent variables: We study a control problem $x_{t+1} = Ax_{t}+ B u_t + f(s_t;\theta), \quad t\in [T]$ with perturbations depend on $k$ latent variables. Bounding the competitive ratio of the proposed control policy requires a significantly different technique (see Appendix E, the competitive analysis in Step 4 of the proof of Theorem 4.2) compared to those appeared in the previous literature. For example, special treatments of the total regret $J(\pi(\lambda)) - J^{\star} = \sum_{\ell=0}^{T-1}\psi_{\ell,T}^{\top}(\lambda) H \psi_{\ell,T}(\lambda)$ are necessary to decompose it as $O\left( \sum_{i=1}^{k}\left(\frac{w\overline{\varepsilon}(i) }{J^{\star}}\left(\lambda(i)\right)^2+\frac{\left(1-\lambda(i)\right)^2}{C_{0}\left\|f^{-1}\right\|^2} \right)\right)$ (see Eq. (43)) in the proof of Lemma 6, Appendix H.

b. Nonlinear $f$: Compared with the linear quadratic control models in [Goel 2022b] and [Li 2022], nonlinear perturbations lead to further technical challenges on learning $\lambda$ since they form a Hadamard product inside the mixing function as in the $\lambda$-CON policy (Eq. (5)): $u_t =-Kx_t- Y\sum_{\tau=t}^{\overline{t}}\left(F^\top\right)^{\tau-t} P f\left(\lambda\circ\widetilde{s}_{\tau|t};\widetilde{\theta}_t\right),$ where $Y:= (R+B^{\top}PB)^{-1}B^{\top}$.

c. Quadratic form transformation lemma: More importantly, we do not assume bounded variations of system perturbations and errors, which are often required in existing self-tuning policy [Li 2022], and robust control literature, such as robust model predictive control MPC [Berberich 2020] to ensure worst-case guarantees. For instance, the competitive ratio bound in [Li 2022] depends on a term $O\left(\frac{1}{J^{\star}}\left(\mu_{\mathrm{VAR}}(\mathbf{w})+\mu_{\mathrm{VAR}}(\widehat{\mathbf{w}})\right)^2\right)$ where the self-variation $\mu_{\operatorname{VAR}}(\mathrm{y})$ of a sequence $\mathrm{y}:=\left(y_0, \ldots, y_{T-1}\right)$ is defined as

We greatly appreciate all the questions received. Further responses to the major questions, including those regarding Questions 1-11, are included in the formal rebuttal and will be revealed on August 6, 11:59 PM AoE, after the rebuttal period.

In this official comment we summarize our responses provided earlier and provide detailed answers to Question 3 and 12.

1. Errors in Formulas Thank you for identifying errors in various formulas throughout the manuscript. We will conduct an extensive review of the entire manuscript. We present responses to the issues one-by-one in the rebuttal section.

2. Theoretical Claims and Assumptions Thank you for your feedback regarding the clarity of certain theoretical claims in our manuscript. We defer our responses to the rebuttal section, where we carefully respond the questions to ensure that all your concerns are adequately addressed.

3. Nonlinear Mixing Case Thank you for your insightful comments regarding our focus on linear mixing in the experiments. We provide additional discussions in Global Rebuttal.

Below we include additional answers to Questions 3 in this comment.

Question 3. For further context, consider the application of online optimization strategies to minimize total regret as depicted in Equation (8). At each time $t$, we obtain $\psi_{\ell,0}(\lambda),\ldots,\psi_{\ell,t}(\lambda)$. For $\psi_{\ell,t}(\lambda)$, since the summation is over $\tau\in \\{\ell,\min\\{\ell+w-1,t-1\\}\\}$, $\psi\_{\ell,t}:\mathcal{I}\rightarrow\mathbb{R}^{n}$ is a function that depends on $t\in [T]$. But in the original total regret formula (up to time $t$) $\sum_{\ell=0}^{t-1}\psi_{\ell,T}^{\top}(\lambda) H \psi_{\ell,T}(\lambda)$ relies on future $\psi_{\ell,t}(\lambda),\ldots,\psi_{\ell,T-1}(\lambda)$. Thus, the formulation of the offline optimization (8) does not conform to a canonical framework suitable for online optimization, necessitating the role of Lemma 1. We will revise the manuscript to clarify.

Thank you so much for your high-quality, very detailed, and insightful comments. They are very helpful and we sincerely appreciate a lot for what you dedicated in the reviewing process. Below are our responses to the questions:

Question 1. Thank you for pointing out the typo. The matrix $H \coloneqq B(R+B^\top P B)^{-1} B^\top$ is indeed defined in Appendix E (see Theorem E.1). We will revise the manuscript to include a definition of $H$ before Eq. (8).

Question 2. Thank you for catching this error. The correct notation indeed involves summation over $\ell$ rather than $t$, aligning with the notation used in Lemma 1. The corrected equation should read:

We will update this in the manuscript.

Question 3. In Equation (9), the upper index on the summation operator is indeed accurate. The clarification in Lines 168-169 highlights that the per-step cost $\psi_{\ell,T}^\top(\lambda) H \psi_{\ell,T}(\lambda)$ in Equation (8) is dependent on the time horizon $T$. Therefore, it cannot be solved as a canonical online optimization problem at each time $t$ since in general $\psi_{\ell,T}(\lambda)\neq \psi_{\ell,t}(\lambda)$ for $t<T$ (see Figure 7 in Appendix D for a pictorial explanation). This necessitates a tailored approach for online learning enabled by Lemma 1, which addresses this challenge and derive an equivalent optimization.

Question 4. Yes, in Equation (8), the total regret is defined as a function of the sequence $\lambda = (\lambda_0, \ldots, \lambda_{T-1})$, where each $\lambda_t$ within the sequence, belonging to the interval $[0,1]^k$, is learned online. We will clarify in the revised manuscript. Thank you for the comment!

Question 5. In Figure 2(a), the offline optimal controller is derived given all future perturbations $(Cw_t:t\in [T])$ of the following discrete-time kinematic model (similar models can be found in [Li et al. 2019] and [Yu et al. 2020]):

for some matrices $A\in\mathbb{R}^{4\times 4}$, $B\in\mathbb{R}^{4\times 2}$, $C\in\mathbb{R}^{4\times 4}$. A detailed description of the drone piloting problem can be found in Example 1, Appendix B.2 and Appendix C.1. The offline optimal control is known as (see [Goel et al. 2022])

$u^*_t=-Kx_t-(R+B^{\top}PB)^{-1}B^{\top}\psi^*_t$

where $\\psi_t^*=\\sum_{\\tau=t}^{T-1}(F^{\\top})^{\\tau-t} PCw_t$, assuming the terminal cost-to-go matrix is $P$. Note that this is equivalent to the MPC-type action in Eq. (5) with true predictions of latent variables and mixing parameters.

Question 6. Thank you for the great suggestion. We have been working on examples for nonlinear mixing functions, and have discussed them in the Global Rebuttal.

Question 7. Thank you for the question. The $\\ell_4$-norm appears because of the Hadamard product between $\\mathbf{1}\_k-\\lambda$ and $s_{\\tau}$. The detailed derivation is as follows.

Applying the Cauchy–Schwarz inequality, the term in Line 778 satisfies

P=Q+A^\top P A - A^\top PB (R+B^\top P B)^{-1} B^\top PA.

Thank you once again for all the comments. We appreciate your feedback on the preliminary results with nonlinear mixing functions and will incorporate the clarifications in the revised manuscript. Please let us know if there are any additional concerns or suggestions.