Maximizing the Value of Predictions in Control: Accuracy Is Not Enough

Thank you for your comments! Please find our response below.

• Connection with predict-then-optimize literature.

We would like to point the reviewer to our section on Related Literature, where we specifically highlighted the connection of our work with the decision-focused learning (DFL) literature, which includes the "predict-then-optimize" setting. (We indeed cite the seminal "Smart Predict, then Optimize" (Elmachtoub and Grigas, 2022) as reference [14].) In our revision, we will clarify that "decision-focused learning" and "predict-then-optimize" are different terms that refer to the same idea.

As you mentioned, a key insight is that minimizing a standard prediction error (e.g., MSE) may lead to suboptimal downstream decisions. Thus, the loss function for training the predictor needs to depend on the downstream optimization task. This is similar to the insight behind our Example 3.3.

A major difference between our work and typical predict-then-optimize literature, though, is that our controller must decide control actions sequentially in a dynamical system, where the predictions are revealed in an online process. The controller can also build its knowledge from past disturbances and predictions to infer future disturbances or the optimal control action. Therefore, our online setting presents unique challenges compared with making a one-shot decision for a classic optimization problem. For this reason, we also focus on the evaluation of a given predictor and leave an end-to-end optimization of the predictor parameter $\theta$ as future work (see Section 5).

We will discuss this connection in the revision.

Thank you for your comments! Please find our response below.

• Adding more qualitative explanations or intuitions.

Thank you for the suggestion! We will add more explanations in the revision based on the feedback from the review process.

• Applicability of Condition 4.1 beyond LQR

Thank you for pointing this out! Since Condition 4.1 is about the Q and C functions, specific assumptions on the stage cost $h_t$ and the dynamics $g_t$ are required to verify it. We want to clarify that Condition 4.1 is more general than the LQR setting. For example, we show it holds under Assumption 4.5 in Section 4.1, which allows non-quadratic cost functions. We will add a clarification in the revision.

• Access to the dynamics and reconstruction of the past disturbances.

Yes, we assume the controller has full knowledge of the dynamical function $f_t$, while the disturbance $W_t$ captures the uncertainties in the dynamics. With linear dynamics discussed in Sections 3 and 4, the controller can reconstruct $W_t$ using the transition tuple $(x_t, u_t, x_{t+1})$. This does not affect our prediction power because we include past disturbances (until time step $t-1$) in the history $I_t(\theta)$, which is available to the controller at time step $t$.

• Potential extensions to partially observed settings.

Thanks for pointing out this extension! We do not think there is a straightforward way to achieve this due to the need to distinguish the disturbances in the system dynamics from the noises in the observation model, and the function form of the optimal policy is hard to define because it depends on the whole history. We will add this point to Section 5 in the revision.

• Clarification about the bias and the structure of predictions.

Our setting does not assume the prediction $V_t(\theta)$ to have any specific form of relationship with future disturbances. And $V_t(\theta)$ does not need to be an unbiased estimator of $W_t$ or any future disturbances.

We note that our Examples 3.3 and 3.4 may leave the false impression that $V_t(\theta)$ must be some structured function (e.g., linear function) of $W_t$. However, our intention is to show how a general phenomenon like the misalignment between accuracy and the prediction power may occur with simple constructions. We will add a clarification in the revision.

• Applicability to temporally correlated disturbances and predictions.

Our definition of the prediction power and the theoretical results (except Theorem 4.8 in Section 4.1) allow the predictions and the disturbances to have temporal correlations across different time steps. When the cost function $h_t$ is non-quadratic, we require stronger joint-Gaussian and independence assumptions (see Assumption 4.7) to establish Condition 4.2. However, we believe it is possible to relax them and provide a justification in Appendix E.

• Intuitions about why more accurate predictions might not improve the prediction power.

First, we want to clarify that this is not because the more accurate prediction is noisy or biased. To see this, note that any prediction $V_t(\theta)$ needs to pass through its corresponding optimal policy $\pi_t^\theta$ to decide the action. If needed, the function $\pi_t^\theta$ can help eliminate the bias and reduce the variance.

Perhaps a simple example that analogs the intuition is computing $\mathbb{E}[Y|X]$. If we replace $X$ with $X'=2X$, the variance of the input increases, but $\mathbb{E}[Y\mid X']$ remains the same. Back to our setting, the prediction power will not change if we replace the prediction $V_t(\theta)$ with $2V_t(\theta)$ because the optimal control policy $\pi_t^\theta$ changes correspondingly.

Second, we would like to provide an intuitive explanation of the idea behind Example 3.3: The key takeaway is that MSEs of estimating each individual entry of the multi-dimensional disturbance $W_t$ are insufficient to decide the prediction power. The off-diagonal entries of the covariance matrix $Cov[W_t|V_t(\theta)]$ also matter, and their impact on the prediction power depends on the specific dynamics $(A, B, Q, R)$. Therefore, a general accuracy metric (like MSE) that is unaware of the costs/dynamics can be misaligned with the prediction power.

Please let us know if the above clarifications help, and we are happy to add them in the revision.

Thank you for your comments! Please find our response below.

• The distinctions between $\xi$ and $\iota$.

$\xi$ is the realization of the problem instance $\Xi$ that contains all disturbances and predictions for all time steps $0, 1, \ldots, T-1.$ $\iota_t(\theta)$ is the realization of the history $I_t(\theta)$, which contains all past disturbances and predictions that are observed until an intermediate time step $t$. We will add a combined explanation of $(\xi, \Xi, \iota_t(\theta), I_t(\theta))$ in the revision, because they are used frequently in our paper.

• How $Q$ and $C$ functions in (3) depend on the particular realization $\iota_t(\theta)$.

We define our instance-dependent $Q$ function and the cost-to-go function $C$ based on the problem instance $\Xi$. Therefore, the $Q$ and $C$ values are determined by the specific realizations $\xi$ of $\Xi$ (see Eq. (3)). Since the history realization $\iota_t(\theta)$ is a part of the problem instance realization $\xi$, our $Q$ and $C$ functions depend on $\iota_t(\theta)$ but are not decided by $\iota_t(\theta)$. To recover the classic definition of $Q$ and cost-to-go functions, one can take the conditional expectations of them given the history $\iota_t(\theta)$.

• Moving parts of Appendix A.3 to the main body.

Thanks for providing this suggestion! We will move (21) and some discussions from Appendix A.3 to the main body to further clarify what is “planning according to the $w_{\tau\mid t}^\theta$.”

• Highlighting technical novelties in a proof sketch.

Thank you for providing this suggestion! Here are some technical novelties that we would like to highlight in the revision. First, in Theorem 4.3, we decompose the task of bounding the prediction power for the whole horizon $T$ to the verification of two conditions at each intermediate time step $t$. Our proof takes inspiration from the performance difference lemma, but we adopt novel methods to bound the difference between cost-to-go functions with the conditional covariance of the predictive policy's actions. Second, to show Lemma 4.6 and Equation (14), we prove and use properties of infimal convolution (see (35) in Appendix C.1) to preserve strongly convexity, smoothness, and the covariance of the input functions or variables. Therefore, we can verify Conditions 4.1 and 4.2 through backward induction, while the optimal policy does not have a closed-form solution.

• Definitions of the baseline prediction and the prediction parameter space in a relevant problem.

In the adaptive video streaming problem, a controller determines which quality level of a video segment to download, aiming to optimize the user’s experience. The predictions are about the future network bandwidth within a lookahead window. The baseline ($\theta = 0$) could be “no-prediction” (e.g., some classic methods are not planning-based.) Other candidates ($\theta = 1, 2,\ldots$) can be a classic exponentially weighted moving average predictor, an ML-based predictor trained on a specific dataset, and other alternative sources of predictions.

• On Line 209, $w^\theta$ should be $v^\theta$.

Sorry for the confusion. Here, we refer to using the predictions $v_t(\theta)$ to compute $w_{\tau\mid t}^\theta$, which is defined as the conditional expectation of future disturbances given the current history (see Line 176). We will add a pointer to remind the readers in the revision.

• Naming collision between the sequence of (disturbance, prediction) pairs and the identity matrix on Line 222.

Thanks for catching this! We will change the notation of the identity matrix to $\mathbb{I}$ and add an explanation in the revision.

I appreciate the author's clarifications. I think the biggest downside of this paper is the notational burden. I am afraid that most readers would get tripped up by the zoo of variables. So, while this paper is good and I would vouch for its acceptance, I think it may not be award-worthy. Therefore, I will keep my current score.

Thank you for your comments! Please find our response below.

• Key differences between our work and the previous literature in [1].

We respectfully disagree with the reviewer's point regarding the originality and would like to highlight several key novelties of our work in comparison to [1]. First, the misalignment issue between prediction power and accuracy (Section 3.1) would not exist under the exact-prediction setting in [1]. Second, discussing the connections with online policy optimization (Section 3.2) only makes sense in our setting, because the optimal predictive policy in [1] is fixed and known even when the distribution of the disturbances is unknown. Third, we extend our proof framework beyond the LQR setting in Section 4, while the proof techniques in [1] rely on the closed-form solutions of LQR. To address the challenge, we develop a novel proof framework in Theorem 4.3, which reduces the problem of comparing two policies on the whole horizon $T$ to verifying conditions about each individual policy at every intermediate time step $t$. Further, we prove and use the properties of infimal convolution (see (35) in Appendix C.1) to preserve strongly convexity, smoothness, and the covariance through backward inductions, which are critical for bounding the prediction power when closed-form solutions do not exist.

• Relationship with online prediction algorithms and how to adapt them.

Online prediction algorithms study methods for making predictions. In contrast, our work focuses on the process of using various forms of predictions to decide control actions. For example, uncertainty quantification (UQ) has recently received significant attention in the field of online prediction algorithms, providing not only point estimates but also confidence intervals. To use UQ as the predictor in our framework, the prediction $v_t(\theta)$ should be the output of UQ at time $t$, including both the point estimate and the confidence interval.

• Clarifications about the distribution of $W_t$, the temporal cross-correlation, and the underlying assumptions.

We would like to clarify that we never assume $W_t$ is an independent distribution. Instead, all disturbances and predictions (i.e., the problem instance $\Xi$) follow a joint distribution. In other words, conditioned on the realizations of past disturbances and predictions, the distribution of $W_t$ may change. We need Assumption 2.2 about $\Xi$, which means its sampling cannot be affected by the controller’s states/actions. For example, the weather and its forecasts do not change because a person goes outside. Under Assumption 2.2, the disturbances and the predictions at different time steps can have correlations.

• Implicit assumptions about the distribution of $W_t$ in Assumption 4.7.

There are no implicit assumptions. We explicitly state the distribution of the pair $(W_t, V_t(\theta))$ at each time step in Assumption 4.7. Since different time steps are independent, you can derive the distribution of $W_{0:T-1}$ from Assumption 4.7, which is a joint Gaussian distribution without temporal correlations.

• Clarification about the sign of the prediction power and the parametric form of the predictions.

When the baseline predictor is no prediction, the prediction power is guaranteed to be non-negative. This is because the optimal policy for the no-prediction case is also a predictive policy when predictions are available, i.e., the controller can simply ignore the predictions. But the prediction power could be zero, for example, when the predictions are independent of the disturbances. We will clarify this in the revision.

We do not require the prediction $V_t(\theta)$ to have any specific form as a function of $\theta$. Generally speaking, $\theta$ is primarily used for distinguishing different predictor candidates. Additional assumptions on the form of $V_t(\theta)$ may be necessary for future directions, such as optimizing the prediction power $P(\theta)$ as a function of $\theta$ (Section 5).

• The specific choice of function $V_t(\theta)$ in Example 3.3.

The intuition behind Example 3.3 does not require a very specific choice of the function $V_t(\theta)$: The underlying idea is that the MSEs of estimating each individual entry of the multi-dimensional disturbance $W_t$ are insufficient to decide the prediction power. The off-diagonal entries of the covariance matrix $\mathrm{Cov}[W_t \mid V_t(\theta)]$ also matter, and their impact on the prediction power depends on the specific dynamics $(A, B, Q, R)$. Therefore, a general accuracy metric (like MSE) that is unaware of $(A, B, Q, R)$ can be misaligned with the prediction power. We hope this intuition is convincing, and we will add more discussion in the revision.

• Applying estimators (with memory) on top of the provided prediction vector $V_t(\theta)$.

Under our definition, the prediction power does not increase if one replaces the original prediction $V_t(\theta)$ by any function of the history $I_t(\theta)$, because this additional step cannot increase the information available at time step $t$. Indeed, since we do not require any particular form of the decision-maker, the predictive policy class $\pi_t$ (Definition 2.3) is general enough to allow using filters like RLS as an intermediate step for deciding the control actions.

• Intuition behind M-GAPS and the notations $V_t(1), V_t(2)$ in Example 3.4.

Intuitively, M-GAPS works by taking the gradient of the cost function with respect to the policy parameters at every time step, and it takes gradient steps to update $\Upsilon_t$, allowing it to converge towards the optimal policy parameters under certain assumptions. Our goal is to highlight the connection between prediction power and online policy optimization, so the specific online policy optimization algorithms and their proofs are not the primary focus here. We will add more intuition in the revision.

As we mentioned before, the prediction parameters “1” and “2” here are used solely to distinguish between different predictors. We will clarify this in the revision.

• Assumptions on $h_t$ for Condition 4.1 to hold.

Condition 4.1 does not impose explicit assumptions on the stage cost function $h_t$. However, since $Q$ and $C$ functions depend on $h_t$, structural assumptions about $h_t$ are necessary to establish Condition 4.1. An example is Assumption 4.5, which assumes that $h_t$ is both strongly convex and smooth for every time step $t$.

• The signs of $\mu_A$ and $\mu_B$ in Assumption 4.5.

Thanks for pointing this out! $\mu_A$ does not need to be strictly positive, but $\mu_B$ does. This is because the prediction power can be zero if $\mu_B$ equals zero (see Theorem 4.8). We will clarify this in the revision.

• Exchanging the strong convexity and smoothness assumptions with more general growth rate assumptions.

Thanks for suggesting this direction of further generalization! We believe the quadratic growth/positive-definite covariance matrix conditions are natural generalizations of the LQR setting in Section 3. Following the intuition of Figure 3, we believe that further generalization of the growth rate condition in Condition 4.1 is possible if one can find the corresponding "variance" condition (i.e., the counterpart of Condition 4.2). This is an interesting future direction.

• Clarification about the first step in Appendix A.2 (16).

Thank you for catching this! The equation holds because the cost-to-go function $C_t^{\pi^\theta}(x; \xi)$ can be expressed as $Q_t^{\pi^\theta}(x, \pi_t^\theta(x; \iota_t(\theta)); \xi)$. Substituting the expression of $\pi_t^\theta(x; \iota_t(\theta))$ into the expression of $\mathbb{E}[Q_t^{\pi^\theta}(x, u; \Xi)\mid I_t(\theta) = \iota_t(\theta)]$ in Proposition 3.1 gives (16) in Appendix A.2. We will add this clarification in the revision.