Variance-Reduced Long-Term Rehearsal Learning with Quadratic Programming Reformulation

Thanks for your detailed feedback and for recognizing the strengths of our work! We hope our responses below can address your concerns.

W1&Limitation. Concerns about the linear assumption.

A1. Thanks for your constructive suggestion. First, we would like to clarify that our linear assumption generalizes the noise distribution to the log-concave family, which is a broader class than that considered in prior works [1, 2]. While this assumption is essential to enabling our QP reformulation, we also further discuss its generality and empirical validity as follows:

• Potential extensions to non-linear settings.
  The core ideas behind our algorithm are not limited to linear models. Specifically, our decision-making objective only requires that the relationship between the aggregated outcome $\sum_i Y_i/T$ and the sequence of alterations $\\{z_i\\}$ can be parameterized with additive aggregated noise. That is, if there exists a functional family $f$ such that $\sum_i Y_i/T = f(\theta; \\{z_i\\}) + \tilde{\varepsilon},$ for some parameters $\theta$ and aggregated noise $\tilde{\varepsilon}$, then the optimization at each decision round $k$ can proceed by minimizing the surrogate loss $\\|f(\theta; \\{z_i\\}) - s_k\\|_2^2,$ where $s_k$ denotes the adjusted center of the desired target region. This observation suggests that our formulation is not inherently tied to linear dynamics and may naturally extend to some general non-linear settings.

• Empirical validation of the linearity assumption.
  To assess the adequacy of the linear assumption, we fitted the model using least squares and evaluated its fit using standard metrics: (i) per-coordinate $R^2$ scores, (ii) normalized mean squared error (NMSE), and (iii) normalized mean absolute error (NMAE).

  The linear model achieves average $R^2$ scores of 0.869/0.686, NMSE of 0.212/0.401, and NMAE of 0.305/0.468 on the synthetic / real-world (Bermuda) datasets, respectively. For other datasets, this validation step can also be performed to verify the suitability of the linear assumption.

Taken together, these points support both the current validity and future extensibility of our formulation. We thank the reviewer again for raising this important issue.

W2. Linear dynamical system and control theory.

A2. Thanks for your insightful question. Our approach differs from classical control methods, such as online linear quadratic control (LQR) [3, 4], in several key ways:

• Different Motivations. Traditional LQR assumes a linear dynamical system of the form $x_{t+1} = Ax_t+Bu_t+w_t$ and aims to control the state trajectory $\{x_t\}$ over time. In contrast, our goal is to maximize the probability that the aggregated outcome $\sum_iY_i/T$ lies within a desired region, without being concerned about the specific trajectory of $X$, $Y$, or $Z$ at individual time points.

• Distinct Formulations. Our system cannot be reduced to the standard LQR formulation, as the alteration can also affect variables within the same time point. Moreover, our problem is fundamentally defined by Eq. (3), and Eq. (5) is a heuristic reformulation under certain assumptions. Even so, the resulting QP in Eq. (5) differs from that in LQR: while online LQR decomposes into per-step QPs (w.r.t. control variables like $\sum_t(x_t^\top Qx_t+u_t^\top Ru_t)$) due to additive structure, our objective involves aggregation of altered variables $z_t$ through matrix $F$, making the problem cannot be decomposed thus more challenging.

• Connection to MPC. While our method shares a high-level similarity with Model Predictive Control (MPC) [5], in that we optimize a future sequence of actions but only execute the first, our predictive model is different from classic ones. We rely on structured dependencies (see A3) that go beyond the scope of standard MPC formulations.

We will incorporate this discussion into the revised version. Thanks again!

W3. Comparison with RL methods.

A3. Thanks for your thoughtful comment. It appears there may be a misunderstanding of our problem setting, and we appreciate the opportunity to clarify the differences between our structural approach and conventional RL, including model-based RL.

Your question highlights a key advantage of the rehearsal learning framework. Simply formulating a reward function with an indicator $\mathbb{I}(Y \in S)$ is insufficient for addressing decision problems with structural dependencies (like causal relations), especially those involving confounding. Even when the indicator function is deterministic, it is crucial to distinguish between samples collected from different data distributions (observation or interaction/intervention), as this distinction fundamentally impacts the decision-making process.

• Observation vs. Intervention.
  Our setting relies solely on observational data to make immediate decisions, where structural (e.g., causal) relations exist. In contrast, standard RL methods assume interventional data from agent-environment interactions. Treating observational samples $\langle x, z, \mathbb{I}(Y \in S) \rangle$ as if they were standard RL tuples $\langle s, a, r \rangle$ implicitly assumes that the observational and interventional distributions are identical, an invalid assumption in the presence of confounding.

  E.g., suppose the structure includes: $X\rightarrow Y$, $X\rightarrow Z_a\rightarrow Y$, $Z_a^\prime\rightarrow Z_u$ and $Z_a\leftarrow Z_u\rightarrow Y$, where $X$ can be viewed as the state in RL, $Z=(Z_a,Z_a^\prime)$ are two actionable variables, and $Z_u$ is an observed but unactionable confounder. Observational data might show a strong correlation between $Z = z$ and $Y\in S$ given $X=x$, leading an RL agent to learn $z$ as the optimal action incorrectly. In contrast, our method can identify the optimal action $z^\star\ne z$ without requiring costly exploration by leveraging structural information and applying backdoor criteria [10].

• Structural vs. Statistical Knowledge.
  While model-based RL can incorporate prior knowledge, it typically captures statistical transition dynamics (e.g., $p(s_{t+1} \mid s_t, a_t)$) rather than fine-grained structural dependencies. Our method leverages the latter, which is essential to distinguish causation/influence relations from purely correlation.

• New Experiments. To empirically illustrate this distinction, we compared our method with both model-free (DDPG [6], SAC [7]) and model-based (MBPO [8]) RL agents. Each was allowed 100 environment interactions, the same sample budget available to our method from observational data.

  Method	Ours	[6]	[7]	[8]
  AUF Probability	$0.827\pm 0.044$	$0.207\pm 0.055$	$0.182\pm 0.039$	$0.228\pm 0.027$

All experiments were run on the same dataset, with the same region $S$ and horizon 1, across 100 rounds and 10 random seeds. RL methods underperform due to their inability to exploit structural information, resulting in slow convergence. Notably, SAC can eventually reach comparable performance, but only after ~6,000 interactions, underscoring the sample inefficiency of RL in our context.

In summary, our work is motivated by the need to incorporate structural knowledge, which is essential for making decisions in the presence of confounding. We hope this clarifies the distinction and illustrates why our approach is better suited to this class of problems than standard RL pipelines.

Q2. Non-isotropic noise.

A4. Thank you for this insightful question. Your observation is very sharp and touches upon a key aspect of our theoretical contribution.

Indeed, when the target variable $Y$ is multi-dimensional, the noise in the aggregated outcome $\sum_i Y_i/T$ is possible to be non-isotropic. However, this does not affect our decision-making strategy. Let us revisit the core objective of our AUF problem: given an observation $X=x$, we aim to choose an alteration sequence $\\{z_i\\}$ to maximize the probability of $\sum_i Y_i/T \in S$. In this context, our Prop. 3.1 guarantees that the entire stochasticity of the aggregated outcome is captured by the aggregated zero-mean noise term, $F\tilde{e}$, which is unaffected by the choice of the choosed action variables $\\{z_i\\}$.

Consequently, our optimization in Eq.(5) is performed over the sequence of action variables $\\{z_i\\}$. The choice of norm for this optimization does not alter the stochastic term $F\tilde{e}$ that governs the randomness of the outcome $\sum_i Y_i/T$. The reason our method successfully maximizes the probability of the outcome falling into the desired region $S$ is guaranteed by established results in probability theory, involving key lemmas such as Anderson's Theorem [9]. Should you be interested, a detailed proof is provided in Appx. D.3.

We hope our detailed responses have addressed your concerns. If any points remain unclear, we would be happy to elaborate further. In light of these clarifications, we would be grateful if you would reconsider your evaluation. Thank you again for your thoughtful feedback!

Reference

[1] Rehearsal Learning for Avoiding Undesired Future. NeurIPS 2023.

[2] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments. NeurIPS 2024.

[3] Online Linear Quadratic Control. ICML 2018.

[4] Naive Exploration is Optimal for Online LQR. ICML 2020.

[5] Model predictive control: Theory and practice—A survey. Automatica 1989.

[6] Continuous control with deep reinforcement learning. ICLR 2016.

[7] Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML 2018.

[8] When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.

[9] The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. American Mathematical Society 1955.

[10] Causality: Models, Reasoning and Inference.

We sincerely thank you for your detailed feedback and for recognizing the strengths of our work, including its strong theoretical grounding, clear presentation, and the practical importance of the long-term AUF setting. We hope our responses below fully address your concerns.

W1. Empirical evaluation of the assumptions.

A1. Thanks for this constructive suggestion. To empirically assess the impact of violating our assumptions, we conducted new experiments where the (i) symmetric desired region and/or (ii) unique target assumptions are violated. The table below presents results on a synthetic dataset (with dimensions of $X$, $Z$, and $Y$ set to 3, 2, and 2, respectively, and a decision horizon of 9). To ensure a fair comparison, when violating assumption (i), we chose a desired region with a total area similar to that in the no-violation setting.

These results show that although performance degrades when assumptions are violated, our method remains substantially more robust than existing approaches [1, 2], particularly when the unique target assumption fails. This offers strong empirical support for the practical utility of our approach, even beyond the scope of our theoretical guarantees.

W2&Q1. A deeper analysis of scalability.

A2. Thanks for raising this important point. We agree that understanding the computational scalability of our method is crucial, especially for large dimensions and long horizons. As shown in Fig. 4 (middle) of our paper, our approach is already significantly faster than prior rehearsal-based methods [1, 2] for long horizons with fixed dimensionality.

To further address scalability, we propose two practical strategies: (i) In high-dimensional settings, heuristic methods can be developed to select a key subset of variables $Z_s \subset Z$ and approximately solve Eq. (5) over $Z_s$; (ii) For long horizons, the matrices ($M$, $N$, $H$ in Eq. (5)) that depend on the horizon length can be precomputed and cached offline to accelerate real-time inference.

We will incorporate this discussion into the revised version. Thank you!

W3&Q4. Empirical comparison with other methods.

A3. Thanks for this insightful question. We agree that positioning our method within the broader decision-making landscape is important. While we primarily compare with rehearsal-based methods due to (i) the specific problem setting (detailed in Appx. A), and (ii) prior empirical evidence showing that alternative methods (e.g., RL) achieve limited effectiveness in this setting [1, 2], we conducted additional experiments in response to your suggestion.

We include comparisons with representative RL methods (DDPG [3], SAC [4]) and a recent causal bandit algorithm [5]. For fairness, we allowed the RL agents to interact with the environment for 100 episodes—the same sample budget available to our method from observational data.

All experiments were conducted on the same dataset, with the same desired region $S$ and a fixed horizon 1, over 100 rounds and 10 random seeds. The table reports the AUF probability mean and variance. Our method clearly outperforms these baselines, as the rehearsal-based approach is more effective at leveraging fine-grained structural dependencies among variables and observed context $X$. We also observed that SAC can eventually reach comparable performance after ~6,000 training steps. However, such extensive interaction is often not feasible in our problem setting.

Q2. Applicability to varying decision horizons.

A4. Thanks for the thoughtful question.

We would like to respectfully clarify that the decision horizon $T$ is a user-specified parameter (thus should not be unknown). For instance, if a trader aims to maximize profit over the next 10 months, they would simply set $T=10$.

For a varying decision horizon setting, our main theoretical result (Thm. 3.4) can be straightforwardly applied to such scenarios. Thm. 3.4 provides a strong guarantee: for any finite future horizon $T$ and any decision step $t$, the solution to Eq.(5) is optimal in maximizing the AUF probability of the aggregated target $\sum_i Y_i/T$, assuming Ass. 3.3 holds. If decision-makers choose to specify different horizons $T$ at different steps $t$, our method remains applicable and optimal in this context. One only needs to recompute the corresponding matrices ($M$, $N$, $H$) in Eq.(5) for the specified $T$.

Q3. Interpretability and sparsity constraints.

A5. Thank you for highlighting this important issue. When the horizon is 1, the solution is easy to interpret: it directly maximizes the probability that the immediate outcome $Y$ lies within the desired region $S$. For longer horizons, interpretation becomes more subtle: the optimal alteration at each step maximizes the probability that the average outcome across the horizon falls in $S$. Using a Texas Hold'em analogy: if the goal is to win the current hand, the strategy should reflect immediate hand strength. But if the goal is to maximize winnings over 100 hands, one may initially build a specific table image (e.g., tight or loose) that improves long-term return, even if it sacrifices short-term gains.

Regarding sparsity, since Eq. (5) is a convex QP, we can include sparsity-inducing regularizers (e.g., L1 penalty, as in LASSO) without losing convexity. This helps generate more interpretable alteration vectors for domain experts.

We will add this discussion to the revised version. Thanks!

Other questions. Suggestions and formatting.

A6. Thanks for your detail and helpful suggestions. We will include a discussion of potential negative consequences in the Appendix and Checklist of the revised version.

Regarding reference formatting: we would like to respectfully clarify that our use of \citet and \citep is intentional, following standard citation practices to distinguish between in-text (e.g., Author [X]) and parenthetical ([X]) citations. Meanwhile, we have used the \bibliographystyle{unsrtnat} style consistently throughout the paper. Thanks!

We hope that our responses and additional experiments address your concerns. We would greatly appreciate your reconsideration of the paper in light of these clarifications. Thank you again for your valuable and constructive feedback.

Reference

[1] Rehearsal Learning for Avoiding Undesired Future. NeurIPS 2023.

[2] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments. NeurIPS 2024.

[3] Continuous control with deep reinforcement learning. ICLR, 2016.

[4] Soft actorcritic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.

[5] Additive Causal Bandits with Unknown Graph. ICML, 2023.

Thanks for your valuable feedback and appreciation of our work! We hope that our responses can address your concerns.

W1. Concerns about the assumptions.

A1. Thanks for the insightful question. While our theoretical guarantees (Thm. 3.3) rely on certain assumptions, our method remains effective in practice even when these assumptions are partially violated, as evidenced by the new experimental results in A2.

The latter two assumptions, i.e., unique targets and symmetric desired regions, are often satisfied in real-world applications, as they are typically determined by the decision-maker. As for the linear system assumption, we would like to clarify that we have already generalized the noise distribution beyond the Gaussian assumption used in prior works [1, 2], allowing for symmetric log-concave noise. Moreover, potential non-linearities could be addressed using standard techniques such as kernel methods [3, 4]. Due to space constraints, we kindly refer you to our response to Reviewer JXfA (A1) for a more detailed discussion of the linearity assumption.

We will incorporate this discussion into the revised paper. Thanks!

W2. Concerns about the experiments.

A2. Thanks for your question. To address your concern, we conducted additional experiments on a new dataset to evaluate our method's performance when the assumptions are violated. The table below shows the performance on this new dataset (where dimensions of $X$, $Z$, and $Y$ are 3, 2, and 2, respectively and the horizon is set to 9) when assumption (i) symmetric desired region and/or (ii) unique target assumptions are violated. To ensure a fair comparison, when violating assumption (i) (symmetric desired region), we used a desired region with a total area similar to that of the no violation setting.

As the results show, while not explicitly discussed in previous work, violating these assumptions degrades performance of the rehearsal learning approaches, especially the unique target assumption. Although optimality is not guaranteed in these settings, our method still significantly outperforms the prior works.

Finally, we would like to respectfully clarify that whenever Ass. 3.3 holds, our method is theoretically guaranteed to perform well, as established by Thms 3.4, 3.5, and 3.6.

W3. Concerns about the scalability.

A3. Thanks for your thoughtful question. Indeed, in sequential decision-making problems, both dimensionality and the time horizon can be large. We would like to highlight that, as shown in Fig. 4(mid) of our paper, our method is significantly faster than existing rehearsal learning approaches [1, 2] for a long time horizon when the dimensionality is fixed. For further reducing the executing time in decision-making, we propose two feasible directions: (i) In high-dimensional settings, heuristic methods can be developed to select a key subset of variables $Z_s \subset Z$ and approximately solve Eq. (5) over $Z_s$; (ii) For long horizons, the matrices ($M$, $N$, $H$ in Eq. (5)) that depend on the horizon length can be precomputed and cached offline to accelerate real-time inference.

We will incorporate this discussion into the revised paper. Thanks!

Q1. Potential distribution shift over time.

A4. Thanks for your question. The rehearsal learning process involves two sequential steps:

• Step 1. Estimate/Update the underlying system parameters $\theta$ from historical observational data.
• Step 2. Use $\theta$ to construct system matrices (e.g., $A$, $B$ in Eq. (2)), and solve Eq. (5) for the decision alterations.

The model uncertainty and potential distribution shifts primarily affect the parameter estimation in Step 1. To handle model uncertainty (e.g., a time-varying graph structure), one could segment the data based on different structures and learn a separate set of parameters for each. To address distribution shifts of the data, if the shift is gradual, some online learning methods can be employed to update the parameters [5]; and if the shift is abrupt, a restart mechanism can be implemented [6], i.e., discard the old parameters upon detecting a major shift and re-estimate them from new samples.

Q2. How to deal with dynamic environments.

A5. Thanks for your question. As noted in A4, our framework follows a two-step sequential process. The challenge of dynamic environments, where system parameters $\theta$ change over time, primarily pertains to the parameter estimation in Step 1. This issue has been explored in prior work on rehearsal learning [2], and similar adaptive strategies could potentially be incorporated into our framework as well.

Q3. How to integrate inter additional constraints.

A6. Thanks for the insightful suggestion. Prior works in rehearsal learning [1, 2] have considered integrating additional constraints (e.g., decision cost). If such constraints define a convex feasible set for the decision variables, they can be naturally incorporated into our QP reformulation (Eq. (5)) without compromising convexity, allowing for efficient optimization.

We will incorporate this discussion into the revised paper. Thanks!

We also take this opportunity to sincerely thank you for the careful review. Your suggestions are very important for further improving the paper. Thanks again!

References:

[1] Rehearsal Learning for Avoiding Undesired Future. NeurIPS 2023.

[2] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments. NeurIPS 2024.

[3] Kernel-based conditional independence test and application in causal discovery. UAI 2011.

[4] Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks. JMLR 2016.

[5] Adaptive online learning in dynamic environments. NIPS 2018.

[6] Learning to optimize under non-stationarity. AISTATS 2019.

We sincerely thank you for your detailed feedback and for recognizing the strengths of our work! We hope that our responses can adequately address your concerns.

W1&Q1. Extension to nonlinear systems.

A1. Thanks for your insightful question. The core ideas of our proposed algorithm can be extended to general nonlinear settings. In principle, as long as the relationship between (i) the mean of the aggregated outcome $\sum_i Y_i/T$ and (ii) the selected alteration sequence $\\{z_i\\}$ can be captured, e.g., via a parameterized functional form $f$ such that $\sum_i Y_i/T = f(\theta; \\{z_i\\}) + \tilde{\varepsilon},$ for some parameter $\theta$ and residual $\tilde{\varepsilon}$, then we can turn to optimize $\arg\min_{\\{z_i\\}}\\|f(\theta; \\{z_i\\}) - s_k\\|_2^2$ at decision round $k$, where $s_k$ denotes the adjusted center of the desired region.

Moreover, our main theoretical results can be adapted to such nonlinear settings. Taking the above example, if the surrogate loss $\\|f(\theta; \\{z_i\\}) - s_k\\|_2^2$ is convex with respect to $\\{z_i\\}$, then Thm. 3.4 still applies. This is because the proof of Thm. 3.4 is based on residual analysis, with a key step showing that the aggregated error $\tilde{\varepsilon}$ remains within the log-concave distribution family (Prop. 3.2). Therefore, as long as the nonlinearity preserves this property, our main results remain valid.

We will incorporate this discussion in the revised version. Thanks!

W2&Q2. Concern about the graph structure.

A2. Thanks for the thoughtful question. We are not entirely sure if we have fully understood your concern; if our response does not address your point accurately, we would be happy to further clarify during the upcoming author-reviewer discussion period.

Our framework focuses on a specific class of decision problems in which structural relations among variables can be leveraged to improve decision-making, particularly when opportunities for interaction are limited. This line of work falls under the umbrella of rehearsal learning [1, 2, 3], with connections to causal bandits and causal RL [4, 5].

Furthermore, even when structural information is not explicitly available in the raw data, it can often be inferred using well-established structure learning methods [6], including in sequential [7] or cyclic [8] settings. Our method can then be applied based on the learned structure.

We will incorporate this clarification into the revised paper. Thanks again!

W3&Q3. Potential applications.

A3. Thanks for the question. We would like to address it in two parts:

• First, our proposed decision-making method leverages structural information. Leveraging structural information to aid decision-making has broad applications in domains such as healthcare [9], finance [10], and environmental science [11]. For instance, Simpson’s paradox [12] illustrates that accurately evaluating the effect of a treatment $X$ on an outcome $Y$ requires accounting for confounding variables, highlighting the need for causal structure in certain decision-making scenario. Recent advances in causal bandits [4, 5] and rehearsal learning [2, 3] similarly emphasize the importance of structure-aware decision-making in complex environments.

• Second, our work extends prior rehearsal learning frameworks by explicitly incorporating sequential information, which is crucial in many real-world scenarios. For example, a physician prescribing medication over time for a chronic disease, or a financial institution regularly updating trading strategies. In such cases, methods that do not guarantee single-step optimality may accumulate errors over time [2, 3]. In contrast, our method guarantees optimality at each step (Thm. 3.4) and controls variance across the full sequence (Thm. 3.5). Our experimental results further support the effectiveness of our method in these settings.

Your suggestions are constructive and important for further improving the paper. Thanks again!

References:

[1] Rehearsal: Learning from prediction to decision. FCS 2022.

[2] Rehearsal Learning for Avoiding Undesired Future. NeurIPS 2023.

[3] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments. NeurIPS 2024.

[4] Bandits with unobserved confounders: A causal approach. NIPS 2015.

[5] Causal bandits: Learning good interventions via causal inference. NIPS 2016.

[6] DAGs with NO TEARS: Continuous Optimization for Structure Learning. NIPS 2018.

[7] DYNOTEARS: Structure learning from time-series data. AISTATS 2020.

[8] Discovering Cyclic Causal Models by Independent Components Analysis. UAI 2008.

[9] Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nature Machine Intelligence 2020.

[10] The causal impact of media in financial markets. Journal of Finance 2011.

[11] Environmental controls on modern scleractinian coral and reef-scale calcification. Science Advances 2017.

[12] Causation, Prediction, and Search. MIT Press 2000.

Dear Reviewer JXfA,

We sincerely thank you for your time and feedback on reviewing our paper. As the discussion period is nearing its end, we would like to politely remind you and would greatly appreciate it if you could take a moment to consider our responses.

If there are any remaining concerns about our work, we would be happy to engage in further discussion.

Yours sincerely,
The Authors