Enabling Optimal Decisions in Rehearsal Learning under CARE Condition

Thanks for the valuable feedback! We hope our responses can address your concerns.

Q1. Extension for non-Gaussian noise.

A1. Thanks for your insightful question. We would like to clarify that the Gaussian noise assumption is primarily used to establish theoretical guarantees. For cases where this assumption does not hold, AUF can also be addressed effectively by:

• Gaussian approximation. It is common to approximate irregular distributions using a Gaussian distribution. E.g., Laplace's approximation [1] can approximate unimodal distributions by fitting a Gaussian distribution centered at the mode. To validate the empirical effectiveness of our method under non-Gaussian noise, we provide additional experiments as shown in Fig. 1 of Anon. Link.

• A numerical solution. An alternative empirical approach can be used to solve AUF in Eq. (2) by training a sampler (e.g., normalizing flow) for the potentially non-Gaussian noise using residuals. This method involves sampling $n$ noise realizations from the trained sampler and selecting $z^\xi$ to maximize the number of instances where $Y\in S$. This can be formulated as the following mixed-integer linear program: $\begin{aligned}\max_{e_i\in\{0,1\}, z^\xi}\ &\sum_{i=1}^n e_i\\\\s.t.\ &M(Ax+Bz^\xi+C\epsilon_i)-d\leq(1-e_i)\alpha,\ i\in[n]\end{aligned}$ where $e_i$ indicates whether $i$-th sample is successful, $\alpha$ is a sufficiently large constant vector to tolerate failed samples, $M,d,A,B,C$ are defined in Thm. 3.6. This approach can be empirically applied with non-Gaussian noise.

We will incorporate this discussion into the revised paper. Thanks!

---

Q2. Potential distribution family making Thm. 3.6 valid.

A2. Thanks for your question. The validity of Thm. 3.6 stems from the fact that Gaussian belongs to both elliptical and exponential families. Hence, the decomposition (Eq. 8) and the log-concavity (lines 611–628) can be established.

The main challenge in extending these results to general distributions, such as the exponential family, is that simultaneously satisfying both the decomposition and log-concavity is difficult. Additionally, the PDF of a general distribution can be complex, limiting the applicability of existing analytical techniques. Recognizing the importance of extending theoretical results to non-Gaussian cases, we are currently investigating a similar theoretical foundation for the log-concave family (including Gaussian, uniform, and Laplace distributions, etc.) as part of our future work. In this case, we find that techniques used in this paper are insufficient, and the Prékopa–Leindler theorem [2] may provide a useful tool for proving log-concavity of generalized distribution families. However, explicit characterization for these distributions remains a significant challenge.

Finally, we would like to emphasize that proving theoretical optimality for probabilistic optimization is challenging, even under Gaussian noise. Establishing theoretical guarantees under Gaussian noise provides insights that extend beyond Gaussian scenarios and is a common practice in studies on previous rehearsal learning [3, 4], time series analysis [5], and control problems [6], among others.

We will incorporate this discussion in the revised paper. Thanks!

---

Q3. Detailed data information.

A3. Thanks for your question.

• For synthetic data. Let $V=[feature_1,feature_2,C_{our},C_{cpt},P_{our},P_{cpt},NCT,TPF]$ with variables defined in Fig. 5. The data generation follows $V=PV+\epsilon,\epsilon\sim\mathcal{N}(0,\Sigma)$: $P=\begin{pmatrix}0&0&0&0&0&0&0&0\\\\0&0&0&0&0&0&0&0\\\\10&0&0&0&0&0&0&0\\\\0&10&0&0&0&0&0&0\\\\0&0&2.0&0.4&0&0&0&0\\\\0&0&0.5&1.3&0&0&0&0\\\\0&0&1.6&0&-0.9&0&0&0\\\\0&0&-1&0&0.9&0&0&0\end{pmatrix},\Sigma=10^{-2}\begin{pmatrix}4&0&0&0&0&0&0&0\\\\0&4&0&0&0&0&0&0\\\\0&0&6&0&0&0&0&0\\\\0&0&0&3&16&0&0&0\\\\0&0&0&16&6&0&0&0\\\\0&0&0&0&0&6&0&0\\\\0&0&0&0&0&0&4&0\\\\0&0&0&0&0&0&0&12\end{pmatrix}.$ As $feature_{1,2}$ have no parents, their covariance can be marginalized thus omitted. Parameters are identical to those used in the code provided in Supp. Material and experimental results can be reproduced by running the code.

• For real-world data. The dataset records values of environment variables in Bermuda, and the decision target is to maintain a high NEC (net coral ecosystem calcification). Due to space limits, we welcome further questions on the dataset and will provide additional details in Appx. E.2 of the revised paper.

Thanks again!

---

References:

[1] Pattern Recognition and Machine Learning, 2006

[2] On logarithmic concave measures and functions, 1973

[3] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments, NeurIPS 2024

[4] Rehearsal Learning for Avoiding Undesired Future, NeurIPS 2023

[5] Time series analysis and its applications, 2000

[6] Online Linear Quadratic Control, ICML 2018

Thanks for your detailed feedback! We hope our responses address your concerns.

Q1. Relevance to RL research.

A1. Thanks for your insightful question. Below, we clarify the connection between RL and AUF problem and explain distinctions.

• Connection between RL and AUF. When interactions are available, AUF can be formulated as an MDP (or reduced to a Bandit if no state transitions) and solved using RL methods. Specifically, at round $t$, state $x_t$ is observed, then an action is taken by altering $z_t$, and the environment provides a reward: $r=1$ if $y_t\in S$ else $r=0$. This aligns with our RL baselines (Tab. 2&3) and additional experiments in A2.

• Distinctions between RL and rehearsal learning. Rehearsal learning focuses on a specialized decision-making setting where interactions are limited or even unavailable. In this case, variables X,Z,Y follow a structured generative process parameterized by $\theta$ (in MDP formulation, transition dynamics and reward function also depend on $\theta$). Leveraging this fine-grained structure, reliable decisions can be made using only a small set of observational samples (for estimating $\hat\theta$) without necessarily requiring interactions with the environment. In contrast, online RL methods rely on extensive interactions for effective policy learning. While offline/hybrid online-offline RL might seem applicable, direct applications would also be unsuitable, as actions significantly shift the distribution of Y as discussed in Sec. 2, rendering the reward functions different between offline and online data.

While RL/MDP provides a general framework for decision-making, real-world interactions can be costly or even unethical. E.g., in healthcare, doctors cannot freely experiment with untested treatments. In such cases, leveraging structural knowledge enables effective policy without interaction-based exploration while also enhancing interpretability.

Finally, integrating a rehearsal-learning policy as an initialization for online RL could serve as a potential bridge between our work and the broader RL community, offering a promising yet challenging direction to improve RL’s sample efficiency in certain cases.

We will incorporate this discussion into the revised paper. Thanks!

---

Q2. RL baseline&Discussion on $\hat{\theta}$.

A2. Thanks for your question. We conduct additional experiments in Fig. 3 of Anon. Link to show that policy gradient RL methods can be effective for AUF when sufficient interactions are available. Note that Fig. 4c in paper illustrates the performance of rehearsal methods w.r.t. the number of offline observational samples, making it unsuitable for evaluating RL methods (including offline/hybrid online-offline RL as discussed in A1).

Furthermore, we address implications of using an $\epsilon$-approximate model $\hat{\theta}$, i.e., $||\hat{\theta}-\theta||\leq \epsilon$:

• Empirical validation. As shown in [1], $||\hat{\theta}-\theta||$ decreases as the number of observational samples increases. Hence, Fig. 4c shows that our approach's performance improves as $\epsilon$ decreases in practice.

• Theoretical analysis. We would like to clarify that our theoretical guarantees are established on the AUF probability conditioned on $\hat\theta$ rather than on the true parameter $\theta$, as discussed below Thm. 3.8. Moreover, deriving a regret bound similar to those in RL literature is generally challenging because the rehearsal-learning policy is obtained by solving a probabilistic optimization (Eq. 2). In this case, although the output action $z$ depends on $\theta$, properties such as L-Lipschitz continuity are extremely difficult to establish, making regret analysis nontrivial. However, if the function $\ell \circ z(\cdot)$ can be assumed to be L-Lipschitz continuous, then the following bound can be derived: $\ell(z(\hat{\theta}))-\ell(z^*)\leq L||\hat{\theta}-\theta||_2\lesssim\mathcal{O}(\frac{1}{\sqrt{n}}),$ where $n$ is the number of observational samples. The last inequality follows from [1]. Finally, online updating of $\hat{\theta}$ can also be incorporated in our approach via an offline parameter-update step after each decision round.

We will refine this discussion in the revised version. Thanks!

---

Q3. The meaning of $p$.

A3. Thanks for your feedback. In Tab. 1, $p$ represents the dimensionality of actionable variables $z^\xi$.

Rehearsal learning has two-stages: (i) estimate $\theta$ from historical samples; (ii) make decisions after observing $x$. Only stage (ii) requires immediate actions, and Tab. 1 reports its time complexity. The sensitivity of theoretical results is discussed in A2.

We will incorporate this clarification in the revised version. Thanks again!

---

References:

[1] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments, NeurIPS 2024.

Thanks for your valuable feedback and appreciation of our work! We hope that our responses can address your concerns.

W1. Further Discussion on the CARE Condition.

A1. Thanks for your insightful question. In practice, the dimensions of $\mathbf{Y}$ are often dependent, meaning that the common region defined as {$a_i \leq Y_i \leq b_i$}, $i=1,\dots,|\mathbf{Y}|$ does not necessarily form a canonical rectangle w.r.t. the covariance matrix of $\mathbf{Y}$. However, the CARE condition remains useful because the desired region $\mathcal{S}$ can be manually specified by the decision-maker, as decisions are made after obtaining some system information. The overall decision-making process in rehearsal learning proceeds as follows:

• Step 1. Initially, historical observational data samples are available, enabling estimation of the underlying system parameters $\theta$.
• Step 2. Consequently, matrices such as $C$ and $\Sigma$ (for Def. 3.5 of the CARE condition) are available before decision-making. Since the desired properties of the target $\mathbf{Y}$ are typically determined by the decision-maker, the target region can be manually adjusted based on the estimated $C$ and $\Sigma$ to ensure compliance with the CARE condition.
• Example. In portfolio optimization, defining a desired region such as $\{a_1 \leq \text{returns} \leq b_1\}$ and $\{a_2 \leq \text{risk} \leq b_2\}$ ensures a balanced strategy with both favorable returns and controlled risk. Alternatively, using desired region such as $\{a \leq \alpha\cdot\text{returns}+\beta\cdot\text{risk}\leq b\}$ and $\{c \leq \beta\cdot\text{returns}-\alpha\cdot\text{risk} \leq d\}$ (where $\alpha, \beta\neq 0$ and depend on $C$, $\Sigma$) can also achieve a balanced strategy while satisfying the CARE condition, allowing for a theoretically optimal solution.

Hence, the CARE condition can be satisfied by appropriately adjusting the desired region if such adjustments are feasible. Additionally, the inner CARE embedding of {$a_i \leq Y_i \leq b_i$} can be computed using techniques for identifying axis-parallel rectangles within polygons [1]. By defining a basis aligned with the specific matrix space, the CARE embedding of the polygon {$a_i \leq Y_i \leq b_i$} can be derived. Additional experimental results provide support of this argument as detailed in A2.

We will incorporate this discussion in the revised version. Thanks!

---

W2. Additional case studies of inner CARE embedding.

A2. Thanks for your thoughtful question. We conducted additional experiments for cases where the dimensions of $\mathbf{Y}$ remain dependent even after alterations, demonstrating that our proposed inner CARE embedding method is effective and flexible. Specifically, the experiments include two types of desired regions: (i) Circular desired region, with the inner CARE embedding computed using Eq. (4); (ii) Axis-aligned rectangular region {$a_i \leq Y_i \leq b_i$}, with the inner CARE embedding computed using [1]. The visualizations are presented in Fig. 2 of Anon. Link, and the detailed performance, measured by AUF probability, is listed below:

These results show that our proposed generalization method, i.e., inner CARE embedding, is effective in several irregular or non-canonical scenarios.

Additionally, we would like to emphasize that finding a maximal axis-parallel inner rectangle for an irregular region is a challenging geometric problem [1], especially in high-dimensional spaces. Hence in practice, one can instead identify an inner canonical rectangle (not necessarily the maximal one) and optimize using this inner rectangle as a surrogate, which still enables a deterministic transformation in Cor. 3.7. This approach remains valid, as any inner region of the original retains a probability mass less than or equal to that of the original region, ensuring consistency with Prop. 3.10 due to the non-negativity of the probability density function.

We will incorporate these results and discussions into the revised paper. Thanks!

---

References:

[1] Finding the largest area axis-parallel rectangle in a polygon, Computational Geometry 1997.

[2] Rehearsal Learning for Avoiding Undesired Future, NeurIPS 2023.

[3] Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments, NeurIPS 2024.

---

We also take this opportunity to sincerely thank you for the careful review. Your suggestions are important for further improving the paper. Thanks again!