Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding

Dear Reviewer Lkaz,

Many thanks for taking the time to read our work and for your positive feedback.

Q1. Compatible worlds

"I'm a little confused about how should I decide whether a world is compatible or not. Apparently, I can start with any confounder space dimensionality, prior $p(z)$ and model architectures and then estimate the parameters using ELBO, and then I have a compatible world? How far I can go here or what should I see to say a world is not compatible here?"

Thank you for your question, which allows us to stress the different contributions of our work.

Formalising compatible worlds. In our basic definition in Section 4, a compatible world is any model that maximises the likelihood of the observational distribution. In the limit of infinite data and no environment stochasticity (i.e., no epistemic or aleatoric uncertainty), finding a compatible world is equivalent to solving this optimisation problem.

In the identifiable setting, all compatible world models should collapse to the true identifiable distribution of confounders and confounder-dependent policy and value function model. In the non-identifiable setting, our motivation is not to attempt to recover the confounders directly, but rather to obtain a plausible set of confounder models that fit the observational data.

One of the main contributions of our work is therefore this formalism of compatible world models and of delphic uncertainty. This basic definition does not require any assumptions: any measure of the disagreement between compatible world models captures delphic uncertainty.

Modelling compatible worlds in practice. A second major contribution is therefore one practical instantiation of delphic uncertainty quantification. Within many possible modelling choices, we use variational inference to estimate confounder distributions. This approach demonstrates the effectiveness of delphic offline RL in our experiments but leaves open further work in investigating alternative modelling approaches for compatible worlds (e.g. different priors, potentially based on prior knowledge such as the sensitivity analysis assumption). We investigate this empirically in our answer to Reviewers 3wVu and x1vz.

We hope this answer addresses your confusion and look forward to hearing your thoughts in follow-up.

Q2. Expected pessimistic behaviour

"The behavior policy is in general a context-aware policy. So, I'd expect enforcing similarity to the behavior policy might result in some sort of context awareness. Is this the case? For instance, in the illustrative bandit example, an optimal context-independent policy only explores $a_0$ and $a_2$, which in World 2 is very different from the behavioral policy and seems to be suboptimal compared to a uniform policy. So, if I get it right, the similarity to behavioral policy is encouraged in this setting with unobserved confounders."

This is a very interesting question that resonates with offline RL work, in which it was found that pessimism with respect to epistemic uncertainty, in practice, does result in regularization towards the behaviour policy (Fujimoto and Gu, 2021). Still, by exploiting the reward signal, offline RL methods result in a different policy than that in the data.

The behaviour policy is only accessible in the data in its marginalised form equal to $E_{\nu(z)} [ \pi_b (a|s,z)]$ where the true confounder distribution $\nu(z)$ is nonidentifiable. While enforcing similarity to this marginal $\tilde{\pi}_b(a|s)$ should help minimise epistemic uncertainty (Fujimoto and Gu, 2021; Levine et al., 2020), we wonder how this would provide context-awareness. It is possible that the counterfactual $Q^{\tilde{\pi}_b}$ could be less prone to delphic uncertainty than the $Q^{\tilde{\pi}}$ of any other context-independent policy $\tilde{\pi}$. Still, as our algorithm is formulated to measure and penalise the uncertainty for the policy $\tilde{\pi}$ at hand, we obtain a different behaviour than simply regularising to $\tilde{\pi}_b$ (as implemented by our offline RL baselines). This would be an interesting theoretical investigation in further work, and we have made a note of this possible behaviour of the penalty in Section 5.2 of our revised manuscript.

It is also an interesting observation that pessimism wrt delphic uncertainty in our bandit example could result in a policy that resembles the marginalised behaviour policy. We explicitly investigate this in Appendix E.1 and detect higher delphic uncertainty on actions prone to confounding, thus identifying the optimal context-independent policy as choosing $a_1$. In terms of probabilities, it is indeed closer to the uniform marginal policy $\tilde{\pi}_b(a|s)$ than the offline RL policy, but still differs from -- and performs better than -- $\tilde{\pi}_b$.

Q4. Partial identification methods

"Is it possible to incorporate other unbiased estimation method in confounded POMDP setting like (Shi. et al, 2022)?"

Thank you for providing this additional reference, we will ensure to cite it as an additional related work.

The challenge of nonidentifiable confounding bias in deep offline RL is paramount (Gottesman et al., 2018; Tennenholtz et al., 2022), but remained unaddressed until now. Prior works consider the challenge in OPE (Kallus and Zhou, 2020; Shi et al., 2022) or online RL (Lu et al., 2018; Zhang and Bareinboim, 2019), but our work is the first to propose a practical solution to hidden confounding bias in offline RL. Our work builds on two key contributions:

1. Defining compatible worlds, i.e. that there exists a set of models that maximise the likelihood of the data but may disagree over counterfactual quantities. We propose one (out of many possible) implementation to find this set, but this is not the focus of our work. In fact, as noted in Table 1, any other partial identification assumption (e.g. known $\Gamma$ in sensitivity analysis) could be adopted to achieve this. We now also emphasize this point in Section 5.1.
2. Next, and more importantly, we propose a practical offline RL algorithm to mitigate this challenge. Rather than optimising for worst-case returns and obtaining excessively pessimistic behaviour (as is done in prior work, e.g. Shi et al., 2022, Wang et al., 2020; Kallus and Zhou, 2019), we propose to measure uncertainty and to apply a pessimism penalty. This is motivated by the theoretical guarantees and empirical success of offline RL algorithms that follow this approach (Levine et al., 2020; Yu et al., 2020).

From this perspective, we agree that the following baselines are useful to measure the added value of our approach. We present our results below, for the sepsis environment with $\Gamma=46$, and include this in Appendix E.2, page 28.

1. comparing to the sensitivity analysis framework for estimating the set of compatible worlds (Kallus and Zhou, 2019). We implement this through our variational approach, discarding world models that do not satisfy the $\Gamma$ assumption (all but 2 out of 10 trained compatible world models). This results in a poorer estimate of delphic uncertainty as the resulting delphic ORL performance is negatively affected. [Updated Nov 19]

2. comparing to worst-case value optimisation (Wang et al., 2020; Kallus and Zhou, 2019). As expected, optimising for the worst-case value function is excessively pessimistic:

---

Thank you again for your feedback and questions which helped improve our paper. We would be very grateful if you would increase your score if we have addressed your remaining concerns.

References

Gottesman et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv:1805.12298, 2018.

Kallus and Zhou. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. NeurIPS, 2020.

Levine et al. Offline RL: Tutorial, review, and perspectives on open problems. 2020.

Lu et al. Deconfounding reinforcement learning in observational settings. arXiv:1812.10576, 2018.

Tennenholtz, et al. On covariate shift of latent confounders in imitation and reinforcement learning. ICLR, 2022.

Wang et al. Provably efficient causal reinforcement learning with confounded observational data. NeurIPS, 2021.

Yu et al. Mopo: Model-based offline policy optimization. NeurIPS, 2020

Zhang, J. and Bareinboim, E. Near-optimal reinforcement learning in dynamic treatment regimes. NeurIPS, 2019.

Dear Reviewer x1vz,

Many thanks for your feedback.

Q1. Practical definition of delphic uncertainty

"It seems that the estimator $Var_{w} (Q_{w}^{\pi} (s,a))$ is not an unbiased estimator for the delphic uncertainty as the other two forms of uncertainty can still enter the estimation (noting that $Q^{\pi}_{w}$ is not the conditional expectation given by Theorem 4.2)."

Thank you for pointing out this possible source of confusion.

From our practical estimator of delphic uncertainty in Section 5.1 ($Var_{w} (Q_{w}^{\pi} (s,a))$), we recover its definition in Equation 1 ($Var_{w} (E_{\theta_w}[E [Q_{\theta_w}^{\pi} | \theta_w, w]]$) iff $Q_{w}^{\pi} (s,a) = E_{\theta_w}[ E [Q_{\theta_w}^{\pi} | \theta_w, w]| w ]$.

• In practice, we achieve this by implementing each world model $w$ as an ensemble, with each particle parameterised by a value of $\theta_w$. We approximate the outer expectation over $\theta_w$ as follows: $Q_{w}^{\pi} (s,a)= \frac{1}{m}\sum_{\theta_w} \mathbb{E} [ Q^{\pi}_{\theta_w}(s,a) | \theta_w, w ]$ where $m$ is the number of particles in the world model ensemble. This sample mean is an unbiased estimator of the true expectation.

• Next, the inner expectation can be obtained by implementing each ensemble particle $Q^{\pi}_{\theta_w}(s,a) | \theta_w, w$ as a probabilistic model. We model the distribution of $Q^{\pi_b}$ (details in Appendix C.1) and use its mean to estimate $Q^{\pi}$ using importance sampling and marginalising over $z$.

We hope this decomposition clarifies how our practical implementation of delphic uncertainty approximates our theoretical definition in Section 4. These uncertainty computation details are discussed in Appendix C.1 (page 19, now in bold), but we have modified Section 5.1 to reflect this better.

Finally, note that both pessimism with respect to both delphic and epistemic uncertainty is generally desirable in the confounded offline RL setting we consider. As a result, while our separation of uncertainties proves useful to theoretically and empirically validate our proposed approach, retaining a biased estimate of both uncertainties (e.g. to avoid modelling ensembles of $\theta_w$) is still a useful quantity to penalise.

Q2. Delphic uncertainty on a trajectory-level

"Is it necessary to evaluate the delphic uncertainty on a state-action level rather than evaluating the same thing for the total reward of the whole trajectory under policy $\pi$ ?"

This is an interesting question, as we could certainly compute the variance over compatible worlds for the return of a trajectory. Our reasons for focusing on the state-action level are two-fold:

• First, focusing on $(s,a)$ pairs allows us to leverage the MDP structure of our problem, and thus to pool together delphic uncertainty estimates for similar pairs in different trajectories.
• Next, and perhaps even more importantly, estimating the uncertainty at a state-action level allows us to use this in a practical pessimistic RL algorithm, in which we penalize $(s,a)$ pairs with high uncertainty as in prior work (Yu et al., 2020). Leveraging uncertainty over trajectories could also be conceivable (e.g. re-weighting offline trajectories based on their cumulative delphic uncertainty), but we are not aware of state-of-the-art offline RL algorithms that take this approach.

"I'm not convinced of the soundness of the method here, as there might be some correlations between different state-action pairs in the Q function."

Yu et al., 2020, also propose a method of estimating uncertainty at the state-action level (over the transition function in their model-based setting) and prove that penalising the value in such a way results in a provably pessimistic behaviour (see their Theorem 4.4) and a practical, effective algorithm. Correlations between state-action pairs do not pose a problem in their framework, but we are curious to discuss more if this remains a concern.

Q3. Identifiability of the behavioural policy

"It is assumed that the offline policy is known. What if $\pi_b$ is unavailable so that the importance sampling method has biased in estimating the value function?"

To clarify, our work does not assume that $\pi_b$ is known or identifiable. We learn it in the context of the local confounder distribution $\nu(z)$ in each world model. If $\pi_b$ were known, confounding would not be a source of error, and our problem setting would correspond to offline RL with partial-observability. In this setting (unlike the confounded one), $z$ could be identified from its effect on transitions or rewards -- up to epistemic and aleatoric uncertainty.

Dear Reviewer qrjW,

Thank you very much for your feedback.

Q1. Notation

Thank you for pointing out the notation issue on $P_w$. We have corrected this in our manuscript.

Q2. Value function stochasticity

"Right after Def 4.1, the author proposes to model the Q function as a random element "value model $Q_{\theta_w}$ is defined by some stochastic model". Where is this randomness coming from? Isn't the value function of a policy just a deterministic function?"

In Section 4, we consider $Q$ as a random variable capturing the sum of discounted rewards in a trajectory, $Q = \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t, z)$, for a given $(s,a)$ pair acting as initial values. In the limit of infinite data and under no confounding, randomness in estimating this variable can be attributed to stochasticity in environment transition and reward functions or in the acting policy. This irreducible uncertainty is qualified as aleatoric.

Limited data and hidden confounding introduce additional stochasticity in estimating $Q$. We formalise this in our next answer (Q3) where we clarify our probability setup. This approach mirrors that of prior work in offline RL (Yu et al., 2020), where stochasticity in the environment model or value function is both inherently modeled (measuring aleatoric uncertainty over $Q$) and quantified with respect to different data bootstraps (epistemic uncertainty).

We look forward to hearing your thoughts on our answer.

Q3. Probability setup

"Related to the previous remark, the meaning of Theorem 4.2 is unclear to me. Could the authors detail the probability setup for this theorem? From my understanding, there is a measure over the space of compatible worlds (by the notation $E_w$), and then there is a measure over Q-value functions (by the notation $E_{\theta_w}$). What is the relationship between these measures?"

Thank you for your question. We clarify our probability setup in the following.

• Within the context of a world model $w$ and a parameterization of this world model $\theta_w$, we model a probability distribution over episode returns, $P(Q|\theta_w, w)$. For instance, this captures the environment stochasticity discussed above.
• Next, the distribution of random variable $\theta_w$ depends itself on the specific world model $w$ it parameterises (i.e. the local confounder distribution being estimated, and how policy and environment depend on it). Our data and optimisation algorithm therefore define a distribution $P(\theta_w|w)$ over the world model's parameter space.
• Finally, we posit a distribution $P(w)$ over the compatible worlds $\mathcal{W}$.

Combining these measures, we obtain the following decomposition for the distribution of $Q$:

$P(Q) = \int \int P(Q | \theta_w, w) P(\theta_w | w) P(w) d\theta_w dw$

Note that in our work, $Q$ is subscripted with $\theta_w$ as a reminder for this dependence on $\theta$ and $w$. We have dropped this here for clarity.

Within this framework, we denote the following expectations, using $x$ as a dummy random variable:

$E [x| \theta_w, w] = \int x P(x | \theta_w, w) dx$ $E_{\theta_w} [x| w] = \int x P( \theta_w | w) d\theta_w$ $E_{w} [x] = \int x P( w) dw$

For instance, this allows us to write: $E [ Q ] = \int \int \int Q P(Q | \theta_w, w) P(\theta_w | w) P(w) dQ d\theta_w dw$ $= E_{w} \left[E_{\theta_w} \left[E[Q | \theta_w, w] | w \right] \right]$

For variances, we define:

$Var (x| \theta_w, w) = \int x^2 P(x | \theta_w, w) dx - \left( \int x P(x | \theta_w, w) dx \right) ^2$ $Var_{\theta_w} (x| w) = \int x^2 P( \theta_w | w) d\theta_w - \left( \int x P( \theta_w | w) d\theta_w \right) ^2$ $Var_{w} (x) = \int x^2 P( w) dw - \left( \int x P( w) dw \right) ^2$

Following the proof in Appendix B, we obtain the variance decomposition in Theorem 4.2: $Var(Q)= E_{w} \left[E_{\theta_w} \left[ Var(Q | \theta_w, w) | w \right] + Var_{\theta_w}\left(E[Q | \theta_w, w] | w\right) \right] + Var_{w} (E_{\theta_w}[\mathbb{E}[Q| \theta_w, w]|w ])$

We hope this helps clarify our probability setup and have modified our manuscript to reflect this.

---

Thank you again for your feedback and questions which helped improve our paper. We would be very grateful if you would increase your score if we have addressed your remaining concerns.

References

Yu et al. Mopo: Model-based offline policy optimization. NeurIPS, 2020

Dear Reviewer 3wVu,

Many thanks for taking the time to read our work and for your positive feedback.

Q1. Leveraging confounder estimation at inference time

"Intuitively, it seems that estimating $z$ from the trajectory and using it as $\pi(a|s,z)$ as in POMDP methods would improve performance for later steps $t$, but is such an extension possible? Also, is the proposed method still superior when such a POMDP method is used as a baseline? I'm wondering if the online identification of the world is possible within an episode through such a formulation."

Thank you for this great question, which really highlights the challenge of our problem setting.

In Appendix E.2, we investigate whether estimating and using confounder values $z$ at inference time helps improve the performance of our main baseline CQL. We implement this by allowing both $\{Q,\pi\}$ models to access the history of trajectories (from which $z$ can be estimated). We reproduce our results from Table 6 below: we find that history-dependence actually degrades performance.

This result can be explained as follows. In partially-observed offline settings, incorporating a learned estimate for the confounders $z$ based on the history can be even more prone to confounding, as the identification model is trained on the history distribution of the behavioural policy and not that of the inference policy. This leads to a compounding of distribution shift at inference time, showing "latching behaviour" in imitation (Swamy et al., 2022) and "model delusion" in offline RL (Ortega et al., 2021). For this reason, in our approach to the problem, we chose to focus on history-independent policies where confounding biases only affect the value function, which we directly address through pessimism.

Q2. Partial identification baselines

"The existing approaches for a similar setting are discussed (e.g. using partial identification) but not compared. Isn't it possible to compare them?"

The challenge of nonidentifiable confounding bias in deep offline RL is paramount (Gottesman et al., 2018; Tennenholtz et al., 2022), but remained unaddressed until now. Prior works consider the challenge in OPE (Kallus and Zhou, 2020), online RL (Lu et al., 2018; Zhang and Bareinboim, 2019), or using proxy variables (Wang et al., 2021), but our work is the first to propose a practical solution to hidden confounding bias in offline RL.

Our work builds on two key contributions:

1. Defining compatible worlds, the set of models that maximise the likelihood of the data but disagree over counterfactual quantities. We propose one (out of many possible) implementation to find this set, but this is not the focus of our work. In fact, as noted in Table 1, any other partial identification assumption (e.g. known $\Gamma$ in sensitivity analysis) could be adopted to achieve this. We now also emphasize this point in Section 5.1.
2. Next, and more importantly, we propose a practical offline RL algorithm to mitigate this challenge. Rather than optimising for worst-case returns and obtaining excessively pessimistic behaviour (as is done in prior work, e.g. Wang et al., 2020; Kallus and Zhou, 2019), we propose to measure uncertainty and to apply a pessimism penalty. This is motivated by the theoretical guarantees and empirical success of offline RL algorithms that follow this approach (Levine et al., 2020; Yu et al., 2020).

From this perspective, we agree that the following baselines are useful to measure the added value of our approach. We present our results below, for the sepsis environment with $\Gamma=46$, and include this in Appendix E.2, page 28.

1. comparing to the sensitivity analysis framework for estimating the set of compatible worlds (Kallus and Zhou, 2019). We implement this through our variational approach, discarding world models that do not satisfy the $\Gamma$ assumption (all but 2 out of 10 trained compatible world models). This results in a poorer estimate of delphic uncertainty as the resulting delphic ORL performance is negatively affected. [Updated Nov 19]

2. comparing to worst-case value optimisation (Wang et al., 2020; Kallus and Zhou, 2019). As expected, optimising for the worst-case value function is excessively pessimistic: