We sincerely thank your reviewers and recognize that certain elements of our work might have been misunderstood, which could have influenced the evaluations. Below, we aim to clarify these aspects and remain eager to engage in further dialogue to resolve any lingering doubts. Generally, we have validated the proposed algorithms across a broad range of practical scenarios, including SCMs with various reward functions and transition dynamics, driving, and healthcare, where the proposed algorithms have demonstrated significant improvements in decision-making outcome performance.

“Assumptions and Generalization: The approach relies on certain assumptions about the partial identifiability of the MDP. In practice, these assumptions might not always hold. Even if these assumptions hold, one may not know the type of partial identification in advance, potentially limiting the generalizability of the methods.”

Standard imitation learning in MDPs assumes both the Markov property (Assumption 1) and Causal consistency (Assumption 2) in the demonstration data. In this paper, we generalize the second assumption as we allow the presence of unobserved confounding or violation of overlap. This means that our proposed methods are applicable to all standard MDP instances, and also generalize well to confounded MDPs where standard imitation learning methods do not necessarily obtain a valid solution. Given the wide adoption of MDPs and imitation learning, and the prevalence of unobserved confounding bias, we strongly believe our proposed method could be applied in many real-world scenarios, e.g., autonomous driving and healthcare.

Table 1 provides a recipe for applying our methods in practice, given the violations of assumptions. First, when Assumption 2 holds and there is no unobserved confounder, the standard imitation learning algorithm obtains an effective policy. When there exist unobserved confounders affecting both the reward function and transition distribution (i.e., both equations in Assumption 2 do not hold), the expert performance is not imitable and the learner should explore additional domain knowledge. When unobserved confounders only affect either the reward function or transition distribution, the imitator could apply our proposed method and obtain a robust policy with performance guarantee.

“Complexity and Practicality: The proposed method CAIL-T requires solving a complex constrained optimization problem in Eq. (18) and Eq. (19). It is difficult to solve this optimization problem in practice when neural networks are employed.”

We have developed specific methodologies to address these challenges. As outlined in Alg. 3 (see Appendix) and further elaborated in Thm. 3, our approach employs a structured algorithm designed to effectively navigate the optimization landscape. Specifically, as stated in L566-568, “The intuition for Alg. 3 is: in order to find the worst case, we need to put as less transition probability mass as possible to the state with maximal values, and allocate higher transition probabilities to states with smaller values.” Moreover, with some approximation techniques (Xia et al. (2023)), it is possible to solve it with NNs.

To substantiate the practical applicability of our methods, we have conducted experiments focusing on Driving scenarios (L300-312). These experiments, detailed in Appendix C, successfully implement the proposed framework using neural networks and demonstrate its effectiveness and efficiency in real-world settings. These results underscore our method's viability and its capability to handle the complexities associated with neural network-based implementations.

“Missing Experimental Details: The paper does not provide details about the implementation of the proposed algorithm and the environments. Besides, the source code is also missing. As such, it is difficult to reproduce their experiments.”

We would like to clarify that our paper has indeed included a comprehensive description of the theoretical framework, algorithms, and step-by-step procedures required to understand and implement the proposed method. Theoretical contributions are summarized in Sec. 2 and 3, and further elaborated in Appendix A and B. Experimental and implementation details are provided in Section 4 and further elaborated in the supplementary materials (Appendix C). These environments are common in the causal imitation learning literature, ensuring that our experiments could be easily replicated. As mentioned in the checklist, we will open-source our codebase upon acceptance of our paper.

Thanks for taking the time to read the paper and your reviews. We recognize the importance of integrating a theoretical framework based on causality into IRL, especially when either the transition dynamics, the reward function, or both, are confounded. Below, we provide further clarification on our contributions, theoretical contributions and the experimental design to enhance understanding and address your concerns effectively.

“extension of the standard imitation algorithm (e.g., GAIL) with the partial identification method”

Firstly, applying partial identification to IL is nontrivial due to the complex interplay between unobserved confounders and the dynamics of MDPs. Partial identification typically deals with static settings in causal inference, but MDPs introduce a temporal dimension where past actions and states can influence future outcomes. Our work extends these methods to handle such temporal dependencies, which involves significant theoretical innovation.

Our proposed algorithms might appear neat and straightforward in their final forms, but this does not mean their derivations are simple. The causal bounding results in Eqs. 9-10 are only applicable to a single time step in MDP. Our contribution significantly extends these bounds across time steps, creating a robust framework that can handle the sequential nature of decision-making in practice, where each decision can influence subsequent states and rewards. We also apply special transformations so that they are tailored to GAIL’s training process. We invite the reviewer to check the Appendix and see if our statement is factual.

“hardness results that in the reward-confounding-only case (or the transition-confounding-only case) there still exists instances such that it is impossible to improve over the expert?”

Yes, it is possible. Consider the reward-confounding case as an example. One could construct an MDP model which consists of a sequence of independent contextual bandit models with side information $S_1, S_2, \dots, S_t$. For each contextual bandit model, given any context $s_i$, the expected reward of each arm matches the worst-case lower bound $E[Y_i | s_i, x_i]P(s_i|x_i)$. It follows from Example 1 that the imitator in this worst-case model is unable to improve over the expert’s performance.

“The derivation of the partial identification bound ...”

Thanks for the suggestion. The causal bounds in Eqs. 9-10 follow application of (Manski, 90). More specifically, Manski studies a canonical decision setting where $Z$ represents the context, $X$ is the treatment, and $Y$ is the primary outcome. Manski showed that for any observational distribution $P(X, Y, Z)$, the treatment distribution of intervening on $X$ is bounded by $P(y, x|z) \leq P_x(y|z) \leq P(y, x|z) + P(\neg x |z)$.

To derive Eqs. 9-10, one could focus on the MDP within one timestep $t$. Applying Manski’s bound by setting $S_t$ as the context, $X_t$ as the treatment, and $S_{t+1}$ (or $Y_t$) as the primary outcome leads to the statement. We will include additional discussion in the updated manuscript.

“How to principally specify the (observational) reward function class $\mathcal{R}$ under the unobserved confounding? That is, ..., how to equip the learning agent a suitable function class $\mathcal{R}$ to match the observational reward function of the unknown SCM?”

We do not require the learner to specify the reward function class $\mathcal{R}$ over the interventional reward $E_x[Y|s]$, but only the observational quantity $E[Y|s, x]$. The observed domain of the state-action pair $(s, x)$ is well-specified and can be determined from the demonstration data. In this case, any measurable observational reward $E[Y|s, x]$ can be approximated using some non-parametric function approximators, e.g., neural networks (NNs). The learner could start with a parametric family of NNs, and further restrict it to inject domain knowledge.

“When applied to a non-confounded MDP (i.e., a standard MDP), how well would the algorithm perform when compared with prior arts designed for non-confounded MDPs? ... How far the interventional probabilities deviate from the observational probabilities.”

For a standard MDP with no unobserved confounding, our algorithm could obtain a policy that is less effective compared to the one learned by the prior arts. The performance gap depends on how close the causal bounds are to the ground-truth interventional probabilities. However, we view this as a feature of our methods. When there is no unobserved confounding in the environment, we recommend the imitator to follow standard imitation learning procedures. When the imitator cannot exclude the unobserved confounding prior, our method obtains an imitating policy with performance guarantee. Our algorithms learn a policy by searching over the worst-case model. We believe this risk-averse approach is principled when facing uncertainty in offline imitation learning, since other models could deviate significantly from the ground-truth, leading to significantly inferior performance.

“In Theorem 1, ... $P(X, S, Y)$ are positive, ... What if one drop this assumption? Does the conclusion change drastically?”

We require the probabilities $P(X, S, Y)$ to be positive to exclude degenerated cases that the imitator and the expert performance happen to match. As mentioned in L518-520, "In some degenerated cases when $\mathbb{E}\_{\pi}[Y_{t} \mid s\_{t}] = 0$ and $\mathbb{E}[Y\_{t} \mid s\_{t}] = 0$, it might coincidentally follow that $V\_{\pi}(s_{t}) = 0$, which is equal to $V(s\_{t}) = 0$.”

However, such occurrences are highly impossible in practical scenarios, and are statistically insignificant (measure zero) when one generates MDP instances uniformly at random. Moreover, from an algorithmic perspective, focusing on degenerate cases would divert focus from more prevalent and practically significant scenarios. As the experiment shown within Fig.4 (a), we have randomly generated over $1000$ MDP instance, and in all cases, the minimal performance gap between the imitator and the expert is negative, i.e., the imitator is unable to achieve the expert’s performance. This empirical evidence further supports the assertion that such degenerate cases, while theoretically possible, are exceedingly rare and do not affect the general applicability and robustness of our proposed methods.

We appreciate your reviews and acknowledge that some aspects of our work might have been misunderstood. Below, we provide clarifications and are open to further discussions should there be any residual concerns.

Our experimental design incorporated similar baselines from Zhang et al. (2020), Kumor et al. (2021), and Ruan et al. (2023), chosen for their direct relevance to our framework. The rationale behind this selection was to showcase the potential of partial identification techniques in improving algorithms such as GAIL. These enhancements are particularly important for achieving expert-level performance despite differences in the observation spaces of the imitator and expert—an aspect often overlooked in OpenAI Gym environments. Most current setups in these environments presuppose identical observation spaces and ignore the presence of unobserved variables, both of which are addressed in our study.

“The experiment evaluation seems insufficient considering that there are already some causal imitation learning baselines such as [1], and some more experiment environments such as OpenAI gym for imitation learning.”

(De Haan et al., 2019) explores the underlying causal relationships in the environment, and exploits the sparsity in these relationships to improve the imitator’s performance. However, their problem setting assumes there is no unobserved confounding in the environment, which is the very motivation and the starting point of this paper. That is, our paper studies the challenges of unobserved confounding for imitation learning in MDPs, which is orthogonal to the setting in (De Haan et al., 2019). Consequently, we did not include (De Haan et al., 2019) in the experiments since their methods and ours are not comparable.

We did investigate some existing benchmarks including the OpenAI gym. Unfortunately, as far as we are aware of, no major RL benchmark simulates the presence of unobserved confounding without violating the Markov property. The corresponding causal diagrams of experts are shown in Fig.2 and Fig. 3. Due to these limitations, we have to build novel simulation environments to evaluate our proposed algorithms. Indeed, we intend to release our experiments and hope it could contribute to the existing RL benchmarks by highlighting the challenges of unobserved confounding. We believe that building our own benchmark should be taken as a strength, not a weakness.

”On what tasks are these approaches experimented on, what do these tasks look like, and how does the demonstration data look like? Will the model performance change with a varying number of expert demonstrations?”

Our learning task is similar to the standard imitation learning setting, where the imitator has access to offline demonstration data generated by an expert. The demonstration data is represented as a collection of sequences of trajectories in an MDP. For every sequence, it contains a finite number of state-action pairs $(s_j, x_j)$; the reward signal $y_j$ is not observed. The main difference is that we consider settings where the demonstration data could be contaminated with confounding bias; unobserved confounders exist affecting the action $X_j$, subsequent state $S_{j+1}$, and reward $Y_j$ (had it been observed).

Specifically, as shown in previous research (Ruan et al. 2022), there exist lots of unobserved variables in real-world human driving datasets, e.g., road conditions. As stated in the driving experiment (L300-312), expert demonstrations include vehicle trajectories, “The state $S_t$ contains some critical driving information, e.g., the velocities of the ego vehicle and the leading vehicle and the spatial distance between them. The action $X_{t}$ represents acceleration or deceleration decisions the ego vehicle makes. The unobserved variable $U_{t}$ represents some information accessible to the expert but inaccessible to the imitator, e.g. slippery road conditions [24].” The details of our medical treatment experiments can be found in L313-325.

Due to challenges caused by the confounding bias existing in expert demonstrations, our paper assumes that the imitator has access to sufficiently much demonstration data. We acknowledge that quantifying the uncertainty due to finite samples in imitation learning is an exciting problem, but it is beyond the scope of this paper.

Dear AC,

Thank you for initiating this discussion. I acknowledge the differences in problem settings between this work and prior efforts (e.g., De Haan et al., 2019). However, I maintain that including additional baselines is essential, given that the underlying structural causal model is unknown a priori to the agent. Incorporating more baselines and discussing their relevance will provide a clearer understanding of the performance of the proposed approaches.

Thank you to the authors for the detailed responses. I do recognize the differences in problem settings between this work and previous studies. However, I still believe it is necessary to include additional baselines, especially since the underlying SCM is not known in advance to the agent. Adding more baselines and discussing their relevance will enhance our understanding of how effectively the proposed approaches perform.