• Clarification of the DGP.
  We agree with your points and will replace "independent trajectories" with "trajectories". "Independent" was used to indicate conditional independence between trajectories given latent two-way fixed effects, but this was confusing. Following your suggestion, we use different use different graphical models to clarify DGP under different assumptions in Figure 1 of the PDF file. We will include this figure if our paper is accepted.

• $\eta^\pi$'s dependence on latent variables and proof sketch.
  To validate our theory and save your time, we provide an outline of our proof below. We first clarify $\eta^\pi$'s dependence on latent variables:

  1. We assume two-way unmeasured confounders are fixed, aligning with existing literature on two-way fixed effects regression.
  2. While latent factors $U_i$s and $W_t$s are considered fixed, not all observations $O_{i,t}$s are fixed. Thus, the variability of $R_{i,t}$ comes from the initial observation and all subsequent intermediate observations along the trajectory, generated randomly by the transition function.
  3. The estimation error of $\eta^\pi$ arises from variability in the initial observation, estimated transition dynamics, latent factors, and distributional shift between behavior and target policy, as highlighted in the first two terms of our error bound in Theorem 1.

  Meanwhile, our theory can also accommodate random two-way unmeasured confounders by bounding the deviation between $(NT)^{-1}\sum_{i,t} \mathbb{E}^\pi[R_{i,t}|U_i,\\{W_{t'}\\}\_{t'=1}^t]$ and $T^{-1}\sum_t \mathbb{E}^\pi[R_{1,t}]$. This deviation can be decomposed into two terms: $\Big[\frac{1}{NT}\sum_{i,t} \mathbb{E}^\pi[R_{i,t}|U_i,\\{W_{t'}\\}\_{t'=1}^t]-\frac{1}{T}\sum_t \mathbb{E}^\pi[R_{1,t}|\\{W_{t'}\\}\_{t'=1}^t]\Big]+\Big[\frac{1}{T}\sum_t\mathbb{E}^\pi[R_{1,t}|\\{W_{t'}\\}\_{t'=1}^t]-\frac{1}{T}\sum_t\mathbb{E}^\pi[R_{1,t}]\Big].$ Assuming $U_i$s are i.i.d. and independent from $W_t$s, we apply Hoeffding's inequality to the first term, resulting in an order of $R_{max}\sqrt{2N^{-1}\log (2/\delta)}$ with probability at least $1-\delta$. The second term requires assuming that the time series exhibits certain mixing properties (e.g., $\beta$-mixing), allowing us to treat each $R_{1,t}$ as dependent only on recent $W_{t'}$s. Using Berbee’s coupling lemma (Berbee 1987) and approximating this term with sums of i.i.d. random variables, we apply Hoeffding's inequality again to establish the tail inequality for the second term. We are happy to revise our theory and proof to incorporate these changes if our paper is accpeted.

  Finally, to facilitate understanding of Theorem 1, we provide a proof sketch. We decompose $|\hat{\eta}^\pi - \eta^\pi|$ into two terms, $I_1$ and $I_2$. $I_2$ represents the absolute difference between the mean of the value function and its expectation, given the latent confounders and transition function are true values. The only difference between the two terms is the expectation over the initial state. Based on the boundedness of the value function (Assumption 2) and the conditional independence of each trajectory given all latent confounders, we can apply Hoeffding's inequality. This gives $I_2$ an order of $R_{max}\sqrt{2N^{-1}\log (2/\delta)}$ with probability at least $1-\delta$. $I_1$ represents the absolute difference between the estimator $\hat{\eta}^{\pi}$ (the mean of the estimated value function) and the mean of the true value function. Using the Bellman equation, we transform the difference into a function of differences between the estimated and true values of the transition function, i.e., a function of $\operatorname{TV}(\hat{\mathcal{P}}(\bullet|a_t,o_t,\hat{u}\_i,\hat{w}\_t)-\mathcal{P}(\bullet|a_t,o_t,u_i,w_t))$. Furthermore, the function of $\operatorname{TV}(\hat{\mathcal{P}}(\bullet|a_t,o_t,\hat{u}\_i,\hat{w}\_t)-\mathcal{P}(\bullet|a_t,o_t,u_i,w_t))$ can be decomposed into the sum of the functions of $\operatorname{TV}(\hat{\mathcal{P}}(\bullet|a_t,o_t,\hat{u}\_i,\hat{w}\_t)-\hat{\mathcal{P}}(\bullet|a_t,o_t,u_i,w_t))$ (denoted as $I_3$) and $\operatorname{TV}(\hat{\mathcal{P}}(\bullet|a_t,o_t,u_i,w_t)-\mathcal{P}(\bullet|a_t,o_t,u_i,w_t))$ (denoted as $I_4$). Based on Assumption 4 and the Lipschitz continuity of the neural network, we can determine that the order of $I_3$ is $\varepsilon_{U,W,\delta}$ with probability at least $1-\delta$. Finally, by leveraging Assumptions 1 and 3, we find that the order of $I_4$ is $\varepsilon_{\mathcal{P},\delta}$ with probability at least $1-\delta$. Combining this with all previously derived conclusions, we can then establish the order of $|\hat{\eta}^\pi-\eta^\pi|$ with probability at least $1-3\delta$.

• Equation (1).

  • Is (1) an estimator?: As you commented, (1) is not our final estimator, since the expectation needs to be approximated based on the Monte Carlo method detailed on Page 5, Lines 205 to 210. It serves as an intermediate estimator that is unbiased to the evaluation target (see our justification of the unbiasedness in the next response). We will make this clear to avoid potential confusion.
  • Validity of (1): We claimed Equation (1) is unbiased to $\eta^\pi$ in Line 200. Demonstrating this unbiasedness is straightforward: Recall that $\eta^\pi$ is the average expected reward across $N$ individuals over $T$ time points. Applying the law of total expectation, it can be expressed as $(NT)^{-1}\sum_{i,t}\mathbb{E}^\pi[R_{i,t}|O_{i,1},U_i,\\{W_{t'}\\}\_{t'=1}^t]$, which equals Equation (1).
  • Introducing (1) earlier: Following your suggestion, we will introduce (1) before the model if our paper is accepted.

• Minor issues.
  We apologize for the typos. In line 126, we will revise $Z_{i,t}=U_i\cup W_t$ to $Z_{i,t}=(U_i^\top,W_t^\top)^\top$, to avoid using the undefined notation $\cup$. Additionally, we will replace 'Gaussian Mixture Model' with 'Gaussian Model' in line 187.

• Omitting observations for avoiding biases.
  This is an excellent comment! We have also carefully read [1] that you mentioned and will include it as well as our discussions in our paper, shall it be accepted. This paper discusses whether an additional random variable $Z$ should be included in the regression equation for estimating the average causal effect (ACE) of a treatment $X$ on an outcome $Y$ in causal inference.

  In our current problem, $\eta^{\pi}$ represents the expected cumulative reward under a target policy $\pi$. To draw a parallel with the causal inference problem, our action $A_{i,t}$ corresponds to the treatment $X$, the expected reward $R_{i,t}$ and the next observation $O_{i,t+1}$ correspond to the outcome $Y$, and the initial observation $O_{i,1}$ corresponds to $Z$. In this case, the causal relationship among $O_{i,1}$, $A_{i,t}$, and $(R_{i,t}, O_{i,t+1})$ aligns with "Model 1" in [1], where $A_{i,t} \rightarrow (R_{i,t}, O_{i,t+1})$ and $A_{i,t} \leftarrow O_{i,1} \rightarrow R_{i,t}$. Under these circumstances, controlling for $O_{i,1}$ or including it as an independent variable in the model is beneficial because it blocks the back-door paths. Therefore, when estimating $\eta^{\pi}$, there is no need to omit the initial observation $O_{i,1}$.

• Settings without unmeasured confounders.
  This is another excellent comment. We fully agree with the importance to investigate the tax of the proposed two-way deconfounder in settings without unmeasured confounders. Indeed, our sensitivity analysis reported in Section 5.3 compares different algorithms with different degrees of unmeasured confounding, characterized by a sensitivity parameter $\Gamma$. A large $\Gamma$ indicates a stronger effect of unmeasured confounding. When $\Gamma$ decays to zero, no unmeasured confounders exist. As shown in Figure 5:

  • When $\Gamma=0$ where there are no unmeasured confounders, the proposed two-way deconfounder (TWD) loses its superiority when compared to standard algorithms that ignore unmeasured confounders, such as MB and TWDIDP2. This demonstrates the prices of looking for confounding where none exists.
  • When $\Gamma=0.3$ where the effects unmeasured confounding are weak, TWD achieves similar performance to MB and TWDIDP2.
  • Finally, when $\Gamma>0.3$, the effects unmeasured confounding becomes (moderately) strong. As a result, TWD achieves the lowest LMSE and LMAE.

  Inspired by Reviewer 7kZ2, we have outlined a hypothesis testing procedure to test the degree of unmeasured confounding, in order to determine the best algorithm to use. This hybrid procedure is expected to enhance the robustness of the TWD performance across diverse settings.

• Minor comments.

  • We apologize for the typos and will correct them. "restrict" was meant to be "restrictive".
  • Shall our paper be accepted, we will add a short explanation of the proposed two-way unmeasured confounding in the introduction.
  • In Equation (1), the conditioning set contains variables that are not affected by policies, such as the initial observation. Inclusion of $O_{i,t}$ is problematic because these observations are influenced by the behavior policy, while Equation (1) is intended to compute the expectation under the target policy.

• Clarification of the dependence on the latent confounders.
  You are absolutely correct. The immediate reward indeed depends on the latent confounders, but the target policy we wish to evaluate is irrelevant of these confounders. We plan to adopt the following changes to make this point clearer:

  • We will rephrase this sentence to emphasize that it is the target policy rather than the immediate reward that does not depend on latent confounders.
  • Motivated by the comments from Reviewer P4Ev, we will include visualizations of the data generating process of the offline data and that under the to further illustrate this point; see Figure 2 in the PDF file.

• Clarification on Figure 1(a).
  Thank you for the constructive comment! The edges indicate that both OWUC and TWUC are special cases of UUC. Specifically, the edges from $\\{Z_{i,1}\\}\_{i}$ to $\\{H_{i}\\}\_{i}$ indicate that the one-way unmeasured confounders can be viewed as special cases of unconstrained unmeasured confounders that remain the same over time. Similarly, the edges from $\\{Z_{i,t}\\}\_{i,t}$ to $\\{U_{i}\\}\_{i}$ and $\\{W_{t}\\}\_{t}$ indicate that the two-way unmeasured confounders can be viewed as special cases of unconstrained unmeasured confounders that do not involve either over time or across trajectories.

• The intended application.
  Thank you for your suggestion. Should our paper be accepted, we will further elaborate on the intended application of our method, which is primarily designed for settings involving human decision-makers. In these contexts, decision-makers often rely on critical but unrecorded information when taking actions, leading to confounded datasets. For example, in healthcare or contexts similar to our case study, doctors frequently use visual observations or patient interactions to inform treatment decisions. Such unstructured data can be challenging to quantify and are often omitted [2]. Similarly, in technology companies where behavior policies involve human interventions, the data can also become confounded [3].

• References.
  [1] Cinelli C, Forney A, Pearl J. A crash course in good and bad controls[J]. Sociological Methods & Research, 2022: 00491241221099552.
  [2] McDonald, C. J. (1996). Medical heuristics: the silent adjudicators of clinical practice.
  [3] Shi, C., Zhu, J., Shen, Y., Luo, S., Zhu, H., & Song, R. (2024). Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119(545), 273-284.

• Enhancing clarity of the problem setting in the Introduction.
  Thank you for your valuable suggestion. We will include additional explanations about the problem setting in the introduction section of the final version.

• Clarifying confusing notations.
  We apologize for any confusion caused by the misused notations. We will correct these issues in the final version.

• Can the transition dynamics be represented using a generative model instead of a conditional Gaussian distribution?
  This is a good point. We did assume the transition function $\mathcal{P}$ follows a conditional Gaussian distribution in our implementation. This assumption is common employed in RL, as seen in papers such as [1] and [2]. As you mentioned, it is also possible to parameterize the transition function using a generative model, such as mixture density networks, generative adversarial networks or diffusion models. This alternative modeling approach would not affect our theoretical results, as our theory does not require the conditional Gaussianity assumption. We will add these discussions in the final paper.

• How can we determine the best-fitting setting for the problem without prior knowledge of the data-generating process?
  We greatly appreciate this excellent comment. Our paper covers four confounding assumptions, corresponding to no unmeasured confounding, one-way unmeasured confounding, two-way unmeasured confounding and unconstrained unmeasured confounding. In practice, we may consider the following sequential testing procedure to infer the unknown confounding structure and select the most suitable confounding assumption:

  • Step 1: Initial testing for the Markov property. At the first step, we use state-of-the-art test procedures, such as [3], to determine whether the original data satisfies the Markov property. If the null hypothesis is not rejected, we do not have sufficient evidence to believe there are unmeasured confounders. Thus, we stop the procedure and conclude the data is likely does not contain unmeasured confounders. If the null hypothesis is rejected, it suggests the presence of some unmeasured confounders and we proceed to Step 2.
  • Step 2: Testing for one-way unmeasured confounding. We next assume the data contains one-way confounders, estimate them from the data, include the estimators in the observations, and perform the Markov test again using the transformed data. If the null hypothesis is not rejected, we stop the procedure and conclude that the one-way unmeasured confounding assumption is likely to hold. Otherwise, we proceed to Step 3.
  • Step 3: Testing for two-way unmeasured confounding: Finally, we impose the two-way confounding assumption, estimate these confounders from the data and apply the Markov test to the transformed data with estimated two-way confounders incorporated in the observations. If the null hypothesis is not rejected, we stop the procedure and conclude that the two-way unmeasured confounding assumption is likely to hold. Otherwise, we conclude that the unconstrained unmeasured confounding assumption is likely satisfied.

  We will add the related discussions shall our paper be accepted.

• Clarification on relational neural network.
  The term "relational neural network" has the same meaning as "neural tensor network" [4] mentioned on line 173, known for its ability to capture the intricate interactions between pairs of entity vectors. We apologize for the confusion and will use "neural tensor network" consistently shall our paper be accepted.

• References.
  [1] Yu T, Thomas G, Yu L, et al. Mopo: Model-based offline policy optimization[J]. Advances in Neural Information Processing Systems, 2020, 33: 14129-14142.
  [2] Janner M, Fu J, Zhang M, et al. When to trust your model: Model-based policy optimization[J]. Advances in neural information processing systems, 2019, 32.
  [3] Shi, C., Wan, R., Song, R., Lu, W. and Leng, L., 2020, November. Does the Markov decision process fit the data: Testing for the Markov property in sequential decision making. In International Conference on Machine Learning (pp. 8807-8817). PMLR.
  [4] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013.

I want to thank the authors for their detailed response, which addresses my major concerns. In particular, the authors present a well-rounded discussion on how to determine the best-fitting setting for the problem without knowing the data-generating process. I have also read the authors' responses to other reviews. From my viewpoint, these responses are convincing and provide interesting ideas beyond the paper. In conclusion, I think this paper is a solid work and will keep my positive rating.

• Engagement with the deconfounder literature, limitations and criticisms of [1]
  This is an excellent comment. A detailed discussion of the criticisms of the deconfounder algorithm also helps better motivate our algorithm. We plan to include the following discussion after our cryptic statement you mentioned, to more clearly elaborate on the limitations of the deconfounder:

  To ensure the latent factors can be estimated precisely, [1] imposed the Consistency of Substitute Confounders assumption, requiring to estimate the unmeasured confounders $Z_i$ from the causes $A_i$ with certainty. However, this assumption indeed invalidates the derivations of Theorems 6-8 of [1], resulting in the inconsistency of the algorithm. Specifically, as commented by [2], if the event $A=a$ provides a perfect measurement of $Z$ such that there is some function $\hat{z}(A)$ such that $\hat{z}(a)=Z$, then the overlap condition fails. As a result, the ATE cannot be consistently identified. [3] expressed similar concerns towards the algorithm. Unlike [1], our algorithm does not require such an assumption. Under the RL setting, the proposed two-way unmeasured confounding assumption effectively limits the number of unmeasured confounders to $O(N)+O(T)$, which facilitates their consistent estimation when both the number of trajectories $N$ and the number of decision points per trajectory $T$ grow to infinity, avoiding the need for the unmeasured confounders to be deterministic functions of the actions.

• Clarifications of Propositions 1-3
  Let us first clarify these propositions:

  • Proposition 1 aims to prove that when the true model satisfies the unconstrained unmeasured confounding (UUC) assumption, we cannot obtain consistent estimation even if we work with the correct model assumption (i.e., UUC).
  • Proposition 2 aims to demonstrate that if the true model satisfies two-way unmeasured confounding (TWUC) and we erroneously assume the model satisfies the one-way unmeasured confounding (OWUC) assumption, then our estimators would fail to be consistent. It reveals the limitations of OUC in the presence of time-varying unmeasured confounders, thus highlighting the necessity of the proposed TWUC.
  • Proposition 3 aims to show that when the true model satisfies TWUC and we correctly specify two-way model, we can obtain consistent estimation.

  Meanwhile, shall our paper be accepted, we will explicitly list these underlying true model assumptions and imposed model assumptions highlighted in bold, to make these propositions clearer.

  Finally, there are two additional assumptions in the proof of Proposition 2:

  • The first assumes the oracle $\zeta$ is known;
  • The second assumes all $W_t$s are i.i.d.

  Our intention was to make the main text concise and the presentation easy to follow, so we did not introduce them in the statement of Proposition 2. The first condition is not necessary to impose, as an unknown $\zeta$ would enlarge the estimator's MSE under OWUC. Shall our paper be accepted, we will remove the first condition and explicitly state the second condition in Proposition 2. We hope the aforementioned changes clarify these propositions.

• Clarification on the experimental results of Figure 4.

  • The issue of higher variance: This is another excellent comment. In response, upon checking the code of a related paper [4], we discovered that our procedure omitted a crucial step—data normalization. Given the large state dimension and significant variance among the values within each dimension in the real data, this omission likely contributed to the higher variance observed in TWD's estimated values. During this rebuttal period, we have incorporated the normalization step and re-analyzed the dataset. The updated results, illustrated in Figure 3(a) of the PDF file, demonstrate that the issue with high variance has been addressed effectively, and TWD still performs well. We will modify this figure shall our paper be accepted.
  • The issue of negative values: The negative values arise due to the design of the reward function. In our submitted manuscript, we set the reward to $R_{i,t}=SOFA_{i,t}-SOFA_{i,t+1}$. This choice has been previously used by [5], who also reported similar negative values. During this rebuttual, we have considered two new reward functions $R_{i,t}=-SOFA_{i,t+1}$ and $R_{i,t}=23-SOFA_{i,t+1}$, which were used in [4], and conducted additional experiments under these designs. As shown in Figure 3 of the PDF file, the estimated values of TWD vary from positive to zero to negative, confirming that the appearance of negative values stems from the choice of the reward functions. We will add these discussion shall our paper be accepted.

• Discussion of limitations, typos and Figure 5.
  We will correct the typos and move the limitations of our algorithm to the main text. Figure 5 will be enlarged shall our paper be accepted.

• References.
  [1] Wang Y, Blei D M. The blessings of multiple causes[J]. Journal of the American Statistical Association, 2019, 114(528): 1574-1596.
  [2] D’Amour A. Comment: Reflections on the deconfounder[J]. Journal of the American Statistical Association, 2019, 114(528): 1597-1601.
  [3] Ogburn E L, Shpitser I, Tchetgen E J T. Comment on “blessings of multiple causes”[J]. Journal of the American Statistical Association, 2019, 114(528): 1611-1615.
  [4] Yunzhe Zhou, Zhengling Qi, Chengchun Shi, and Lexin Li. Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach. In International Conference on Artificial Intelligence and Statistics, pages 6704–6721. PMLR, 2023.
  [5] Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602, 2017.