Off-Policy Evaluation under Nonignorable Missing Data

Thanks for your thoughtful questions and the time you spent reviewing our paper. We really appreciate your insights and are happy to discuss any further ideas or questions you may have.

Regarding Assumptions (a), (b), and (e):

Below is a more detailed explanation of what each assumption means and how it is used in the proof. We hope this will help improve the clarity of the assumptions, and we will incorporate these explanations in the final version of our paper.

• Assumption (a): We assume smoothness of the transition kernel $\mathcal{P}$. Specifically, when $0 < p \leq 1$, $\lfloor p \rfloor = 0$, and the condition becomes equivalent to assuming $h$ satisfies $\sup_{x, y} \frac{|h(x) - h(y)|}{\\|x - y\\|_2^p} \leq c$, which is a form of Hölder continuity. Under this assumption, it can be shown that there exists a constant $c' > 0$ such that $Q(\pi; \cdot, a) \in \Lambda(p, c')$ for any policy $\pi$ and $a \in \mathcal{A}$. This ensures that the $Q$-function has bounded derivatives up to order $\lfloor p \rfloor$, which is critical when deriving inference for the value function.
• Assumption (b): This assumption is more of a claim or explanation rather than a strict assumption. Here we consider two types of basis functions, which are commonly used sieve basis functions. These are standard choices for such problems and serve to simplify the analysis while ensuring general applicability.
• Assumption (e): Here, $L$ controls the smoothness of the basis function, which in turn determines how closely the linear sieve basis function can approximate the true function. This assumption is used to ensure the $Q$ function is well approximated. In the proof of asymptotic normality, we rely on this condition on $L$ to establish that $\sup_{s \in \mathcal{S}, a \in \mathcal{A}} |Q(\pi; s, a) - \Phi_L^\top(x) \beta^*_{\pi, a}| = O(L^{-p/d})$, which guarantees the consistency and asymptotic behavior of $\hat{\beta}$ and the value estimates.

Regarding "Other Comments Or Suggestions”:

• The reviewer mentioned that Line 110 is only valid for a discrete state space. We would like to clarify that our system does not impose any constraints on whether the state space is discrete or continuous. In fact, we explicitly allow a multi-dimensional continuous state space, i.e., $S\in \mathbb{R}^d$, as stated in Line 115-116. We recognize that the definition in Line 110 might be somewhat misleading, as $p$ actually refers to the transition kernel in the MDP and is not a probability mass function. We believe it would be clearer to revise the current statement to $p:\mathcal{S} \times\mathcal{A}\rightarrow\mathcal{S}$, and we hope this revision will help clarify the matter.
• We sincerely appreciate the reviewer's feedback, and will include detailed definitions of B-splines and wavelets in the final version of our paper for completeness.

Regarding Assuption A.1 (d) (i):

We appreciate the reviewer’s question regarding the ease of satisfying this assumption. A special case that may help clarify the strength of this assumption is when $\pi$ is deterministic, $b$ is the $\epsilon$-greedy policy with respect to $\pi$ satisfying $\epsilon \leq 1 - \gamma^2$, and $\mu=\nu_0$. In this case, Assumption A.1 (d)(i) can be shown to be naturally satisfied (see Sec C.1 of the supplementary material of Shi et al. (2021b) for proof details).

We recognize that this is a somewhat non-intuitive assumption, and the verification may require some calculations. We also appreciate the reviewer’s suggestion to explore whether and how this assumption could be simplified by incorporating the specific form of $\Phi$ and the moving behavior distribution $b(a|s)$. Checking whether the LHS of this assumption is positive-definite (or finding an easier-to-understand sufficient condition for it) essentially requires us to quantify the positive definiteness of the matrix part, i.e., $\xi\xi^\top -\gamma^2 u_\pi u_\pi^\top$. The multiplication by $b(a|s)$ and $\mu(s)$ is more of a weighting method to determine whether the matrix, after taking the expectation, remains positive-definite, which is not the main focus. Thus, deriving a boundedness assumption for $\mu$ from below seems implausible. We would greatly appreciate any further thoughts or ideas you may have on this aspect and look forward to your suggestions.

Once again, we sincerely appreciate your time reviewing our paper and welcome any further questions or discussions.

Response to "Summary-Claims And Evidence”:

The reviewer raised a concern that the comparison between V-IPW and V-CC relies on the assumption that V-CC is negatively biased. Actually, Our real-world experiment consists of two parts: the first (Table 2) is based on the original sepsis dataset, while the second (Table 3) is derived from a quasi-real dataset where we controlled only the dropout hazard function, ensuring it follows a known form. We would like to emphasize the following points:

• In the offline dataset, where the true dropout pattern is unknown, there is no ground truth for the true value function under no dropout. This means we cannot directly assess the bias in V-CC or definitively validate the extent to which V-IPW corrects it. Our inference in Table 2 is based on common knowledge and the intuition that V-CC may underestimate the value function due to early discharge, though we acknowledge that this inference lacks definitive empirical support. We appreciate the reviewer’s attention to this issue, which motivated our quasi-real data analysis using MIMIC-III, as presented in Table 3.
• In Table 3, we explicitly controlled the dropout hazard model using the function $\lambda(\cdot)$, specified above Sec 7, which follows a functional form inspired by prior MIMIC-III studies (Kramer & Zimmerman, 2010; McWilliams et al., 2019). This setup allows us to obtain an unbiased estimate of the value function under no dropout (reported in the first row), serving as a ground truth for comparison. The results in Table 3 consistently show that V-IPW outperforms V-CC in correcting the underestimation bias caused by missing not at random (MNAR) dropout.

Response to "Summary-Experimental Designs Or Analyses”:

1. Regarding the "Propensity Accuracy Analysis", we have provided additional support to validate the accuracy of the dropout model estimation in https://anonymous.4open.science/r/OPE_MNAR_ICML_rebuttal-832D/, where the first plot shows small MSEs close to zero, and the second plot compares the estimated parameters to their true values. Additionally, as all V-IPW experiments under MNAR require prior propensity score estimation, Figure 2 and the last two rows of Table 1 (MNAR (P) and MNAR (SP)) in the main paper confirm the accuracy of these estimations through bias, standard error, and empirical coverage probability. Without stable estimation results, the statistical properties of the value function would not hold. We hope the added accuracy analysis regarding the propensity score, along with the existing tables and figures in the main paper, will help address your concerns regarding the dropout function estimation performance.

2. Regarding the "Low-Dimensional Settings" and "Limited Environment Diversity": we sincerely appreciate the reviewer's suggestion on trying more diverse simulation settings and environments to possibly test the wider applicability of our approach. Due to time limit, we might not able to provide another group of setting and detailed analysis during this rebuttal time, but we will incorporate more settings to the final version of our paper.

3. Regarding the "Analysis of the important samples requirements": We appreciate the reviewer's question regarding the sample requirements in order to guarantee a stable estimate of the value function. Based on the asymptotic results in Theorem 4.7, the sample size requirement actually can be derived with a few steps. Here due to space limit we provide a brief analysis about the requirement, and we will add it to the final version of our paper to help with it. As $\hat{V}^{\pi}\sim \mathcal{N}(V^{\pi},\hat{\sigma}^2/(nT))$ (for simplicity we omit $\mathbb{G}$ in parenthesis and the subcript 'IPW'), applying Gaussian tail bounds we have $\mathbb{P}(|\hat{V}^{\pi}-V^{\pi}|\geq \epsilon)\leq 2\exp\\{-nT\epsilon^2/(2\sigma^2)\\}$. Therefore, to achieve an error bound $\epsilon$ with probability at least $1-\delta$, the required sample size would be $n\geq \log(2/\delta)\cdot 2\sigma^2_{\pi}/(\epsilon^2T)$. For example, by setting $\delta = \epsilon = 0.01$, $T = 100$, and $\sigma^2_{\pi} = 0.5$ (a value comparable to the simulation setting), we find that $n \geq 530$ trajectories would suffice.

Response to Weakness 1:

Although we did not explicitly state the convergence rate of V-IPW, it is provided by the asymptotic normality result in Theorem 4.7, which shows that $\hat{V}^{\pi}$ converges to the true value at a rate of $O(n^{-1/2}T^{-1/2})$. This ensures fast bi-directional convergence in both $n$ and $T$, and this rate matches the parametric convergence rate, leaving no room to further improve the order of convergence.

Response to Weakness 2:

Thank you for your suggestion. We agree and will reintegrate the dropout propensity section into the main text to enhance readability and clarity.

Once again, we appreciate your time and effort in reviewing our paper and look forward to your questions for further discussion.

Thanks for your thoughtful questions and the time you spent reviewing our paper. We really appreciate your insights and are happy to discuss any further ideas or questions you may have.

Clarification on Terminology:

We sincerely appreciate the reviewers' feedback on clarifying terminology across different fields, particularly regarding the comments on Relation to Broader Scientific Literature. We will provide thorough definitions of concepts such as monotone missingness and censoring in the introduction or at their first occurrence to enhance readability and coherence throughout the paper.

Regarding Essential References Not Discussed:

Thank you for highlighting the related work by Ji et al. (2021) on early discharge in the MIMIC-III dataset. After a careful review, we notice that their study primarily applies RL methods to identify potential issues with the existing vasopressor treatment policy. Notably, part of their findings aligns well with our problem motivation, which demonstrates that early discharge can introduce significant modeling bias, thereby impacting evaluation. We will incorporate this reference into the introduction, as it further substantiates the motivation behind our work.

Clarification on Types of Missingness in the Introduction:

We acknowledge that Line 14 may be somewhat vague in discussing different types of missingness, which could be unclear to readers. Here, the missingness types (as introduced in the subsequent paragraph) include both MAR and MNAR, specifically concerning the observation status of the reward being evaluated. Within the RL framework, this missingness affects both the reward and next-state, denoted as $(R_{t+1}, S_{t+1})$. Given the presence of non-ignorable missingness, the absence of $(R_{t+1}, S_{t+1})$ naturally leads to cascading missingness in subsequent state-action-reward sequences. While a detailed explanation may not be feasible in the introduction, we will clarify in Line 14 that the missingness primarily pertains to the reward in general settings. We hope this could ensure greater clarity for readers.

Finally, We sincerely appreciate your time and effort in reviewing our paper and look forward to your questions and further discussion.