Off-policy Evaluation with Deeply-abstracted States

We greatly appreciate your valuable assessment of our work, many of which will lead to a more readable and self-contained version of our paper. Below, we address certain specific concerns.

--Computational overhead. Thank you for pointing this out. We completely agree that the iterative procedure will incur some computational cost. However, we have found that performing the iteration just four times is sufficient to achieve a noticeable increase in accuracy. As a result, the computational burden is relatively small compared to the improvement in accuracy. Additionally, we believe there are methods to optimize the computation in the future.

--Integration into existing OPE frameworks. Thank you for your comment. Indeed, the proposed method can be easily integrated into existing OPE frameworks, as detailed in Appendix B.

We greatly appreciate your valuable assessment of our work. Below, we address certain specific concerns.

--Backward model irrelevance. Since there are different OPE methods, backward model irrelevance is employed to ensure the ($\rho^{\pi}$) $w^{\pi}$ irrelevance while the forward model irrelevance is used to ensure the $Q^{\pi}$ irrelevance, which is the key for the (SIS) MIS method and Q-function-based method, respectively.

What we propose is an iterative method, as highlighted in the title. Our approach alternates between forward and backward abstraction on the state space obtained from the previous iteration. Each iteration ensures that the cardinality of the state space does not increase, effectively maintaining or reducing complexity. Consequently, this iterative procedure progressively reduces the state cardinality, ultimately yielding a deeply abstracted state. We refer to this approach as deep state abstraction (DSA).

Simulation results demonstrate that this iterative method achieves the smallest absolute bias and mean squared error (MSE), validating the effectiveness of our approach.

--Typos. Thank you for pointing this out. We will correct all the typos in the revised paper and carefully proofread the paper in future versions to ensure accuracy.

We greatly appreciate your valuable assessment of our work, many of which will lead to a more readable and self-contained version of our paper. Below, we address certain specific concerns.

--Comparison to non-state-abstraction methods. In fact, we have compared to one of the non-state-abstraction methods denoted as FQE in our simulation parts. Furthermore, we have claimed clearly that " Both figures show that the baseline FQE applied to the ground state space performs the worst among all cases. This comparison reveals the usefulness of state abstractions for OPE".

--Backwards model irrelevance. The concept of backward model irrelevance refers to our observation that the state is tied to the behavior policy.

In the context of our first toy example, confounder selection serves as a special case of our problem under certain conditions:(i) The state transition is independent, effectively transforming the MDP into a contextual bandit; (ii) The action space is binary, with the target policy consistently assigning either action 0 or action 1, aimed at assessing the average treatment effect; (iii) State abstractions are confined to variable selections. It is worth noting that this formulation mirrors the iterative confounder selection procedure discussed in Guo et al. (2022). While our proposed iterative procedure shares similar spirits with the aforementioned algorithms, it addresses a more complex problem involving state transitions. Additionally, our focus is on abstraction that facilitates the engineering of new feature vectors, rather than merely selecting a subset of existing ones. Consequently, such an iterative procedure progressively reducing state cardinality, which ultimately yields a deeply-abstracted state. We thus refer to our approach as deep state abstraction (DSA). The simulation results indicate that such iterative method can get the smallest absolute bias and MSE.

--Reference.

• F Richard Guo, Anton Rask Lundborg, and Qingyuan Zhao. Confounder selection: Objectives and approaches. arXiv preprint arXiv:2208.13871, 2022.

We greatly appreciate your valuable assessment of our work. Below, we address certain specific concerns.

--Fisher consistency. Fisher consistency is described as “Roughly this requires that if the whole ‘population’ of random variables is observed, then the method of estimation should give exactly the right answer", see Cox and Hinkley (1974, p. 287) , Tasche (2017). Thus, $\mathbb{E} [f_1(Q^{\pi})]=\mathbb{E} [f_2(\rho^{\pi})]=\mathbb{E} [f_3(w^{\pi})]=\mathbb{E}[f_4(Q^{\pi},w^{\pi})]=J(\pi)$ shows that the OPE methods (without abstraction) themselves are Fisher consistent. For our methods,

• $J(\pi)=\mathbb{E} [f_1(Q^{\pi})]=\mathbb{E} [f_1(Q^{\pi}_{\phi})]$ implies the Q-function-based method is Fisher consistent;
• $J(\pi)=\mathbb{E} [f_3(w^{\pi})]=\mathbb{E} [f_3(w^{\pi}_{\phi})]$ means MIS is Fisher consistent;
• $J(\pi)=\mathbb{E} [f_2(\rho^{\pi})]=\mathbb{E} [f_2(\rho^{\pi}_{\phi})]$ indicates SIS is Fisher consistent;
• $J(\pi)=\mathbb{E}[f_4(Q^{\pi},w^{\pi})]=\mathbb{E} [f_4(Q^{\pi}_{\phi},w^{\pi} _{\phi})]$ describes Fisher consistency of the double robust method.

This logic can also be found in Park and Weisberg (1998), equation (6) in page 232.

--Two-step procedure. We find this comment regarding the 'two-step procedure' confusing. The two-step approach refers to our earlier version submitted to NeurIPS. For this conference, we submitted a modified version, yet the term appears twice in the reviewer’s comments. A similar concern was raised by one of the reviewers in our previous submission. This reviewer, who provided the same score, had expressed a willingness to revise their score based on updates to our proof.

In response, we devoted significant effort during the rebuttal period to provide a clear and accessible proof within the word limit. Unfortunately, during the discussion period, the reviewer became reluctant to engage, only responding in the final hours. From their last response, we understood that their primary concerns were addressed. However, they did not specify which aspects of our responses were inadequate but maintained their original score, stating that "a substantial revision is needed."

Additionally, we would like to clarify that we did not perform data splitting to train abstractions and OPE estimators. Both procedures were conducted on the same dataset, ensuring no reduction in sample size that could compromise the performance of the final estimator.

--Reference:

• Cox, D. R., & Hinkley, D. V. (1974). Theoretical statistics. Chapman and Hall, London.
• Park, C., & Weisberg, S. (1998). Fisher consistency of GEE models under link misspecification. Computational statistics & data analysis, 27(2), 229-235.
• Tasche, D. (2017). Fisher consistency for prior probability shift. Journal of Machine Learning Research, 18(95), 1-32.