Active Causal Learning for Conditional Average Treatment Effect Estimation

1. For the problem formulation in Eq. (3), directly minimizing $\hat L_f$ might not be practical enough. For many applications, a hard constraint, i.e., $\hat L_f < \epsilon$ could be better.

Even though the proposed method might be simple, I still appreciate the proposed new setting and problem formulation.

Cons: This article comes under the umbrella of active learning, and I am cautious that more causal active learning (at the theoretical level as well as at the applied level) needs to be extensively and deeply contrasted in the work related to this article; secondly, I think that the methodology of this article is a bit oversimplified and that the theoretical level of the analyses is a bit thin

• There is no identification for CATE with masked covariates. The SUVTA condition is only assumed for full covariates $X_i$. Please discuss how the SUTVA and other identification assumptions may need to be modified or extended when dealing with partially observed covariates. It is suggested to provide theoretical justification or empirical evidence for the validity of CATE estimation with masked covariates.

• The estimation model $f_w$ is trained only with fully observed $X$, i.e., the input is $f_w(\mathcal{X}(X,\mathbf{1}))$. For observation $\mathcal{X}(X,M)$ with other mask $M$, the performance of $f_w(\mathcal{X}(X,M))$ can be very poor. Have you considered training $f_w$ with various masked inputs to improve its performance on partially observed data? How might this affect the overall performance of the proposed method?

• It seems that the policy in (3) is a stochastic policy as authors propose to use PPO to learn this policy, there is a missing expectation over action $a_t$ according to $\pi$ in the objective function. Please clarify whether the policy is stochastic and, if so, explain how the expectation over actions is incorporated into the objective function. Also, why is there an indicator function in the constraints as $M$ is a binary mask vector?

• In the POMDP setting, the transition should be $\mathbb{P}: \mathcal{S} \times \mathcal{A}\rightarrow \mathcal{S}$, and reward does not depend on $\mathcal{O}$. The policy typically leverages the past history, not only the current observation. I suggest to discuss how incorporating past history into the policy might affect the performance of the proposed method.

• Only when the discount factor $\gamma=1$, the RL value function matches the objective in (3). Why choosing $\gamma < 1$ in the experiments? How this choice affects the relationship between the RL value function and the original objective?

The idea of using RL to define a policy for sequential feature acquisition is not new. There is a great deal of work on this problem in the supervised setting. A few examples that come up from a quick search:

Li and Oliva. Active feature acquisition with generative surrogate models. ICML 2021.

Ghosh and Lan. DiFA: Differentiable Feature Acquisition. AAAI 2023.

Zheng Yu, Yikuan Li, Joseph Chahn Kim, Kaixuan Huang, Yuan Luo, Mengdi Wang Deep Reinforcement Learning for Cost-Effective Medical Diagnosis. ICLR 2023.

The paper doesn't discuss any of this related work or how the proposed methods/problem formulation is related. While this paper focuses on causal estimation, the way that it addresses the problem effectively reduces it to supervised learning: the predictions of an existing causal inference method using the historical (complete) data are used to provide surrogate outcomes, after which the proposed reward function treats it exactly as in the supervised case. I believe that the paper needs to either illustrate a fundamental difference between supervised learning and CATE estimation in this setting, or compare to existing methods in this literature and demonstrate improved performance.