Quantile-Optimal Policy Learning under Unmeasured Confounding

Weakness. The motivation for studying quantiles in policy learning is not clear. The authors use an example of income in a job training program to illustrate the relevance of analyzing reward distribution, which has merit but remains insufficient. In particular, the related works section lacks any mention of quantile policy learning literature. Although work in this area is limited, introducing relevant studies on quantile treatment regimes and quantile treatment effects would help clarify the significance of studying quantiles in policy learning.

Response. For additional motivating examples, please refer to our first response to Reviewer K74W. We also provide a motivating example involving hospital readmission times for negative controls in Appendix Section J.1 (p. 33). We appreciate the reviewer’s suggestion to enhance the literature review. In the revised manuscript, we have expanded the related works section to include the literature on quantile policy learning.

Weakness. The experiments lack persuasiveness.

Response. For experiments under more complex settings and applications on real-world datasets, see Global Response III point 1).

Weakness. Readability is lacking. The theoretical analysis in Section 4 is overly detailed, occupying excessive space. The authors could condense Section 4, such as simplifying the paragraph following Condition 4.4, and instead elaborate on experimental results in Section 5 to improve readability. Additionally, symbols and indices are overly complex, and further simplification is advisable.

Response. We sincerely apologize for any inconvenience caused while reading our paper. For details on how we plan to improve its readability, please refer to Global Response III, point 2).

Weakness. Numerous typos and errors exist in the writing...

Response. We thank the reviewer for highlighting the typos and errors. We have thoroughly proofread the revised manuscript to address these issues.

Weakness. Theoretical results in Section 4 rely on the strong Condition 4.4. While the authors mention in Appendix E that Condition 4.4 is satisfied when $\eta_n = \mathcal{O}(n^{-1/2})$ and both $\mathcal{H}$ and $\Theta$ are linear spaces. However, they do not discuss Condition 4.4 under more complex cases, nor do they consider whether Condition 4.4 can hold without the assumption of $\mathcal{H}$ and $\Theta$ as linear spaces.

Response. For a detailed discussion of Condition 4.4, please refer to Global Response II, point C.

Question. Could the authors provide a more detailed motivation for studying quantiles, like in which situations quantile policy learning offers a clear advantage over standard policy learning (mean)? Additionally, the authors could supplement the experiment part to highlight the advantages of QPL over standard policy learning.

Response. As stated in the previous point, for additional examples, please refer to our first response to Reviewer K74W. We also provide a motivating example involving hospital readmission times for negative controls in Appendix Section J.1 (p. 33). For experiments under more complex settings and applications on real-world datasets, see Global Response point 2).

Weakness. The notations are horrible...

Response. We thank you for your careful review and sincerely apologize for any inconvenience caused while reading our paper. In the revised manuscript, we have replaced $W$ with $V$ and reformulated the problem setup to eliminate the need for notation representing the general observational space.

Weakness. The bounds hide most dependency on the parameters...

Response. We introduce $o_p(\cdot)$ solely to demonstrate the consistency of certain estimators of $\mathbf{h}^*_\alpha$, thereby justifying the local expansion around the neighborhood of $\mathbf{h}^*_\alpha.$ The $o_p(1)$ term appearing in the main theorem is included for simplicity of presentation, and the hidden variable it represents is inconsequential. All dependencies on other parameters or constants are detailed in the proofs provided in the appendix. We also note that the dependency on the constants specified in Assumptions 4.9 and 4.10 is explicitly reflected in the convergence rate of regret presented in Theorem 4.11.

Weakness. The assumptions are stated messily and lack sufficient explanation...

Response. As stated in Global Response I, we have added Section N in the Appendix to provide a detailed discussion of the assumptions and conditions presented in Section 4. Specifically:

• For Assumption 4.1, please refer to Global Response I, point A.
• We acknowledge that Condition 4.4 may initially seem challenging to interpret, as it is specifically formulated to solve a quadratic equation of the test function $\theta$ (see Appendix Section G.1) in the proof for deriving a statistical fast rate. Similar formulations can be found in classical textbooks and papers that focus on achieving statistical fast rates, such as Theorem 14.20 in Wainwright (2019) and Lemma 12 in Foster and Syrgkanis (2023). Unfortunately, these sources also lack an intuitive explanation. For additional discussion on Condition 4.4, please refer to Global Response II, point C.
• Regarding Assumption 4.10, we clarify its connection to the standard concentrability assumption in Global Response II, point E.
• For Assumption 4.9, please see Global Response II, point D.
• For Assumption 4.9, please see Global Response II, point D.
• In short, we need $\epsilon_{\lambda_n}$ for the consistency result in Theorem 4.12. to hold. Intuitively, when $\epsilon_{\lambda_n}$ is small, the rate of convergence of $\hat{h}^{\pi^*}_{R}$ is slow. Consequently, the local expansion error $c_0$ could be large, which in terms leads to a slower regret rate. For the technical details, please refer to the proof of the Theorem C.3 in Appendix G.3.

Contribution. It is hard...

Response. For details on how we have improved readability, please refer to Global Response III, point 2. We assume the reviewer is referring to Lu et al. (2022). The primary distinction between our paper and the two listed studies lies in what we consider the most substantial contribution of our work: we optimize the average structural quantile of the reward distribution, whereas they focus on the average reward. Unlike the average reward, which is linear in the structural function, the average structural quantile is a nonlinear function of the unknown structural quantile function. Many of the assumptions and technical components in our paper are explicitly designed to address this challenge, including Assumption 4.1, 4.7, 4.9 and Theorem 4.8. These assumptions are standard in quantile IV literature.

We have explicitly highlighted the challenges and distinctions of our work in the introduction section, motivated why policymakers might prefer quantile objectives in Section 2, and emphasized the associated technical problems throughout Section 4. Additionally, we have clarified how our work compares with the two stydues in the "Quantile Objective and Regret" paragraph of Section 2:

Existing works focus on learning the policy that maximizes the average reward, $\mathbb{E}_{\hat{\pi}}[Y]$ Lu et al. (2022), Guo et al. (2022). The main goal of this paper is to maximize the average structural quantile of the reward.

References.

• Wainwright (2019): Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Volume 48. Cambridge University Press, 2019.
• Foster and Syrgkanis (2023): Dylan J. Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879–908, 2023.
• Guo et al. (2022): Hongyi Guo, Qi Cai, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably efficient offline reinforcement learning for partially observable Markov decision processes. In International Conference on Machine Learning, pp. 8016–8038. PMLR, 2022.
• Lu et al. (2022): Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang.

Weakness. The authors motivated the unmeasured confounders with a healthcare example, and quantile optimization with job training programs. However, these are distinct cases, and I wonder whether there is one unified motivating example that requires both.

Response. We actually motivate both the unmeasured confounding and quantile optimization using the job training programs. In Section 2, line 130, we state that: "$U_i$ could represent unmeasured factors such as the worker's motivation, which influence both their participation decision $A_i$ and their utility outcomes $Y_i.$"

Another example is in finance. Consider the design of a finance portfolio optimization model, aimed at minimizing portfolio risk. In this setting, $Y_i$ represents the negative loss generated by the $i$-th portfolio. $X_i$ refers to the pre-investment characteristics of the portfolio, such as market conditions and investor risk tolerance. $A_i$ denotes the portfolio strategy implemented, such as the weighting of different asset classes. $Z_i$ serves as an instrumental variable that influences the portfolio strategy but does not directly affect the portfolio's returns, such as regulatory requirements or external economic indicators. In practice, financial institutions often seek to minimize a specific quantile of the loss distribution, a measure known as Value at Risk (VaR) Duffie and Pan (1997) in portfolio theory. The reason is that the expected loss can be heavily skewed by highly volatile portfolios, making the mean loss a less reliable measure of risk. Examples of unmeasured confounders include investor sentiment, which may simultaneously influence asset allocation decisions (treatment) and market performance (outcome), or unobserved market shocks, such as geopolitical events or unexpected economic announcements.

We also include a motivating example of hospital readmission times for the negative controls in Appendix Section J.1 (p. 33).

Weakness. Section 4, while highly technical... Here are a few main points that cause confusion: How strong is Assumption 4.1? What would this imply if you bring it to the tabular setting, for instance?

Response. Please refer to Global Response I, point A, for a detailed discussion of Assumption 4.1. Specifically, for the tabular case where the context $X$ and action $A$ are discrete, Theorem 2 of Chernozhukov and Hansen (2005) provides sufficient conditions for this global identification result to hold. We also remark that our paper, like almost all studies in nonparametric quantile IV literature, assumes that the reward $Y$ is continuous. Please see Global Response I, point B, for a discussion of continuous rewards.

Weakness. Theorem 4.5 -- again, if you could explain this in the tabular setting, that might help a lot.

Response. Theorem 4.5 holds as long as Assumptions 4.1–4.3 hold, regardless of whether other variables are discrete or not.

Weakness. Presentation became overly complicated... I think not many readers, even the experts in RL, can go through the subsections here.

Response. We sincerely apologize for any inconvenience caused while reading our paper. For details on how we plan to improve its readability, please refer to Global Response III, point 2.

Weakness. I think the experimental section could have been reinforced in the main text.

Response. Thank you for your valuable suggestion to strengthen the experimental section. We will incorporate this improvement in the revised manuscript. Additionally, please refer to Global Response III, point 1, for details on the new semi-synthetic data study we plan to analyze.

References.

• Chernozhukov and Hansen (2005): Victor Chernozhukov and Christian Hansen. An IV model of quantile treatment effects. Econometrica, 73(1):245–261, 2005.
• Duffie and Pan (1997): Darrell Duffie and Jun Pan. An overview of value at risk. Journal of Derivatives, 4(3):7–49, 1997.

Weakness. Since the paper deals with causal inference with unmeasured confounding, it should include experiments on real-world datasets. Handling real-world datasets introduces many challenges (e.g., discrete outcomes, missing covariates, etc.). The current experimental setup considers a simulation-based setting that uses linear functionals, one-dimensional context, and is too simple to demonstrate the effectiveness of the proposed method.

Response. For a discussion on discrete outcomes, please refer to Global Response I point B. For experiments on real-world datasets, see Global Response III point 1).

Weakness. The authors make several strong assumptions in order to prove theoretical results. First, it is assumed that the parameter $\eta_n = O(n^{-1/2})$, but this is only true for simple classes of functions (as shown in Appendix E) and is not true in general. Second, Assumption 4.7 assumes that the conditional density is continuous, which eliminates all causal inference problems with discrete outcomes. Third, I believe Assumption 4.9 is quite strong, and there is no justification as to why it should be true.

Response. For the first concern, please refer to Global Response II point C. For Assumption 4.7, see Global Response I point B. For Assumption 4.9, see Global Response II point D.

Weakness. Overall, I found the paper hard to follow. Many steps require further explanation, and the assumptions require further justification.

Response. Please refer to Global Response III point 2).

Question. Can you please justify why Assumption 4.9 should be true?

Response. Please refer to Global Response point II D.

Question. Solving offline reinforcement learning requires some type of coverage condition on the data (e.g., single-policy concentrability). In the paper, I haven't seen any explicit mention of such coverage conditions. Can you please clarify what you assume regarding the overlap between the optimal policy and the data-generating policy?

Response. Please refer to Global Response point II E.