Distributionally Robust Policy Learning under Concept Drifts

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments and helpful suggestions. Please refer to our latest manuscript for the revised version, in which we have made the following edits according to the reviewers' suggestions: (i) we revised the introduction section to explain our contribution of providing distributional robust inference and policy learning under concept shift more clearly; (ii) we proved a lower bound guarantee to show the optimality of our policy learning regret bounds; (iii) we added a new experiment in a more realistic setting that mimics real-world concept shifts.

The following details our responses to the reviewer’s comments and questions.

Weakness #1, Reply: Thank you for your comment. We apologize for the ambiguity in our presentation. We would like to note that we do not attempt to claim Lemma 2.3 as our major contribution, and we have modified our narratives accordingly with sufficient credit given to the classical results in our revised manuscript.

To clarify, the main contribution of our work is to provide distributional robust inference and policy learning schemes under concept shift. For this problem, previous work [2] can only be applied to discrete distributions on finite support with lower bounded probability mass function. Our work provides an algorithm allowing for general types of covariates, substantially generalizing the applicability of the algorithm. In our revision, we have also included a matching regret lower bound that characterizes the fundamental difficulty of the learning problem we consider.

We also note that technical development is not trivial, as the duality form in Lemma 2.3 poses challenge due to the high dimensional nuisance parameters space. The nuisance parameter $\alpha$ in conventional distributional robust offline policy learning literature with joint distribution shift, is a scalar; while in our work, both nuisance parameters $\alpha_\pi(x),\eta_\pi(x)$ are functions of the covariate $x$ and policy $\pi$. For policy estimation, the major challenge is the slow estimation rate of $\alpha_\pi,\eta_\pi$, which is overcome by incorporating a de-biasing technique introduced in Section 3.1. For policy learning, the critical challenge is that it is difficult to estimate $\alpha_\pi,\eta_\pi$ as a function of $\pi$ using the offline data. We overcome this by decoupling the dependence of $\alpha_\pi,\eta_\pi$ on policy to the dependence on actions, which helps us to construct an learning algorithm based on policy trees. We discuss this technique in details in Section 4. Proof-wise, we developed uniform concentration techniques for an estimator with a large nuisance parameter space, utilizing chaining (Appendix B.3).

We have revised our introduction section to better explain the research question and our contribution. Please refer to our latest version.

Weakness #2, Reply: We would like to first note that $X$ follows a uniform distribution as stated in Section 5. We have added a new experiment to include a more realistic simulation setting that mimics real world concept shifts. Please refer to Table 3 in the latest version.

Weakness #3, Reply: Thank you for your comment, and this is exactly the motivation of our paper --- that knowing the source of the distribution shift effectively shrinks the uncertainty set, thereby yielding less conservative results.

The way we see our work is that, instead of saying our algorithm improves upon the existing ones based on joint modeling, we would rather say that it makes it possible to make use of the one extra bit of information (that the source of shift is in the reward distribution) to obtain less conservative results both theoretically and empirically. The result of Table 1 demonstrates that our learning algorithm succeed in doing so. Meanwhile, our attempt to make this learning paradigm feasible is technically nontrivial (as discussed in our response to Weakness 1), and has a wide range of applications, including new product launches in a familiar market and campaign strategy in face of changes in the value system of the population.

The authors would like to kindly remind the reviewer to read the responses before the discussion deadline. We have revised the manuscript according to the helpful suggestions. Any further comments would be deeply appreciated!

Thank you so much for your response! We would like to ask if you have remaining concerns that have not yet been addressed?

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments and helpful suggestions. Please refer to our latest manuscript for the revised version, in which we have made the following edits according to the reviewers' suggestions: (i) we revised the introduction section to explain our contribution of providing distributional robust inference and policy learning under concept shift more clearly; (ii) we proved a lower bound guarantee to show the optimality of our policy learning regret bounds; (iii) we added a new experiment in a more realistic setting that mimics real-world concept shifts.

The following details our responses to the reviewer’s comments and questions.

#1, Reply: Thank you for the comments!

Contributions. We apologize for the ambiguity in stating our contribution. In fact, we do not attempt to claim Lemma 2.3 as the contribution of our work --- we have modified our narratives accordingly in the revised manuscript, giving credit to the standard results.

To clarify, the main contribution of our work is to provide distributional robust policy evaluation and policy learning schemes under conditional reward shifts --- that is, the inference problems. The type of distribution shift model we consider leads to an objective that contains nuisance parameters that are high-dimensional (they are functions of the covariates); substantial works have been devoted to dealing with these nuisance parameters to achieve an efficient policy evaluation/learning algorithm. We have also included a matching regret lower bound in the revision, proving that our proposed learning algorithm is minimax-optimal in terms of the sample size and the policy class complexity. In the revised manuscript, we have included a table detailing the comparison with our work and preious ones.

Solving for $\eta^*$. Indeed, for the DRO problem, $\eta^*$ can be solved analytically and previous work [1] indeed adopts Theorem 1 in [2] to establish strong duality; such a formulation, however, leads to an objective function

$

\max_{\mathbf{\alpha}} -\mathbb{E}\bigg[\mathbf{\alpha}(X) \log \Big(\mathbb{E}\Big[\exp\Big(-\frac{Y(\pi(X))}{\mathbf{\alpha}(X)}\Big) \mid X\Big]\Big) + \mathbf{\alpha}(X)\delta\bigg].

$

In turns out that it is hard to work with the objective function above to derive an efficient estimator (and correspondingly the learning algorithm) because of the conditional expectation in the logrithmic function. We then choose to work with the objective function in our work. We have revised our introduction section to better explain the research question and our contribution. Please refer to the latest version.

#2, Reply: We apologize for the ambiguity in our text. We have edited the manuscript to make the definition clearer. We define $g_\pi(x):=\mathbb{E}[G_\pi(X,Y(\pi(X)))|X=x]$. The behavior policy $\pi_0$ is the action-assignment during data collection, and the propensity score function $\pi_0(a\mid x)$ is the probability of $\pi_0$ assigning action $a$ given the covariate $x$ (Line 166 in the revised version).

#3, Reply: Thank you the question. As we have mentioned in Line 303 of our submission, a previous paper [3] has provided detailed examples where $\theta_\pi^*(x)$ is continuous in $x$ (in their Appendix B.2). For instance, when the conditional distribution of $Y(\pi(X)) \mid X = x$ is continous in $x$, then the resulting $\mathbf{\theta}_\pi(x)$ is continuous in $x$ (e.g., $Y(\pi(X)) \mid X = x\sim \mathcal{N}(\mu(x),\sigma^2)$ for some smooth function $\mu(x)$).

#4, Reply: Thank you for the comment. We would like to first note that we do not assume knowing the behavior policy $\pi_0$ and use the data to learn it (this is typically not the case in offline RL unless there are additional assumptions on the reward distribution) --- without the overlap condition, learning the behavior policy $\pi_0$ becomes a challenging problem itself.

Meanwhile, we note that Assumption 2.1 is standard in offline policy learning literature (see e.g. [1,4-7]), in which the overlap assumption ensures sufficient exploration when collecting the training dataset. The regret bound in Theorem 4.2 shows the dependence of the performance on the concentrability coefficient $\varepsilon$: if given the covariate $x$, some action $a$ has little probability of being explored by the behavior policy $\pi_0$, then it is hard to learn the outcome of the action covariate pair $(x,a)$ and thus incurring a larger regret.

We agree that partial coverage is an important and challenging problem in offline policy learning. In the setting when we have access to $\pi_0$ (potentially without overlap), we conjecture that we could leverage the pessimistic framework in a fashion similar to [8] to achieve a similar bound in the absence of overlap conditions. We leave the investigation to future work since it is outside the scope of the current one.

The authors would like to kindly remind the reviewer to read the responses before the discussion deadline. We have revised the manuscript according to the helpful suggestions. Any further comments would be deeply appreciated!

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments and helpful suggestions. Please refer to our latest manuscript for the revised version, in which we have made the following edits according to the reviewers' suggestions: (i) we revised the introduction section to explain our contribution of providing distributional robust inference and policy learning under concept shift more clearly; (ii) we proved a lower bound guarantee to show the optimality of our policy learning regret bounds; (iii) we added a new experiment in a more realistic setting that mimics real-world concept shifts.

The following details our responses to the reviewer’s comments and questions.

Weakness #1: "While the paper is motivated by the idea that existing methods may be suboptimal under concept shift alone, it does not clearly demonstrate how its proposed rates improve upon existing ones. The improvement noted in Remark 4.3 seems more a result of advanced theorem-proving techniques (e.g., chaining) rather than a genuine improvement in performance metrics."

Reply: Thank you for the question. We would like to first clarify that our work improves the existing literature [1] not only in the regret bounds but also in the applicability of the algorithm. To be specific, [1] also investigates policy learning under conditional reward shift; their methodology and theory, however, require the covariate $X$ to have a finite support and a lower-bounded probability mass function, which restricts its applicability to real-world problems with continues covariates. Our work can handle general types of $X$ (with a finite or infinite support). This extension substantially improves the applicability of the algorithm, and is technically highly nontrivial --- in terms of algorithmic design and theoretical treatment. (At a high level, the challenge here is that we cannot enumerate all the possible values in the support of $X$ as in [1].) In the meantime, we also improve the regret bound $O(n^{-\frac{1}{2}}\log n)$ in [1] by a $\log n$ factor through the chaining proof technique.

Weakness #2: "Additionally, while the introduction raises the question of "optimal worst-case average performance" (lines 75-77), the paper lacks an optimality analysis of the derived bounds. Consequently, the theoretical contributions do not align strongly with the paper's motivation and feel somewhat insufficient in addressing the posed questions."

Reply: Thank you so much for your helpful suggestion. We have added a lower bound on the regret that matches our upper bound in terms of sample size and the policy class complexity, establishing the optimality of our algorithm. Please refer to our latest version for details.

Weakness #3: "Furthermore, the experiments are conducted solely in a synthetic environment, with no practical examples illustrating the benefits of the proposed algorithm."

Reply: Thank you for your helpful suggestion! In the revision, we have added a new experiment with a more realistic setting that mimics real-world concept shifts (we evaluate the learned policy under target environment instead of directing evaluating the theoretical worst-case policy value). Please refer to our latest version. We leave the experiments using real-world dataset for future works, once we have access to such datasets.

The authors would like to kindly remind the reviewer to read the responses before the discussion deadline. We have revised the manuscript according to the helpful suggestions. Any further comments would be deeply appreciated!

Thank you so much for your response! We would like to ask if you have remaining concerns that have not yet been addressed?