Off-Policy Selection for Initiating Human-Centric Experimental Design

We sincerely appreciate your time and efforts on evaluating our work. Please find our point-by-point response below.

Q1. The proposed method is applicable only to problems with a finite number of policies, as policy selection is based on evaluating each candidate target policy separately.

R1. Good question. We focused on a common scenario when deploying RL policies to real human participants in practical human-centric systems, where all deployed policies have to be highly regulated and scrutinized by departmental committees enforcing the guidelines for human-centric experiments, where the total number of deployed policies would be generally limited [1-4],

Q2. The real-world experiment was not a randomized controlled trial that directly compared the proposed method against baseline methods. Instead, in the IE system, a policy was randomly assigned to each student. The authors then tracked the outcomes for students in each subgroup who were assigned the policy recommended by each method.

R2. Unfortunately, as we had to follow the pre-defined guidelines agreed with a departmental committee, that each student has to be treated equally during the experiment (i.e., they either opted-in for testing out all the methods or none). As a result, we were not able to run typical randomized control experiments. However, the chi-squared test was employed to check the relationship between policy assignment and sub-groups, and it showed that the policy assignment cross sub-groups were balanced with no significant relationship (p-value=0.479), as described in Appendix A.2.

Q3. What is the error bar in Figures 1 and 2? Is it the standard deviation, standard error, or confidence interval? If it is the standard error, the difference between the baseline and the proposed method is small compared to the variation within each method.

R3. The error bar represents standard error. Though the difference may not be large for potential high-performing sub-groups (e.g., $K_1$ & $K_2$), we observed that the baselines can even have a negative effect on some sub-groups ($K_3$ & $K_4$) as in Figure 2, which is undesired in human-centric experiments. In empirical human-centric scenarios, such as education, the behavior policy is generally highly regularized by department committees strictly following guidance from human-centric experiments, such that the target policy would not be dramatically opposed to the behavior policy – suggesting the underlying assumption that the divergence between behavior and target policies could be intrinsically bounded. We will definitely clarify this in the camera-ready version, if accepted.

Q4. In Figure 1(b), most methods overestimate the reward with the proposed estimator. Could the authors explain the reason for this? Is there a systematic bias in the estimator?

R4. This is probably due to the unobserved confounding existing in human-centric systems, leading the method that assumed sequential ignorability for the behavior policy to overestimate or underestimate, and similar findings were observed in [5-6].

Q5. In Figure 2, FPS and the best possible baseline combinations perform exactly the same in subgroups K1 and K2. Is this because the two methods assign the same policy to every student in these first two subgroups?

R5. Both FPS and baselines selected the same policy for sub-groups $K_1$ and $K_2$.

We hope these answers provide some explanations to address your concerns and showcase that our work is solving a significant challenge in a satisfying manner. We are happy to answer any followup questions or hear any comments from you.

References

[1] Mandel et al. Offline policy evaluation across representations with applications to educational games. AAMAS 2014.

[2] Gao et al. Offline policy evaluation for learning-based deep brain stimulation controllers. International Conference on Cyber-Physical Systems 2022.

[3] Zhou et al. Hierarchical reinforcement learning for pedagogical policy induction. International Conference on Artificial Intelligence in Education 2019.

[4] Abdelshiheed et al. Leveraging deep reinforcement learning for metacognitive interventions across intelligent tutoring systems. International Conference on Artificial Intelligence in Education 2023.

[5] Namkoong, et al. Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.

[6] Fu et al. Benchmarks for Deep Off-Policy Evaluation. ICLR 2021.

We sincerely appreciate your time and efforts on evaluating our work. Please find our point-by-point response below.

Q1. How is this sub-group partitioning actually conducted?

R1. We used an off-the-shelf algorithm called Toeplitz inverse covariance-based clustering (TICC) [1] to obtain the initial partitioning as described in Appendix A.3.5, to jumpstart the iterative optimization process toward objective (1) that follows. It could be our writing that causes such a confusion, and we will clarify that in the camera-ready version, if accepted. Please see the response below for the concern over variance.

Q2. Variance in the sub-group partitioning phase:

Q2.1. How does this algorithm deal with the variance in the sub-group partitioning phase?

R2.1. We thank the reviewer for this thoughtful comment where we carry out additional analyses to address it, which lead to additional findings. Specifically, we alter the total number of subgroups, $M \in \{4,5,6\}$, and re-run the sub-group partition. It is observed that a minimal percentage of students have had their group changed over different $M$'s -- see the table below. This may not be surprising, because in empirical human-centric scenarios, such as education, the policies are generally highly regulated and scrutinized by departmental committees strictly enforcing the guidelines for human-centric experiments, so that both behavioral and target policies deployed at scale would not interfere students' learning greatly. There it implies the underlying assumption that the divergence between behavior and target policies could be intrinsically bounded. Such a finding is important and can be potentially generalized to common human-centric environments, and we plan to further pursue such an avenue in broader contexts both empirically and theoretically in the future.

Q.2.2. Using silhouette scores is distance-based and does not consider the variance. It would be useful if there is a way to determine the number of clusters (M) in a data-driven way, taking both bias and variance into account.

R2.2. Given our work is the first targeting and solving a practical challenge often encountered in OPS, we provided the framework in a straight-forward manner, and silhouette score has been broadly employed and exhibits good performance across human-related tasks [1-3]. However, we greatly appreciate this comment and agreed that it would be an interesting topic to investigate in the future toward bias and variance in partitioning.

Q3. Improvement over Keramati et al. seems incremental:

Q3.1. I understand the practical advantages, however, the technical progress is not convincing, as the way the proposed method overcomes the challenges in sub-group partitioning without full trajectory is not well-described.

R3.1. We respectfully disagree with the opinion that the work is incremental over Keramati et al. Though the theoretical analysis of bound of variance in both work may be grounded on [4-5], please note that the objective of our work is to identify the policy that can possibly work the best for each arriving individual of HCSs from a set of policy candidates, without access to any prior offline data or entire trajectory collected over the individual, which targets to address a common bottleneck in empirical human-participated studies. In contrast, Keramati et al. assumed that there existed a population-level (optimal) policy and then identified the sub-group that may benefit most of that policy, which is the opposite side of our problem statement. Moreover, they required the entire trajectory for each individual to be pre-given before sub-grouping, which would not be practical for HCSs in the real-world, as the initial state would be the only observation available at the time an individual arrives at the HCS right after which a group needs to be assigned (e.g., doctors need to sketch out diagnoses/treatment plan for patients soon after they stepped into the triage room).

Q3.2. how does the performance of the proposed method differ from Keramati et al.?

R3.2. As we mentioned above, given the complicated real-world human-centric environments and highly limited information of each incoming individual, it’s hard to directly implement Keramati et al. work to our empirical experiments, as the assumption required would not be met, nor the availability of the trajectory beyond the initial state for their method to use for sub-grouping.

Q4. Why FPS-P can be quite pessimistic while FPS and FPS-noTA are rather optimistic.

R4. There could be some unobserved confounding that exists in human-centric experiments, leading the method that assumed sequential ignorability for the behavior policy to underestimate or overestimate, and similar findings were observed in [6].

We hope our responses sufficiently addressed your concerns and clarified that our work is solving a significant practical challenge effectively. We are happy to respond to any follow-ups you may have.

References

[1] Toeplitz inverse covariance-based clustering of multivariate time series data. KDD 2017.

[2] Domain generalization via model-agnostic learning of semantic features. NeurIPS 2019.

[3] A Hierarchical Clustering algorithm based on Silhouette Index for cancer subtype discovery from genomic data. Neural computing and applications 2020.

[4] Learning bounds for importance weighting. NeurIPS 2010.

[5] Policy optimization via importance sampling. NeurIPS 2018

[6] Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.

Thank you for your time and efforts on evaluating our work, and your positive comments that the paper is making an important impact. Please find our point-by-point response below.

Q1. Assumption of Independent Initial State Distributions The FPS framework assumes that the initial state distributions for each participant are independent and can be uniformly sampled from the offline dataset. This assumption may not hold true in real-world scenarios where participants’ initial states can be influenced by various contextual factors and past interactions. The independence assumption may oversimplify the complexity of human-centric systems and may lead to suboptimal policy selections. The authors may consider addressing the potential dependencies in initial states.

R1. Thank you for the insightful comment. That independence assumption we made was inspired by the use of uninformative prior in Bayesian methods where we do not have much prior knowledge in terms of sub-grouping's potential outcome before the experiment starts. We agreed with the reviewer that investigating the potential confoundings in initial states would be important -- and we plan to do so as a separate work in the future, as it could be very challenging to capture those in practical scenarios, e.g., there exist a few literature attempting to soly address that issue [1-3].

Q2. Lack of Consideration for non-stationary state transition FPS focuses on the initial state for policy selection without considering longitudinal data that captures the progression of participant states over time. This, in the AI community's language, is a non-stationary state transition issue faced by meta-RL. Is it possible that the state transition for each patient is also independent of each other? In other words, chances are that the state transition for each patient 𝑝𝑖 is sampled from a distribution. Would this FPS work in such a case?

R2. Great point. Initially we did not consider the progression/transition of participant states since the overall scenario our work considered was that each individual needs to be assigned a group upon arrival where the initial states would be the only observation available. However, we agreed that our framework can be further extended toward also considering each participant's follow-up visits once they are available. And if the state transitions are also independent from each other, a simple variation of our method may work, as one could re-run the partitioning algorithm at each new visit; this way it would not violate the assumptions needed by our work. Further, in the future we plan to explore the possibilities of temporally dynamic sub-group partitioning, where the agent can consider all the historical states of a participant upon arrival of each visit. It would make the problem setup more challenging as it falls more under a POMDP schema. We greatly appreciate both of these comments as they indeed helped us to shape the future work down the avenue and led us to think of more challenging, interesting and realistic setups. We also hope that our work could possibly inspire more researchers to recognize and emphasize the empirical challenges faced by deploying RL/OPS in real-world systems, if accepted.

Q3. The paper does not thoroughly address FPS's scalability and applicability in large-scale, real-world settings, especially in healthcare. I suggest the authors clarify the scalability and emphasise the importance of clinical guidance. One to two sentences will be enough.

R3. We sincerely appreciate your suggestion. We will add the following discussion in the camera-ready, if accepted, i.e., "Compared to IE systems, HCSs in healthcare would be considered even more high-stakes, thus may further limit the options (i.e., policies) that are available to facilitate sub-grouping experiments, due to stricter clinical experimental guidelines. However, FPS has demonstrated its extraordinary capabilities over a real-world experiment that involved >1,200 participants with years of follow-ups, which showed its efficacy and scalability toward working with more challenging systems and larger cohorts as in healthcare, as the assumptions needed by FPS across these two systems would not change fundamentally. Moreover, potential underlying confoundings may exist across the patient's initial states in healthcare, and it is also important to consider inputs from healthcare professionals during sub-grouping. As a result, one may further extend our framework toward such a direction, allowing it to function better in the healthcare domain."

We hope these answers provide some explanations to address your concerns and showcase that our work is solving a significant challenge in a satisfying manner. We are happy to answer any followup questions or hear any comments from you.

References

[1] Namkoong et al. Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.

[2] Xu et al. An instrumental variable approach to confounded off-policy evaluation. ICML 2023.

[3] Tennenholtz et al. Off-policy evaluation in partially observable environments. AAAI 2020.

Dear authors, reviewers, ACs,

Just for a quick clarification,

degraded the rating from 4 to 3 without providing any reasonings behind after our latest clarification response

I changed my evaluation for this reason: in the initial evaluation phase, I still had some questions on how the user-partition is conducted, therefore, I was setting a neutral score on my evaluation. After reading the rebuttals, it turned out that the user partition itself does not have any further advancement over existing work, especially regarding the theoretical analysis. I thought this point should be carefully considered as a NeurIPS paper, and therefore, I updated the evaluation towards a not-neutral score.

However, I do acknowledge the authors' effort on the augmentation part to enable OPE with only the knowledge about the initial state distribution of the evaluation policies. Also, I am not strongly against the acceptance.

I'd be happy to discuss the above points with other reviewers and ACs.