On Trajectory Augmentations for Off-Policy Evaluation

Thank you for your time and efforts on evaluating our work. Please find our point-by-point response below.

Q1. The main paper omitted some significant parts and placed them in the appendix, however should be explained in the main context, such as how the latent latent prior is constructed. And how the C different clusters are initially divided when identifying the PST is not clear.

A1. Due to the space limitations, we leave the technical details straightforwardly adapting VAE into MDP in Appendix. But we moved the details of constructing latent prior to Section 2.2 (highlighted in blue). The number of clusters is determined by a fully data-driven procedure following the original TICC paper (i.e., C is determined with the highest silhouette score in clustering historical trajectories) [1].

Q2. Even though the experiment could show that the OPE method performance is improved, the paper is suggested to make a fair analysis of how reliable the reward is from the algorithm augmentation, which, however, is significant for the result value of off-policy methods.

A2. We provided Tables 4-16 in Appendix C.1, presenting the original results, in terms of four different metrics (i.e., mean absolute errors (MAE), rank correlation, regret@1, and regret@5), on each OPE method without and with augmentation in Adroit. Given MAE directly measures the distance between estimated and true returns, we can observe that OAT leads to more accurate return estimates for different OPE methods on target policies. Especially for IS that purely relies on the importance ratio over behavior and target actions and rewards from historical trajectories, OAT effectively facilitates it on different tasks, indicating OAT providing more reliable rewards. Moreover, we added scatter plots plotting the true returns of each target policy against the estimated returns, in Appendix A.2.1. We can observe that the estimated rewards of each OPE method were generally improved by OAT, with less distance to true rewards.

Q3. Could the authors discuss if it's only applicable for the behavior data to be augmented? Or can we also augment the target policies?

A3. Our works follow the general OPE problem setup [2], where only a fixed set of offline data collected from behavioral policy is available, and the goal is to use such data to estimate the return of target policies (that are different from behavioral policies), without any online deployment of the target policies. Consequently, we would not be able to augment the trajectories collected under target policies, as they would not be available.

Q4. When identifying the PST, at the first step, How are these C different clusters divided? And how is the number of c determined?

A4. The C different clusters are divided by solving the TICC problem [1], by capturing graphical connectivity structure of both temporal and cross-attributes information. We added those details to Appendix A.3.1 (highlighted in blue). Following [1], C is determined by the one with the highest silhouette score clustering historical trajectories within range [10,20].

Q5. As shown in Fig3, It is suggested to introduce how the latent prior is constructed, since it is the key step in constructing the new augmented PST.

A5. Thanks for the suggestion. We moved the details of constructing latent prior to Section 2.2 (highlighted in blue).

Q6. What is the complexity of the training VAE-MDP? Is there any analysis conducted to show the relations between training time and TDSS length/Potential trajectory scale?

A6. Given that training VAE-MDP requires stochastic gradient descent algorithms with parameter tuning (e.g., for step size and network architecture), we found that analyzing the complexity theoretically would be challenging, as there lacks a standard framework to facilitate that. Instead we provide additional empirical results here on training time -- specifically, we plotted relations between training time and length of PSTs/corresponding TDSSs, in Appendix A.1.1. The results show that the training time of VAE-MDP is increased almost linearly with the length of PSTs.

Dear pod6,

The author reviewer discussion period is ending soon this Wed. Are you satisfied with the author's clarification w.r.t. your questions or there are still outstanding items that you would like to have more discussions?

Thanks again for your service to the community.

Best,
AC

Thank you for your time and efforts on evaluating our work. Please find our point-by-point response below.

Q1. There are some confusing parts/writing errors such as: what is "human-involved RLs" (third line in intro), "while human may behave" (5th line last para of intro), Figure 2 I believe should be referencing Sections 2.1/2.2/2.3 (not 3.X), "intrinsic nature that human may follow" (2nd line Section 2.1).

A1. Thanks for pointing out the typos. We have fixed the references on Figure 2 in the manuscript. For the mentioned “confusing parts’’, please see our comments point-by-point below:

• Human-involved RLs generally refer to the context of RL with human involvements, including human-involved decision-making processes, human-guided RL algorithms, etc., which are commonly discussed in related works [1-5].
• The phrase, “while human may behave”, is discussed under the context that human-involved systems have limited state-action coverage, which can be caused by homogeneous behavior policies (as discussed in 1st paragraph of Introduction); on the other hand, human can behave diversely under different policies [6-7]. Following different policies can result in varied state-action coverage [8-10], which is a fundamental problem in RL.
• The phrase, “intrinsic nature that human may follow”, is under the context discussing an intrinsic nature/characteristic in human-involved systems, that human could follow homogeneous behavioral policies or specific guidelines when they perform their professions.

We have added more references in our manuscript, for readers' potential interests.

Q2. The notation in Section 2.1 is very difficult to parse. I suspect there is an easier way to explain this. Also Figure 3 is very confusing, and is not explained in the text (there are some references in the appendix, but I think there should be much more explanation of it given the complexity of the diagram).

A2. We understood that the notations/equations may be a bit intense in section 2.1, as we are the first work that introduces to OPE with sub-trajectory augmentation, which is a relatively new framework. As a result, we chose to present our methodology as detailed as possible so readers with different backgrounds can get the idea thoroughly (in case they would like to build on top in the future). We have double checked that there do not exist any sub-/super-scripts or variables that are redundant. We further noticed that the other reviewer pod6 specifically pointed out that this part is easy to follow, as well as WbKh who pointed out our figures are informative -- we would greatly appreciate it if the reviewer can point out the specific part of methodology that is redundant/hard to follow.

Q3. I think drawing connections to how humans behave (such as in Figure 2 caption) is too strong. It is appropriate for intuition purposes but making general statements on how humans behave seems a bit strong.

A3. When the trajectories are collected from humans, the states and/or actions are highly related to human behaviors. Similar sub-trajectories may exhibit similar human behaviors. Such findings are reported in related works [21-23].

Q4. I don't think the paper gives a good intuition for why this works. (Q4-1) While coverage is important, it seems like that is insufficient.

A4-1. We respectfully disagree with the comment. Many existing works have justified that improving state-action coverage is an important and still open-ended problem [4, 8-10]. Our work is the first to carry it onto the OPE domain, and attempts to solve it through offline trajectory augmentation.

(Q4-2) Another challenge is distribution shift. Prior work [1] has discussed that under extreme distribution shift, OPE methods can diverge. However, this work does not discuss this.

A4-2. We agree distribution shift is a challenge for OPE. Although the mentioned work by Wang et al. only considers a very specific type of OPE method under a specific setting (i.e., FQE with linear approximation using pre-trained features from neural networks), DOPE benchmark [10] also found that OPE methods in general have such issues. This is beyond the scope of this work, as we use existing OPE methods as backbones to process the augmented trajectories. Hope in the future, the communities will come up with new OPE methods that can resolve the distribution shift in OPE.

We sincerely appreciate your time and efforts on evaluating our work. Please find our point-by-point response below.

Q1. The unsupported justifications of the results.

A1. We added related citations and detailed evidence to justify our findings and statements in the revised manuscript (highlighted in blue).

Q2. The lack of ablations to validate the proposed innovations: (Q2-1)The authors compare against a wide set of benchmarks, although, as I'm not an expert in OPE, it is unclear to me whether these are explicitly tailored for the OPE problem or not.

A2-1. To the best of our knowledge, no prior work systematically investigates trajectory augmentation for facilitating OPE, our work is the first to investigate it and provide a possible solution (i.e., OAT). Thus, we examine a broad range of possible data augmentation methods, which were original proposed towards data that may sharing some characteristics to offline trajectories, or sharing some intuitions with OAT, including: 4 RL-oriented (TDA, Permutation, jittering, scaling), 2 generative models (TimeGAN, VAE), 2 time series-oriented methods by capturing similar behaviors (SPAWNER, DGW). Moreover, we investigated 5 popular OPE methods that are broadly used in prior works [1-11], spanning 3 major categories of OPE as defined by Yue’s group [11].

(Q2-2) Moreover, since the proposed framework is a composed of multiple smaller contributions (discretization, VAE architecture, etc.), the authors should make sure to isolate each of these contributions individually and support the claims with evidence and experiments.

A2-2. Given the key idea is to mine potential sub-trajectories and considering Markovian nature on trajectories, we compared OAT to several ablations:

• OAT w.o. Discretization & support determination (TDA)
• OAT w.o. PSTs mining (VAE-MDP)
• OAT w.o. MDP (VAE)

Note that both discretization and support determination are two common successive techniques to extract shared high-level representations (e.g., PSTs) from high-dimensional complex data [14-16]. Thus, we ablated the proposed PSTs mining from two perspectives: ablating the support determination to identify PSTs from TDSSs by randomly selecting sub-trajectories to augment (which aligns with the original idea of TDA [17]); ablating the concept of PSTs by applying VAE-MDP on entire trajectories (VAE-MDP). The third ablation is applying VAE on entire trajectories, without adaptation to Markovian setting.

Q3. The relatively narrow-scoped experiments. Did the authors consider using this approach within an RL context? How would this perform?

A3. In this work, we specifically focus on the OPE problem [1-10] as the improvements resulting from the model we designed can be isolated, as opposed to policy optimization where the policy improvement step will also impact the performance. We followed the guidelines and standardized procedures introduced in a recent benchmark, DOPE [1], from Levine's group, which provided for each D4RL environment the target policies to be evaluated as well as an off-policy training dataset. The effectiveness and robustness of OAT was extensively validated over 34 datasets with varied characteristics including applications, state-action coverage, human involvements, horizons, density of rewards, dimensions of states and actions, etc. The experiments contained 2 real-world applications, education and healthcare. Moreover, OAT can be stand-alone to generate trajectories without any assumptions over target policies. And it can be easily utilized by built-on-top works such as policy optimization and representation learning. We added such discussions in the conclusion section -- highlighted in blue.

We hope these answers provide some explanations to address your concerns and showcase that our work is solving a significant challenge in a satisfying manner. We are happy to answer any followup questions or hear any comments from you.