Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation

We would like to thank the reviewer for valuable and thoughtful feedback. In particular, we appreciate the supportive comments and acknowledgment of our contributions. We will respond to the key comments and questions below.

The classic sharpe ratio uses the mean return of the portfolio as the numerator instead of the best policy's return. Is there a particular reason that the best return is used in the proposed metric?

This is a great point. We focus on the best policy’s performance in the numerator because only the best policy in the policy portfolio will be finally chosen via A/B testing as a production policy, as illustrated in Figure 1. To define the numerator, we also subtract the behavior policy’s performance ($J(\pi_b)$). By doing so, we can avoid an undesired edge case where an estimator that chooses only policies that are worse than the behavior policy but accidentally has extremely small std (denominator) is evaluated as the best method under our metric. We will clarify this in Section 3.

The benchmark results show that SharpeRatio@k can sometime diverge significantly from the conventional metrics in terms of estimator selection. Additional discussion should be added on how practitioners may consolidate the insights given by the different metrics evaluated under certain scenarios.

Thank you for the valuable feedback. The estimator selection results under SharpeRatio@k sometimes diverge from others because SharpeRatio takes the risk into account when evaluating OPE estimators, while other metrics ignore the risk factor. In general, it would be ideal to be able to develop an estimator that performs best in terms of both existing metrics and SharpeRatio. If it is not possible, one should prioritize SharpeRatio@k more when they do not want to take any risk of deploying detrimental policies in the A/B test. When an estimator is preferred by only existing metrics, the estimator is likely to be high-return but also high-risk, so it should be avoided in high-stakes scenarios such as medical applications. We will provide an in-depth discussion about this point in the revision.

We would like to thank the reviewer for valuable and thoughtful feedback. We will improve the presentation of the paper following the reviewer's suggestions. We will respond to the key comments and questions below.

it wasn’t clear in Fig 2 what the red dots are, black dots are and how to interpret the axes.

I found the sentence “X underestimates the performance… Y overestimates ” confusing – I was wondering if it should be the other way around. If the current sentence has to be true, then my understanding of the interpretation is that black policies suffer an underestiation (i.e. even though x-axis value is high, y-axis value is lower than ground truth) and so the top-k ends up picking next best policies. Is this correct or am I missing something?

Thank you for catching the ambiguity. The red dots indicate the policies whose values are estimated correctly by the estimator, while the black dots indicate the policies whose values are not correctly estimated.

Your understanding about estimator X (“my understanding of .. next best policies”) is correct indeed, and the three points in the shaded region are chosen as the top-$k$ policies under estimator X (these three policies have higher estimated policy value (y-axis) than others).

On the other hand, estimator Y substantially overestimates the value of black policies. As a result, one red policy (the rightmost one) and two black policies are chosen as the top-$k$ policies (policy portfolio) under this estimator, which is considered risky because the true values of these black policies are quite low.

Related work: In .. “Risk Measures and risk-Return Tradeoff in Statistics and Finance”, the paper discusses the Sharpe,1998 paper in length .. However, there are only two other related works .. I’m not an expert in this domain but I suspect there has to be more prior research done in this space – and if that is true, the existing related work section seems a little thin.

Thank you for the valuable comment. Indeed, there are several variants of risk measures used in finance such as the information ratio, value at risk (VaR), marginal VaR, and entropic VaR (some of them have been used in bandits and RL). We will add comments about these relevant metrics in the section and keep improving the writing of the paper.

top-k policies are considered in several resource allocation setups where there are limited resources and some index is computed so that resource can be allocated to the top k indexes .. Can the method be using for designing robust top-k selection/resource allocation policies in this context? I also wonder how it compares to existing robustness in bandits work such as .. and whether those ideas can be applied to the OPE evaluation setting discussed in the current paper.

We agree that the high-level concept of the risk-return tradeoff can be applied to the problem of resource allocation and restless bandit as suggested by the reviewer. A notable difference is that we used the concept to define a new metric to evaluate OPE estimators while the same concept can be useful to define a new performance measure of a bandit policy in the field of resource allocation and restless bandit. We can actually find some similar attempts such as the risk-aversion bandit (Sani et al. 2012). Even though this is an independent interest, we thank the reviewer for sharing this interesting thought.

(Sani et al. 2012) Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-Aversion in Multi-armed Bandits. NuerIPS2012.

We would like to thank the reviewer for valuable and thoughtful feedback. We will respond to the key comments and questions below.

My biggest concern is about the technical contribution of the work. While the idea seems novel it also sound quite natural and simple and bears concern about the actual interest that would spark in the community.

Thank you for the great comment. First, we think that the simplicity of our evaluation metric is actually one of its important advantages. It means that it can be used fairly easily in future research and practice. Reviewer n49S acknowledges that being natural and easy-to-use is one of the main advantages of our metric. Second, a method’s simplicity does not necessarily mean that it is not novel and valuable. The concept of evaluating OPE estimators’ risk-return tradeoff and the use of SharpeRatio as an evaluator of OPE estimators are the ideas that we have never seen in the relevant literature. Besides, the similar types of contributions have been evaluated highly by the relevant community. We can actually list several published papers that share similar types of contributions to ours, i.e., (1) highlighting the problem of existing evaluation protocols and (2) resolving the issues with simple and easy-to-use metrics.

• (Fu et al., ICLR2021) (1) This paper identified that an OPE estimator that has a lower estimation error (MSE) does not always align candidate policies better than those with a higher estimation error. (2) To address this gap in evaluating policy alignment in OPE, the paper proposed simple rank correlation and regret to see if OPE estimators align candidate policies well. These metrics are now recognized as important metrics of OPE and are used in e.g., (Tang and Wiens, 2021), (Qin et al, 2022), and (Chang et al., 2022).

• (Agarwal et al., NeurIPS2021) (1) This paper identified that taking the naive average of the performance of online RL can be sensitive to variance, particularly when only a handful of trials are allowed due to high computational costs. (2) To enable a more statistically robust evaluation of online RL in such situations, the paper proposed to use the simple average of the middle 50% of trials. This paper was awarded the Outstanding Paper Award for NeurIPS 2021.

• (Chan et al., ICLR2020) This paper (1) identified that the naive average of the performance of online RL may overlook a critical safety concern. (2) To see if the RL policy is risk-averse or not, the paper proposed to report the simple average of the worst $\alpha$% of trials (referred to as conditional value at risk; CVaR). CVaR is widely used in safe-RL.

It would thus be great if the reviewer could evaluate our contributions based on their quality, significance, soundness, impact, rather than what the proposed method looks like. It would also be great if the reviewer could pay attention to another key contribution we have provided, i.e., the development of the open-source software.

====

(Fu et al., ICLR2021) Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, Tom Le Paine. “Benchmarks for Deep Off-Policy Evaluation.”

(Agarwal et al., NeurIPS2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.”

(Chan et al., ICLR2020) Stephanie C.Y. Chan, Samuel Fishman, John Canny, Anoop Korattikara, Sergio Guadarrama. “Measuring the Reliability of Reinforcement Learning Algorithms.”

(Tang and Weins, 2021) Shengpu Tang, Jenna Wiens. "Model selection for offline reinforcement learning: Practical considerations for healthcare settings." Machine Learning for Healthcare Conference, 2021.

(Qin et al., 2022) Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. “NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning.” NeurIPS Datasets & Benchmarks, 2022.

(Chang et al., 2022) Jonathan D. Chang, Kaiwen Wang, Nathan Kallus, Wen Sun. “NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning” ICML, 2022.

We would like to thank the reviewer for valuable and thoughtful feedback. We will respond to the key comments and questions below.

My biggest issue with this paper is that the work is very limited in scope which limits the benefit to the larger community.

Thank you for the great point. First, off-policy evaluation (OPE) of bandits and RL is an established and broad research area and is widely recognized at top-tier conferences like ICLR, ICML, and NeurIPS (more than 15 relevant papers were accepted to these venues in each of 2021-2023). Second, our work highlighted the issue of the existing evaluation protocol of OPE estimators and proposed a new framework to evaluate their risk-return tradeoff. In Section 4, we have indeed demonstrated that estimators that were considered pretty basic can be the most effective method under our metric. This would have a potential to change the way we evaluate the effectiveness of estimators and develop new estimators in the future of the entire OPE research community as we discussed in Section 5. It would also be possible that our concept of evaluating the risk-return tradeoff will lead to a development of new evaluation metrics for other fields such as fair machine learning, recommender systems, and treatment effect estimation. Thus, we believe that our work is not limited in scope but provides a whole new perspective regarding the evaluation of OPE and potentially many other fields. We will add this discussion in the revision.

While the proposal to use ideas from portfolio theory is interesting, the authors focus on a fairly simple definition

Thank you for the great comment. First, the simplicity of our evaluation metric is actually one of its important advantages. It means that it can be used fairly easily in future research and practice. Reviewer n49S also acknowledges that being natural and easy-to-use is one of the main advantages of our metric. Second, a method’s simplicity does not necessarily mean that it is not novel and valuable. The concept of evaluating OPE estimators’ risk-return tradeoff and the use of SharpeRatio as an evaluator of OPE estimators are the ideas that we have never seen in the relevant literature. Besides, the similar types of contributions have been evaluated highly by the relevant community. For example, (Fu et al., ICLR2021) proposed the rank correlation and regret as simple evaluation metrics for OPE. (Agarwal et al., NeurIPS2021) proposed the average of the middle 50% of trials as the online RL performance. The values of these papers have been recognized widely because they highlight the issues of existing evaluation protocols and develop a simple metric to deal with them. So is our work, and we believe that the reviewer’s comment actually highlights one of the most important pros of our metric. It would thus be great if the reviewer could evaluate our contributions based on their quality, significance, soundness, impact, rather than what the proposed method looks like.

===

(Fu et al., ICLR2021) Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, Tom Le Paine. “Benchmarks for Deep Off-Policy Evaluation.” ICLR, 2021.

(Agarwal et al., NeurIPS2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.” NeurIPS, 2021.