ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making

Dear Reviewer Yyxa,

We appreciate your feedback and have provided responses to your questions below.

1. Could you provide a substantive argument of why optimizing ROI could offer a better framework for cost/returns trade-offs?

Constrained reinforcement learning is a framework that aims to maximize the return within a given cost budget, rather than optimizing the trade-off between returns and costs. In other words, constrained RL agent does not try to minimize the accumulated cost if it is below the budget. However, ROI maximization focuses on maximizing the return/cost ratio, meaning the accumulated cost can be minimized further to achieve a higher ROI.

If our main goal is to maximize ROI, ROIDICE is much more efficient compared to utilizing constrained RL algorithms. As described in Section 5.1, for ROI maximization, constrained RL requires multiple runs with different levels of cost budgets, and the policy with the best ROI performance can be selected afterward. However, ROIDICE requires only a single run to obtain the policy with the best ROI performance, eliminating the need to search among cost budget candidates.

2. It is fairly common in RL to approach cost/return trade-off by integrating the action costs as negative rewards.

In our paper, ROIDICE has been compared with offline RL and offline constrained RL. As the reviewer has pointed out, the return/cost trade-off can be optimized with a regularized RL framework with its reward $r(s,a)-\lambda c(s,a)$. However, we believe that constrained RL is a better baseline for ROIDICE when considering practical usages.

Theoretically, there always exists a regularized RL with a certain $\lambda$ that provides the same solution as constrained RL for a given cost budget. In some constrained RL algorithms using Lagrangian, $\lambda$ is implicitly optimized to satisfy the cost constraint[1]. However, finding an appropriate $\lambda$ to compare regularized RL methods with ROIDICE can be difficult, particularly when the scale of reward and cost varies significantly. In contrast, constrained RL is more practical, as a suitable cost budget can be inferred from the dataset, alleviating the need for an explicit search for $\lambda$.

3. How stable are the policies optimized in problems where the returns and possible costs may have very different dimensions and distributions?

As mentioned in the Limitations section, ROIDICE can be influenced by the design of reward and cost functions. The variation of reward and cost across state-action pairs significantly impacts ROIDICE as it maximizes the ratio of return to accumulated cost. For instance, as noted in Section 3, when the cost is constant across all states and actions, the ROIDICE policy becomes equivalent to the standard offline RL policy. Conversely, as the distributions of cost become more diverse, the ROIDICE policy deviates from the offline RL policy.

On the other hand, analogous to typical RL algorithms not being affected by the scales of reward function, ROIDICE is basically invariant to different scales of rewards and costs apart from the choice of appropriate $\alpha$. We also empirically observed that ROIDICE resulted in a same level of performance when $\alpha$ is appropriately scaled to align with the cost scales.

4. Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain?

We can come up with a number of practical scenarios where ROI would be a good objective, such as optimizing the mileage (km/L) of an autonomous vehicle, the (decision optimality/planning time) trade-off when adopting a meta controller for real-time decision making, or the speed-accuracy tradeoff (generation quality/inference time) of a large language model (LLM). We illustrate a comparison between two approaches for optimizing the mileage problem.

• Constrained RL: An agent trained with constrained RL uses a limited budget to maximize the reward. Setting an appropriate budget that is feasible for the problem may require domain knowledge. Therefore, to compare the ROI performance of its policy, it is necessary to conduct multiple runs of constrained RL with different levels of constraints (e.g., fuel consumption $\leq$ [5, 10, 15, 20]). The policy with the best mileage is then selected from these multiple runs of constrained RL.
• ROI maximization: A single run of ROIDICE is sufficient to find a policy that maximizes the vehicle's mileage.

Ideally, the mileage (ROI) performance of the ROIDICE policy would be equal to or greater than that of the constrained RL policy. Equality occurs when the upper bound of the constraint matches the fuel consumption of the ROIDICE policy.

5. How significant is the decrease in policy performance compared to cost-free policies in practice for?

The remark in Section 3 notes that the optimal policies of standard RL and ROI maximization are equivalent when the cost is constant across all states and actions. This gap increases when the cost varies among states and actions or when there is an action with high cost that leads to high return. The difference between standard RL and ROIDICE can be managed by adjusting the cost distribution.

6. Does the proposed approach provide some guarantees on cost?

Current form of ROIDICE does not guarantee the upper bound on cost. However, ROIDICE with the cost guarantee can be easily derived by adding the cost constraint $\sum_{s,a}d(s,a)c(s,a)\leq C_{\text{threshold}}$ to the LP formulation of our algorithm, ROI-LP.

Thanks for pointing out the issue with references. We will fix this in the next revision.

[1] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022.

Dear Reviewer Yyxa,

Could you please take a look at the rebuttal and share your thoughts on the authors' reply? The author has provided a detailed response to the initial comments - does it address your major concerns?

Thank you very much for your time and effort!

Best,

Your AC

Dear Reviewer T1wG,

We thank the reviewer for the detailed feedback. We address your remarks below.

1. Important baselines are missing: apart from DICE-based methods, comparisons should include other offline constrained RL methods.

Our experiment in continuous domains from Section 5.2 includes non-DICE offline constrained algorithm, CDT[1]. We additionally provide experiments on other offline constrained RL frameworks, VOCE[2] and CPQ[3]. Table 1 in the rebuttal supplementary presents the ROI performance of these constrained offline RL algorithms in the Hopper environment[4]. These algorithms use two cost budgets, corresponding to the 50th and 80th percentiles of the accumulated costs from the offline dataset.

Over various experiments, constrained offline RL algorithms commonly have shown low ROIs. We conjecture that these results are mainly due to the overestimation of costs, making constrained offline RL algorithms to be overly conservative on choosing costly state-actions. On the other hand, ROIDICE seems to be less affected by such overestimation, as overestimated costs are partially mitigated with overestimated rewards when estimating ROIs.

2. How is the proposed method implemented in continuous domains?

We provide the pseudocode of ROIDICE in the rebuttal supplementary. Lagrangian multipliers are parameterized and updated to optimize ROIDICE objectives presented in our paper. Optimal policy $\pi^*_\theta$ is extracted from the optimal ratio $w^{*}_{\nu,\mu,t}$ obtained from the Lagrangian multipliers.

3. In the FinRL task, why the unconstrained RL performed better than constrained RL and what makes the constrained RL methods perform poorly?

As reported in the appendix, in FinRL task, constrained RL algorithms are trying to achieve lower cost compared to unconstrained RL algorithms by sacrificing large amount of its return. While both constrained and unconstrained RL agents are resulting in a policy with smaller cost compared to the cost budget, we assume that constrained RL algorithms are largely overestimating the cost, resulting in a overly conservative policies.

4. Can you provide experimental results on widely recognized safe RL environments?

We evaluate our approach on OpenAI SafetyGym[5] tasks including CarGoal and PointPush. Table 2 in the rebuttal supplementary materials provides results averaged over 5 seeds across 10 episodes. Similar to the reported results in the paper, we observed that ROIDICE outperforms COptiDICE in terms of ROI. We use the rewards and costs from the environment, adding a constant value of $\epsilon=0.1$ to each cost to maintain our assumption that $c(s,a) > 0 \ \forall s,a$. The offline dataset is collected by PPO Lagrangian. ROIDICE uses $\alpha=0.001$ for CarGoal and $\alpha=0.01$ for PointPush. COptiDICE[6] uses $\alpha=0.01$ for CarGoal and $\alpha=1.0$ for PointPush. We will include the results in the final version of the paper if accepted.

[1] Z. Liu, et al. Constrained decision transformer for offline safe reinforcement learning. ICML, 2023.

[2] J. Guan et al. VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning. NeurIPS, 2024.

[3] H. Xu, et al. Constraints penalized q-learning for safe offline reinforcement learning. AAAI, 2022.

[4] J. Fu, et al. D4rl: Datasets for deep data-driven reinforcement learning, 2020.

[5] A. Ray, et al. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019.

[6] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022.

Dear Reviewer 8zrU,

We appreciate your comprehensive comments. Please find the response to your questions below.

1. Theoretical analysis for the sample complexity in ROI-LP.

Sample complexity analyses in the LP formulation of RL traditionally estimate a policy's return using a linear combination of its stationary distribution and reward. On the other hand, ROI is defined as a ratio between the linear combinations of the stationary distribution with reward and cost, making existing approaches not directly applicable.

To illustrate this, we demonstrate that the method proposed in [1] cannot be directly applied to determine the sample complexity of the ROI-LP. Specifically, Lemma 3 in [1] cannot be straightforwardly derived from the ROI-LP. The lemma assumes an approximated stationary distribution ${\theta}\in \mathbb{R}^{|S||A|}$ and policy $\pi_{\theta}$ extracted from $\theta$. $d_{\pi_{\theta}}\in \mathbb{R}^{|S||A|}$ is the true stationary distribution generated by policy $\pi_{\theta}$. Lemma 3 provides an error bound that relates the Bellman flow constraint violation of $\theta$ and the absolute difference between $\sum_{s,a}\theta(s,a)r(s,a)$ and $\sum_{s,a}d_{\pi_\theta}(s,a)r(s,a)$.

Lemma 3 For any $\theta\geq0$ and $r(s,a)\in[0,1]$, we have $|\sum_{s,a}r(s,a)(\theta(s,a)-d_{\pi_{\theta}}(s,a))|\leq{||M\theta -(1-\gamma)p_{0}||_{1} \over 1-\gamma}$,

where $M=B_{\*}- \gamma T_{\*}$.

We show that a straightforward extension of the lemma is not feasible in the ROI-LP context. The pair $(\tilde{\theta},\tilde{t})$ serves as an approximation of $(d^{'},t)$ in ROI-LP, and the policy $\pi_{\tilde{\theta}}$ is extracted from $\tilde{\theta}/\tilde{t}$. The actual stationary distribution produced by $\pi_{\tilde{\theta}}$ is $d_{\pi_{\tilde{\theta}}}^{'} / t_{\pi_{\tilde{\theta}}}$. We begin by assessing the extent to which $(\tilde{\theta},\tilde{t})$ violates the scaled Bellman flow constraint in ROI-LP.

$\Vert M\tilde{\theta}-(1-\gamma)p_{0}\tilde{t}\Vert_{1} = \Vert M\tilde{\theta}-M\tilde{t}d_{\pi_{\tilde{\theta}}}^{'} / t_{\pi_{\tilde{\theta}}}\Vert_{1}\nonumber\geq(1-\gamma)\Vert B_{\*}\tilde{\theta}-B_{\*}d_{\pi_{\tilde{\theta}}}^{'} \tilde{t}/t_{\pi_{\tilde{\theta}}}\Vert_{1}$

where a valid stationary distribution $d_{\pi_{\tilde{\theta}}}^{'} / t_{\pi_{\tilde{\theta}}}$ satisfies $(1-\gamma)p_0 = Md_{\pi_{\tilde{\theta}}}^{'} / t_{\pi_{\tilde{\theta}}}$.

The ROI difference between $\sum_{s,a}\tilde{\theta}(s,a)r(s,a)$ and $\sum_{s,a}d_{\pi_{\tilde{\theta}}}^{'}(s,a)r(s,a)$ can be bounded by,

$|\sum_{s,a}r(s,a)(\tilde{\theta}(s,a) - d_{\pi_{\tilde{\theta}}}^{'}(s,a))|\leq\sum_{s}|\sum_a \tilde{\theta}(s,a) - d_{\pi_{\tilde{\theta}}}^{'}(s,a)|=\Vert B_{\*}\tilde{\theta}-B_{\*}d_{\pi_{\tilde{\theta}}}^{'}\Vert_{1}$

It can be noted that, due to the additional factor $\tilde{t} / t_{\pi_{\tilde{\theta}}}$ that cannot be easily bounded under finite samples, ROI-LP cannot use the same proof technique to [1] for sample complexity analysis.

While the sample complexity analysis is indeed an important research topic, we believe that conducting a new type of sample complexity analysis appropriate for the Linear-Fractional Programming framework is beyond of the scope of this paper.

[1] Ozdaglar, Asuman E., et al. "Revisiting the linear-programming framework for offline rl with general function approximation." International Conference on Machine Learning. PMLR, 2023.

Dear Reviewer Z94y,

We thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.

1. Could you provide details on the computational resources and time consumption for different methods

We provide details on the resources and runtime to demonstrate that ROIDICE does not require a significant amount of resources for large-scale domains, such as locomotion and finance. All algorithms, including baseline methods, were trained for 100K iterations on a single NVIDIA RTX 4090 GPU.

2. What if the data distribution in the offline dataset significantly differs from that encountered in the online environment?

In the context of offline reinforcement learning (RL), a major challenge arises from the degradation in performance due to increasing discrepancies between the offline data distribution and the online environment. To mitigate this issue, DICE-based RL frameworks—such as ROIDICE, OptiDICE, and COptiDICE—employ f-divergence to regulate the extent of conservativeness, which is modulated by the hyperparameter $\alpha$. As illustrated in Figure 5 of Appendix D, which corresponds to Figure 1 in the supplementary materials, selecting an appropriate $\alpha$ can identify an offline RL policy that is less affected by distributional shift.

[1] J. Lee, et al. Optidice: Offline policy optimization via stationary distribution correction estimation. ICML, 2021.

[2] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022.

[3] Z. Liu, et al. Constrained decision transformer for offline safe reinforcement learning. ICML, 2023.