Improving Offline RL by Blending Heuristics

Thank you very much for your constructive feedback and inspiring comment on our draft. We hope to answer your questions by providing more theoretical details and additional experiments.

r dependence on the behavior policy may cause problems when data is collected using multiple behavior policies.

In the theoretical analysis, we made a simplifying assumption that the heuristic $h$ is a function of the state in Theorem 1. In this case, HUBL can be viewed as solving a stationary reshaped MDP. In Theorem 2, we further consider a special case that $h(s)$ is the value function of a policy e.g. the behavior policy $\mu$ when proving an exact finite sample upper bound. This extra assumption that the heuristic is a policy’s value function is made as a sufficient condition to establish the heuristic is improvable (see [Definition 4.1, Cheng et al., 2021]), which roughly means that the heuristic value is attainable and as a result can guide the learning (i.e. $\tilde{V}^*(s) \geq h(s)$ [Proposition 4.4, Cheng et al., 2021]). This sufficient condition is used to show the second term in the regret $h(s’) - \tilde{V}^\pi(s’)$ can be non-positive:

$\textnormal{Regret}(\pi, h, \lambda):= \tilde{V}^{\pi^*}(d_0) - \tilde{V}^{{\pi}}(d_0) + \frac{\gamma}{1-\gamma}E_{(s,a,s')\sim d^\pi}[\lambda(s') ( h(s') - \tilde{V}^{{\pi}}(s')) | s, s'\in\Omega ]$

On the other hand, we totally agree that in practical scenarios with mixed behavior policies, the estimated heuristics in Section 4.2 may lead to a non-stationary MDP. As discussed above, the current analysis would work through if this heuristic (in expectation) turns out to be improvable. We think analyzing the exact condition for this to happen in a scenario of mixed policies is interesting, but it’s beyond the scope of the current work. When the resulting heuristic is not improvable, it eventually will lead to an extra error term in Section 4.2. As a result, the performance of HUBL is affected by how well the heuristic function is calculated, which is affected by many factors like the randomness of the environment as you pointed out, the amount of data, the coverage of the data, and so on. Specifically for the randomness in environments, we provided experiments of HUBL with stochastic environments in Section D.3, where HUBL still provides performance improvement.

Choosing lambda may not be straightforward

When the trajectories perform equally well, HUBL with Rank labeling behaves exactly as you expected: the trajectories will have similar lambda. However, this does not mean that HUBL will necessarily hurt the performance, as there is $\alpha <1$ discounting the amount of heuristics blended into bootstrapping. Empirically, we applied HUBL with Rank labeling on hopper-random-v2, walker2d-random-v2 and halfcheetah-random-v2. The results are provided below. We can see that HUBL is able to improve the performance. However, in this setting, the performance improvement of HUBL is limited. We believe the reason is as you suggested: the $\lambda$ function could not find well-performing trajectories to blend with bootstrapping. A better design for $\lambda$ is required for such scenarios, which is one of our future directions. We will include such discussions in the final version of the submission.

Why is the training of a value function needed in step 2.

We are not quite sure which step you are referring to as we do not train a value function in Step 2 on page 4. We train a value function in Step 1: Computing heuristic $h_t$ on page 4 (footnote 3). That is because sometimes a trajectory ends not due to reaching a terminal stage but due to a timeout. When this happens, the simple Monte-Carlo sum of the rewards underestimates the value of this trajectory. Therefore, we need to estimate a value function to add at the end of such trajectories.

Thank you very much for your positive feedback and pointing out additional references. We have updated the draft addressing the presentation issues and additional citations you mentioned.

Thank you very much for your constructive feedback and positive comment on our draft.

For someone not already familiar with offline RL, it can be challenging to follow the comprehensive theoretical analysis developed in this paper, especially in the appendices.

Thank you for pointing out this presentation issue. We have uploaded an updated draft by adding more intuitions and explanations to the proof in Appendix A and B. Such modifications are highlighted in blue.

This claim should be justified: "Despite their strengths, existing model-free offline RL methods also have a major weakness: they do not perform consistently."

We hope to justify our claim by two types of evidence:

1. Our experiments show that existing model-free offline RL methods show inconsistent performance over different tasks. See Appendix D.2 where provide the performance of each base offline RL algorithm on different tasks. Specifically, with the suggested hyperparameter tuning procedures, such offline RL methods especially underperform in some tasks while performing well in other tasks.
2. Such inconsistent performance can also be found in empirical results of other offline RL papers like Table 2 of [1]. In our experiments, we show that HUBL can mitigate this inconsistency issue and provide significant performance improvement in tasks where the base RL method underperforms. In the updated manuscript, we provide references for this statement.

In the experiments section I miss the learning curves where a reader can compare how the reward evolves with and without HUBL.

We apologize for missing such a plot. As we focus on offline RL problems, the learned policy in each epoch does not need to interact with the environment. Therefore, when implementing the experiments, we only interacted with the environment at the end of the training to collect final performance results without documenting the performance in each epoch.

Following your suggestion, we redid the experiment of HUBL with TD3+BC on hopper-medium-v2, and provided a figure of average normalized sum of rewards over time . Please see Figure 4 on page 23 of the updated submission.

I believe the manuscript needs a brief discussion on how the proposed model would be applied to problems with sparse rewards. For instance, in environments like Antmaze, has it been tested?

Following your suggestion, we conducted additional experiments on applying HUBL with TD3+BC and Rank labeling to antmaze-medium-diverse-v0, antmaze-medium-play-v0, antmaze-umaze-diverse-v0, and antmaze-umaze-v0. The results are provided below. HUBL is able to provide performance improvement to TD3+BC in the considered environments.

Theoretically, we did not pose any assumptions on whether the reward is sparse or not. Potentially, HUBL can be more helpful in sparse-reward environments if given a proper heuristic. The reason is that even if the reward is sparse, the heuristic can be dense and guides decision-making. We defer a thorough analysis of HUBL in sparse-reward environments to future work. We will include such discussions in the final version of the submission.

[1] Tarasov, Denis, et al. "CORL: Research-oriented deep offline reinforcement learning library." arXiv preprint arXiv:2210.07105 (2022).

Thank you for your reply. We are grateful for your thorough reading and constructive comments, especially your suggestions on the presentation of experiment results.

Presentation

We have uploaded an updated draft, where the plots and tables are modified as you suggested. For figures, a horizontal line at 0 is added to each plot, and bars are colored in red and blue, with blue for positive performance improvement and red for negative improvement. For tables, the best performing configuration in each row is highlighted in bold.

Limitations

Although having trajectory data is a prerequisite of the current design of HURL, most real world decision-making data are actually collected as trajectories. Collecting independent transitions is rarely feasible. Thus, while we impose an assumption that might seem strong in the mathematical sense, it is actually weak in practice.

Is there a way to ensure that its enhancements are consistently non-negative?

Indeed, HUBL may occasionally hurt performance empirically on some datasets – this can happen in situations where offline RL algorithms already attain near-optimal results. Such performance decrease is more due to finite sample error, as the room for improvement on these datasets is small – for any offline learning technique. Nonetheless, in most medium-expert datasets where the base RL method already performs well, HUBL still provides positive performance improvement. Overall, the main motivation behind HUBL is to consistently improve the performance of the base algorithm whenever the baseline algorithm performs suboptimally on its own (compared with other offline RL algorithms), and HUBL succeeds at this, sometimes with a relative improvement up to 50%.

It is an interesting direction to design a method that can guarantee non-negative expected improvement. The tricky bit is that different offline RL methods have different characteristics like regret, approximation error, optimization error and so on. Therefore, it is hard to design one $\lambda$ that is rigorously guaranteed to improve expected performance for various offline RL methods without overtunning $\lambda$ for each problem.

On D4RL, HUBL shows significant improvements over baseline algorithms for the hopper task several times. I wonder whether there exist any shared or general characteristics in situations where HUBL offers substantial advantages.

The empirical performance improvement of HUBL is most significant when the base offline RL method shows inconsistent performance and performs suboptimally compared to other offline RL algorithms. This can be seen from the results in Appendix D.2, where we provide the performance of the base RL algorithms in the experiments.