Thank you for your recognition of our extensive empirical studies and constructive criticism of our work.

Weaknesses

It would be beneficial to include results on additional datasets like those in neorl/neorl2 or other diverse domains.

We believe that our robotics benchmarks already capture significant diversity: Across 7 environments and 21 datasets, we include locomotion, manipulation and maze solving. We also evaluate our methods on all datasets of the three most common mujoco environments. We also agree that neorl2 is an interesting benchmark and are currently looking into adding it for the camera-ready version.

The taxonomy omits a class of techniques that tune offline RL algorithms based on Q-values without requiring online interaction.

These methods fall under our taxonomie's category (1), since (as the reviewer noted) they use no online interaction before deployment. We will cite the reviewer's suggested reference in the next iteration of our paper. Furthermore, we already cite two references that estimate policy performance offline (line 317) and elaborate in more detail in section L.1 of the Extended Related Work in the Appendix.

Minor: The notation in Equation 1 is slightly inconsistent.

We thank the reviewer for their keen eye and have fixed this inconsistency.

Several recent offline RL methods, such as Sparse Q-Learning (SQL) and Exponential Q-Learning (EQL) [2], as well as A2PR [3], are not discussed or compared.

We are citing 21 RL methods and implementing 10 algorithms as well as 2 novel algorithms. This includes all offline-only algorithms in (Jax-)CORL and almost all algorithms in OfflineRLKit. We aimed to provide one of the most comprehensive offline RL libraries that currently exist, and argue that we have succeeded. Still, we are open to implementing the suggested algorithms and are planning to add them for the camera-ready version. Our work is also intended to be a scaffolding for the community to seemlessly add concise implementation of different methods.

Answers to Questions

Could $the authors$ elaborate on why their implementation outperforms JAX-CORL?

Profiling the runtime of jax-corl was outside the scope of our work. From looking at their codebase, it seems like they are not using lax-based (GPU-compilable) control flow, which can lead to significant performance decreases

In Taxonomy point 2b, if only one policy is eventually deployed, how is the selection carried out based solely on evaluation?

This depends on the specific offline RL method. For example, Active Offline Policy Selection (Konyushkova et al., 2022, https://arxiv.org/abs/2106.10251XiV) uses Bayesian optimization over a Gaussian process of policy values. We elaborate in Appendix L.1 (line 960).

Which hyperparameter selections are the authors targetting through online finetuning? What is the range of these hyperparamters?

When selecting policies via a bandit, we select between policies trained with different hyperparameter settings. We use the hyperparameter ranges from the respective methods' original papers, as described in line 175. Our algorithm's hyperparameter ranges can be found in Appendix H.

How many iterations does the bandit algorithm require for convergence in practice? Can convergence be quantified or benchmarked?

This is highly dependent on the HP robustness of the evaluated algorithm. For example, if the algorithm produces high-performing policies for a wide range of hyperparameter settings, convergence is fast.

Why is the Decision Transformer (DT) not included in Figure 3?

We include it in the camera-ready version and present results below. We did not include it in the initial set of algorithms as the Decision Transformer is not a traditional RL algorithm, even though it is technically trained offline on the same datasets. Furthermore, as done in CORL [1] and the original DT paper [2], the sequential nature of the evaluation function is different from the other algorithms. We will report the results from the same DT evaluation setup used by CORL [1].

Preliminary DT results

We ran our implementation of the DT algorithm for three seeds each.

Also how does MoBRAC handle the overfiiting observed by model based RL since it uses MOPO to learn the world dynamics.

MoBRAC uses the trained MOPO models as they are. We propose the algorithm as an improvement over current approaches and demonstrate it by intentionally using the same models. MoBRAC is an ablation in itself, showcasing that behavior regularization leads to agents that are more robust, despite being trained using samples from somewhat flawed dynamics models.

Why does MoBRAC exhibit such high variance in the AntMaze task (Fig. 15)?

The variance is very low in practice, but might appear high due to the extremely small scale of the y axis. Most likely, a few policies have recieved a reward by sheer luck, hence the performance is ever-so-slightly above zero.

How sensitive are the results to random seeds or initialization?

We did not explicitly investigate this. However, every hyperparameter sample also has a different randomly sampled seed, so our metric evaluates an even stronger form of robustness (which includes hyperparameters).

[1] Tarasov et al. CORL: Research-oriented Deep Offline Reinforcement Learning Library (2022)

[2] Chet et al. Decision Transformer: Reinforcement Learning via Sequence Modeling (2021)

Thank you for your valuable feedback. Given our responses and clarifications to your questions and weaknesses, we kindly ask you to consider increasing your score.

We thank the reviewer appreciating both our novel algorithms, the implementation and taxonomy.

Weaknesses

The impact of individual Unifloral components (e.g., different critic objective terms, ensemble sizes, entropy weights) on algorithm performancne is not clear in this paper. I suggest the authors make systematical analysis on these components which may bring more insights.

We believe that the strong empirical performance of our algorithms demonstrates a clear benefit of combining components that have been independently proposed. Additionally, the algorithms Unifloral is based on represent ablations of Unifloral itself. For example, ReBRAC is an ablation of TD3-AWR (no Adwantage Weighted Regression), and IQL is an ablation of TD3-AWR (basic BC loss instead of TD3-BC value loss). Therefore figure 6 and 7 explicitly analyse the impact of individual Unifloral components.

It lacks analysis of the hyperparameter space itself. It would be great to have visualizations of the hyperparameter landscape, e.g., sensitivity heatmaps to show the component interactions.

We decided to focus our limited experiment budget on a more comprehensive and complete comparison of methods. Analyses of hyperparameter landscapes are not standard in the field, and we are not aware that any paper we cite or implement visualises sensitivity heatmaps. Therefore, we respectfully disagree that this is a weakness of the paper.

Questions

I don't quite understand the difference between pre-deployment and post-deployment Offline-RL

The difference is if returns achieved are measured during or after training, i.e. before (as in 2a) or after (as in 2b) deployment.

When tuning hyperparameters, one may only be interested in the performance after tuning. Evaluating policies during tuning is pre-deployment interaction, before the final policy is deployed and evaluated.

When training a recommender system, one deploys after offline training, and starts evaluating. After deployment, one finetunes the system using live user data, requiring post-deployment adaptation.

We thank reviewer rtSN for their questions and criticism of our paper, and we would like to clarify on their remarks.

Weaknesses

After reading the paper and re-checking the sentences, I am still a bit confused about the difference between pre-deployment interaction and post-deployment adaptation.

Recall that we use deployment as the point in time at which we stop training and start evaluating our agent. In offline RL, an agent has no access to the environment before deployment. However, some methods relax this strict separation, allowing one (or both) of:

• pre-deployment interaction, in which the agent may interact with the environment prior to deployment -- for instance, to select one of several policies to deploy based on online performance. Hyperparameter tuning needs pre-deployment interaction, since policies' returns are used to sample new hyperparameters to test, but the final performance metric is the performance after tuning, i.e. after deployment.

• post-deployment adaptation allows the agent to continue learning after deployment, and the performance metric includes all returns collected after deployment. For example, when training a recommender system offline and finetuning it with live user data.

We believe that this clarification should address most of your doubts, and have improved the wording in the paper accordingly (including removal of the overloaded word "cost" mentioned in line 106 of the paper).

As to the taxonomy, $...$ I have no idea about what the exact principle to separate the two categories $(2a) and (2b)$ .

The answer above should make the difference between categories (2a) and (2b) of the taxonomy clear: (2a) allows for pre-deployment interaction only; (2b) allows for post-deployment adaptation only. The difference is if returns achieved are measured during or after training, i.e. before or after deployment.

Is the "Limited pre-deployment interaction" necessary for Offline-to-Offline RL?

To answer your question, could you elaborate on what you mean by offline-to-offline RL?

I found the "fixed sampling range of each hyperparameter" in "A Definition of Offline RL Methods" somewhat problematic, as the method that uses different ranges can be converted to use a large and fixed range, which seems to fit this definition well.

We agree that specifying different hyperparameter ranges for different environments, especially without justification, is problematic. In fact, we explicitly criticise this in line 123 of our paper. Instead, the hyperparameters are part of a method, which should not be environment-specific. For fair comparison between methods, we do convert methods that specify multiple ranges into one that specifies one range by taking the ranges' union (line 175).

For the evaluation and experiments, $...$ zero-shot evaluation is still an important category.

We note that you can easily compare zero-shot performance by looking at the first datapoint in each of our evaluation plots, as it represents performance after 0 episodes of pre-deployment policy selection.

episodes can have different lengths at least in general. This does not seem to be a proper measure in this sense.

We argue that both episodes and steps are valid units for budget, both with advantages and drawbacks:

• Performance in RL is usually measured as episodic return, which makes episodes a natural fit.
• In practice the cost for evaluating robotics applications is usually dominated by resetting the environment (think - placing objects or the robot itself back to their initial positions), which is done once per episode.
• When setting a budget in steps, one might observe a different number of episodes between policies, and has to compare performances with different confidence levels.

Answers to Questions

How the authors think about the empirical evaluation for zero-shot offline RL (and Offline RL with post-deployment online policy selection)?

As mentioned above, zero-shot evaluation is a special case of our protocol, namely zero pre-deployment interaction episodes. We believe that our method can be extended to offline RL with post-deployment by modifying the policy selection via a bandit, but this was beyond the scope of our paper.

Will decision transformer and its follow-up methods be scheduled to add to the code base?

We have already implemented the decision transformer and present some preliminary results as D4RL normalized scores on three seeds each:

Thank you for your valuable feedback. Given our response and clarifications to your questions and concerns, as well as the new preliminary results, we kindly ask you to consider increasing your score.

Thank you for appreciating our work's contribution, the thoroughness of our evaluations and the value of our codebase. We address your questions and concerns below.

Weaknesses

The proposed evaluation procedure uses random hyperparameter sampling followed by UCB over evaluation rollouts. Why was random sampling chosen over the standard grid search?

For fair comparison via our bandit protocol, the number of policies trained with each algorithm should be constant. When using grid search however, the number of policies trained would be equal to the number of hyperparameter combinations. Thus, it is necessary to subsample the set of hyperparameter combinations, which we do with random search to remain unbiased.

Aren't we ultimately interested in the score these curves converge to? While it’s true that in practice we may be limited in the number of evaluations, doesn't the evaluation curve’s plateau reflects the best estimate of a method's performance?

Performance given a limited budget is of very high practical and theoretical significance. For example, zero-shot performance is a practically relevant metric, and siginifcant efforts have been made to maximize zero-shot performance based on offline analysis of trained policies (see e.g. the works referenced in our Related Work section, line 317). Our evaluation method gives the flexibility to compare methods at different budgets, including zero-shot and final, i.e. plateaued, performance. Thus it is more informative, and measures a method's performance in a more general sense.

$...$ we need to be sure that the range of hyperparameters we use reflects a "reasonable" range $1$ -- which is admittedly not well-defined. What happens during evaluation if we set the hyperparameters range to something obviously too wide? This... would ultimately lower the score evaluation converges to after collecting a large number of trajectories, right?

That is correct - methods with wide hyperparameter ranges, that require a large tuning budget to perform well, will perform poorly under our metric. Our evaluation was designed precisely to demonstrate this point, as prior work considers only the maximum performance within the hyperparameter range, disregarding tuning cost. It is also for this reason that we define offline RL methods (Section 3.1) as including the hyperparameter range, so an improved range over the same hyperparameters is viewed as a superior method.

Minor Comments

Thus, the state-of-the-art remains ambiguous, with no method demonstrating uniformly strong performance across all datasets.

This phrasing suggest that if we evaluated algorithms fairly, one SOTA algorithm would rise to the top. But it's common for some methods to perform well on some tasks but not others.

We stand by our assessment that SOTA remains ambiguous, since no algorithm has risen to the top. The existance of a single SOTA algorithm is not to be taken for granted, as you have pointed out.

Line 186: Missing period/colon after bold header

We have fixed that for the camera ready, thank you for pointing it out.

Questions

1. Why is there no discounting in equation 1?

We consider finite-horizon MDPs, which do not require discounted return in their mathematical formulation as the episodic return is finite. Despite this, many methods still include a discount factor to aid credit assignment, but this is not theoretically necessary.

2. The difference between post

We are making an educated guess that this refers to the post and pre-deployment setting; we are happy to clarify further if you were referring to something else.

We use deployment as the point in time at which we stop training and start evaluating our agent. In offline RL, an agent has no access to the environment before deployment. However, some methods relax this strict separation, allowing one (or both) of:

• pre-deployment interaction, in which the agent may interact with the environment prior to deployment -- for instance, to select one of several policies to deploy based on online performance. Hyperparameter tuning needs pre-deployment interaction, since policies' returns are used to sample new hyperparameters to test, but the final performance metric is the performance after tuning, i.e. after deployment.

• post-deployment adaptation allows the agent to continue learning after deployment, and the performance metric includes all returns collected after deployment. For example, when training a recommender system offline and finetuning it with live user data.

We believe that this clarification addresses most of the reviewers' doubts, and have improved the wording in the paper accordingly (including the usage of the overloaded word "cost" mentioned in line 106 of the paper).

3. Why specifically do the authors recommend those particular datasets (line 164)? It seems like the argument is simply "more is better"

We select a fixed set of datasets to make future evaluations consistent and immune to cherry-picking. Of these, the individual datasets were selected to provide coverage over the maximum number of environments and behavior policies, as well as focusing only on datasets where existing methods do not achieve trivial performance (contantly nil or perfect), providing meaningful signal for future algorithm comparison. We note that for halfcheetah and walker datasets, the bandit evaluations converged to a clear final ranking that is different from other intermediate points throughout the bandit's score trajectory. This further demonstrates the additional insight our proposed evaluation procedure provides.

4. My interpretation is that "distractor policies" are just policies with higher variance returns (perhaps because they have not yet converged to a nearly deterministic policy)?

You're correct that distractor policies have higher variance, however, it is typically assumed that this is due to an increased number of low-performance episodes. Instead, we show that this is also due to a higher maximum performance, which consistently high-performing policies fail to achieve. This is a presents challenges for bandit sampling, causing bandits to initially prefer distractor policies, as shown in Appendix D.

5. Could the authors include these hyperparameter ranges in the appendix? Given that there are many algorithms evaluated, it would be a bit tedious for a reader to track down these values across many papers. This would also help answer one of my questions posed in weaknesses section.

We use ranges from each algorithms's original paper, as described in line 175. Our algorithm's hyperparameter ranges can be found in appendix H. This presents in the unified space of hyperparameters, allowing for direct comparison of hyperparameters across methods, a feature missing from previous work.

6. I'm a bit confused; how does pre-deployment interaction carry no cost? And what does it mean for performance to "include" the return during addition interaction?

The performance of an agent before deployment does not affect the final evaluation metric. For example, you might test a few policies on a robot, while only measuring the performance of the best one (which you deploy). We have changed wording in the camera ready version to correctly use the word "cost" in the context of rolling out episodes, rephrasing the relevant sentence.