Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

We thank Reviewer Q4z6 for their time and comments. We are glad they found our work to be easy to understand and well-supported with strong empirical evidence. Please find our responses to your questions below, and we kindly ask whether Reviewer Q4z6 would be willing to increase their score or let us know if further clarifications are needed.

Suggested theory paper in tabular MDPs

We thank Reviewer Q4z6 for pointing us to this interesting reference. We were not aware of this work, but we have now included it in our Related Work section.

Other representation learning methods

In principle, representation learning methods other than the CEB can be used within PTGOOD. However, as we highlight in Section 4.2, CEB-learned representations have been shown to work well in out-of-distribution detection tasks, which is a property that is useful for PTGOOD’s purposes.

Request to include environments from the Cal-QL study

We specifically avoid using datasets that contain demonstrations of expert or hand-crafted policies (i.e., policies that successfully complete the task in the case of binary success tasks like Kitchen or optimal policies like in HalfCheetah Expert.) We motivate this setting in our introduction and reiterate it in our Related Work section. Below, we will go through each of the datasets used in the Cal-QL paper’s main results (Figure 6), and highlight that each of these datasets contains trajectories that violate this dataset selection criterion.

All four of the AntMaze datasets used in the Cal-QL paper are collected by expert policies. The D4RL documentation (https://github.com/Farama-Foundation/d4rl/wiki/Tasks#antmaze) describes the trajectories as being generated by “commanding random goal locations in the maze and navigating the ant to them.” Also, the D4RL paper (Fu et al., 2021) states that the AntMaze datasets are generated by first “training a goal reaching policy.”

All three of the Kitchen datasets are filled with trajectories that successfully complete the task. The documentation (https://github.com/Farama-Foundation/d4rl/wiki/Tasks#frankakitchen) states that the datasets “includes demonstrations of all 4 target subtasks being completed” and the D4RL paper describes the Kitchen datasets as coming from “human demonstrations” that are either “performing all of the desired tasks” or “performing subtasks.”

The Pen, Door, and Relocate datasets are also filled with trajectories that successfully complete the task. The Cal-QL paper describes these datasets with: “Each of these tasks only provides a narrow offline dataset consisting of 25 demonstrations collected via human teleoperation and additional trajectories collected by a BC policy.” Additionally, the D4RL paper describes these datasets with: “This domain was selected to measure the effect of a narrow expert data distributions and human demonstrations.”

The Cal-QL authors do not provide the dataset used in the Visual Manipulation environment, but they do refer to its datasets as containing “narrow” behavior, which is verbiage often used to describe datasets collected by expert policies. These datasets appear to come from Ebert et al., (2021), who describe the datasets as being collected via teleoperation specifically for imitation learning purposes, which suggests that the trajectories within the dataset successfully complete the tasks.

We thank Reviewer 3FpJ for their time and comments. We are glad they found our paper to be well-written and easy to follow and that our work provides an interesting perspective on the offline-to-online problem. We have added additional baselines that Reviewer 3FpJ requested, which can be seen in our newly uploaded PDF. Please find our responses to your questions below, and we kindly ask whether Reviewer 3FpJ would be willing to increase their score or let us know if further clarifications are needed.

Request for more environments from D4RL

We specifically do not use all of the environment-dataset combinations in the D4RL suite because many of them violate our selection criteria (explained in section 6.2.) Many of the environment-dataset combinations in D4RL can be solved to (near-) optimality during the offline pretraining phase. Such environment-dataset combinations are uninteresting in the OtO setting and violate our criterion 1, which is defined in Section 6.2. On the other hand, some of the environment-dataset combinations provide such poor coverage of the MDP (e.g., the Random dataset in D4RL’s Walker2d) that offline pretraining provides virtually no benefit (violating criterion 2 which is defined in Section 6.2.) As a result, we collect several of our own datasets in environments that are not represented in D4RL, such as Ant, Humanoid, and DMC Walker, for a total of 7 environment-dataset combinations. These datasets comprise a portion of our work’s contribution to the research community, and we will release them to the public in our work’s code repository.

Request for more baselines

We have added two of the requested baselines (PROTO and PEX) to the main results in Table 2, Figure 11, and Figure 12. Two of the other requested baselines (E2O and SUNG) do not have code provided by the authors. The final baseline (AWAC) is outperformed by the two baselines we have already added, and therefore it would likely not provide additional information. We highlight that PTGOOD, our contribution, significantly outperforms both PROTO and PEX in all of our benchmarks.

Cal-QL performance

The environments used in the Cal-QL paper (Figure 6) contain trajectories of expert policies. This property is useful for Cal-QL since it contains a learning mechanism that keeps the learned policy close to the policy that collected the datasets (Equation 3.1 in the Cal-QL paper.) In contrast, we specifically avoid datasets that show expert behavior and instead only choose datasets collected by highly-suboptimal policies. We outline our dataset selection criteria in Section 6.2. As we explain in paragraph four of section 6.3, Cal-QL’s poor performance in these datasets should not be surprising, since it is actively keeping its learned policy close to the highly-suboptimal dataset-collection policy. The same should be true for any of the other algorithms that use the “conservative” and/or “pessimistic” mechanisms.

Also, it is worth noting that the Cal-QL paper records agent returns over one million or more online fine-tuning steps. In contrast, our work records agent performance over only 50k timesteps, which represents a small fraction of the x-axis of the plots in Figure 6 of the Cal-QL paper.

Traditional exploration methods

We display in our experiment section (e.g., Table 2) that traditional exploration methods, such as intrinsic rewards and UCB exploration, work fine, but not as well as PTGOOD. Also, we perform hyperparameter tuning for each one of these baselines, as seen in Figures 3-7. We also show that following the policy without any exploration guidance (Naive) works fine but also significantly underperforms PTGOOD.

Algorithm used during online fine-tuning phase

All of our baselines (except for Cal-QL, PROTO, and PEX) and PTGOOD use model-based policy optimization (MBPO) and Soft-Actor critic (SAC), as we mention in the first paragraph of Section 6.1. Combining SAC with MBPO is standard practice and works well, as shown in the original MBPO paper (Janner et al., 2019).

We thank Reviewer Ftkq for their time and comments. We are glad they found our work to be novel. Please find our responses to your questions below, and we kindly ask whether Reviewer Ftkq would be willing to increase their score based on our responses or let us know if further clarifications are needed. Also, please see our updated PDF to see additional results we have added during the rebuttal period.

Question about finding relatively-high reward regions

We highlight in Section 6.4 that the relatively high-reward regions are found by keeping exploration “close” to the improving policy. Due to the agent’s learning process (via RL), it should visit increasingly higher-reward regions of the MDP’s state-action space naturally. As we show in the experiment in Section 6.4 and Figure 2, keeping the exploration process “close” to this improving policy is critical. This is shown by the large decrease in returns throughout training when the planning noise is too high.

Question about the relation between dynamics modeling accuracy and agent performance

This is an interesting question. Ideally, the learned dynamics model would be an oracle over the MDP’s entire state-action space (both in terms of reward- and transition-prediction.) However, our paper’s focus is on the limited-interaction setting. Therefore, we cannot afford to have the agent explore the MDP exhaustively, which would likely be required to achieve an oracle model. In our case, it is not so clear over which regions of the MDP’s state-action space we should measure the dynamics model’s accuracy and other performance metrics.

Also, some recent works (e.g., Lambert et al., 2020; Wei et al., 2023), have shown that dynamics model prediction accuracy is not well correlated with agent performance. This is often called the “objective mismatch” between dynamics modeling and policy optimization of rewards. Because of this mismatch, it would probably be difficult to draw meaningful conclusions by observing modeling accuracy.

Explanation of the low variance in Figure 1

Could Reviewer Ftkq please clarify their question? In Figure 1, sometimes the variance across seeds is small when lambda is small, but sometimes the variance across seeds is large when lambda is small. The same is also true for relatively large values of lambda. So we are unsure what the reviewer is referring to specifically.