D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning

The proposed benchmark does not include much more diverse domains compared to D4RL. I think with some past works such as RoboMimic [1] and some recent works such as RL-ViGen [2], similar and even more diverse benchmarks could be easily made, by running RL agents to collect offline datasets.

We believe our tasks are significantly more diverse than RoboMimic. In particular:

1. Our environments use a number of different objects for the manipulation setting
2. We use two different types of robot arms - a Franka Emika and a WidowX, as well as an A1 quadruped robot.
3. The tasks we consider are more complex and involve achieving multiple objectives (i.e. arranging multiple objects in a scene).
4. Furthermore, our tasks have a number of continuous distribution shifts, such as scene, texture and camera randomization.
5. Moreover we have collected over 20 hours of realistic human manipulation data beyond pure expert demonstrations, such as play, failures, and sub-optimal execution, which was not analyzed by RoboMimic.

Thank you for bringing RL-ViGen to our attention. We view this work as complementary to ours, since our manipulation environments do not overlap.

1. We aimed to create a lightweight and accessible benchmark for data-centric RL, which does not involve heavy simulators, such as CARLA and Habitat.
2. Our proposed environments have naturally occurring distribution shifts, such as diverse objects, visuals, lighting and camera angles, while the ones in RL-ViGen are rather artificial.
3. We also aimed to utilize realistic human-collected datasets, rather than replay-buffer based ones.

Possibly unfeasible settings. As shown by evaluation results, most SOTA algorithms are not working well on this benchmark. This could possibly be attributed to the difficulties of the task, but from my perspective, there could also be some key factors that are not right, such as the problem formulation and the framework of offline RL. (This point might be not significant, but it is my personal concern about the benchmark.)

We have included improved results on some of the tasks, achieved by a more extensive hyper-parameter search (see table below). The fact that SoTA RL methods achieve good performance on other benchmarks (such as D4RL) but do not achieve good performance on our benchmark is echoed by results on many real world robot manipulation tasks, in which behavior-cloning based methods tend to out-perform current RL methods. We hope that our benchmark can drive the development of RL methods which are more successful in real robot applications. For example, a recent model-based RL method [1] is able to fully solve the Standard Franka Kitchen from images with online fine-tuning.

[1] MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning Rafael Rafailov, et. al, Conference on Robot Learning, 2023

Overclaim of the paper title. This work names the newly proposed benchmark as D5RL, which is obviously following the name of D4RL. However, this could be misleading for the entire community to keep pursuing the limited improvements on a new but essentially similar benchmark.

We believe there are substantial differences between d4rl and our proposal. The main ones being:

1. Focusing on realistic robotic tasks versus simple simulated embodiments.
2. Learning from realistic observation spaces rather than ground-truth states.
3. We introduced a significant amount of diversity in terms of objects to manipulate (which require different grasps) and visual environments.
4. We introduced a number of new realistic human-collected datasets, such as undirected manipulation and sub-optimal execution data, rather than using RL replay buffers.

Limited number of tasks for the A1 legged locomotion. Tasks such as jumping, obstacle avoidance, etc. can also be of interest in this. Here we present an initial version of our benchmark. If there is sufficient interest in the community, it is possible to add additional locomotion tasks, including more complex embodiments in the future.

No evaluation of any of these tasks/settings in the real world. To what extent can the models trained on these datasets transfer to the real-world? While we design our benchmark to reflect challenges of real world robot learning, policies trained in simulation could face challenges in real world deployment due to environment mismatch and the sim-to-real gap. The primary goal of our benchmark is not to test sim to real transfer, but rather to provide a fast and easy to use test-bed for the new offline RL algorithms, which can speed up the development and iteration cycle.

My impression from Tables 1 and 2 is the proposed benchmark is too difficult for existing offline RL methods to achieve meaningful performances, since most methods except BC and IQL have near-zero returns. While having a challenging benchmark is good, a too-challenging benchmark will not provide much signal to improve the performance of current methods.

Thank you for your suggestion. We have added some updated results to improve the performance on the WidowX task (see Table below), one of these challenging tasks that are referred to. To increase task performance, we refactored the environment by modifying the layout of the environment, the tractability of grasping objects in the scene and collecting longer demos:

• Augmenting the layout of the environment so that there is more room to grasp the objects found in the center of the scene

• Reducing the range of the rotation of objects and the size of objects in the scene to increase grasp success (as the simulated robot doesn’t have a rotation in gripper space)

• Collecting for a longer horizon to ensure that the full sorting procedure occurs for the scripted sorting policy for each episode collected.

We have refactored the tasks to be (1) sorting, where the agent has to sort two objects into their respective bins and (2) sorting bin noise, where the agent still has to sort the two objects but there is a probability that each object is placed into the wrong bin. Along with this modified task, we have included the performance of a random policy on our tasks, which achieves a 0% success rate across all tasks.

The paper is currently lacking ablation studies to understand what components of the benchmark are challenging and/or why current algorithms fail.

While we do not have ablation studies of the current RL methods, since our goal is not to develop better RL methods but rather to provide a benchmark, the benchmark tasks themselves are already designed to provide different difficulty tiers, and therefore can help researchers understand where their methods work better or worse. This was already discussed to some degree in Section 4, but we have since expanded that discussion. Specifically, the Standard and Randomized Kitchen environments evaluate the ability of methods to learn to compose multiple manipulation tasks over a long time horizon. The Randomized Kitchen environment additionally tests the visual robustness of methods and their ability to generalize to variations in object shape and location. The play dataset of the Randomized kitchen tests the ability of methods to learn from sub-optimal (non-demonstration quality) data.

I would like to see a random policy as a simple baseline. I wonder if the benchmarked methods even do better than a random policy.

Random policies achieve zero success on our manipulation environments. We have added the result to the experiment table.

Why is there such a big gap between IQL and other similar methods such as CQL and TD3+BC? These methods often perform very similarly in D4RL, at least in the locomotion tasks.

The fact that value-based RL methods perform worse than behavior cloning-based methods on our benchmark is an important insight/contribution, not a disadvantage. In many real robot manipulation tasks, behavior cloning-based methods tend to outperform offline value-based-RL methods, too. While value-based RL methods have achieved SoTA performance on simpler benchmarks, such as D4RL, we hope that our benchmark can help drive the development of offline RL methods that perform well on more realistic robot manipulation tasks

Section 5.2 states that the goal of the Frank Kitchen environment is to study offline RL and online fine-tuning, but the following text does not mention how the proposed task facilitates online finetuning.

As far as we are aware no current method can solve the environment fully with only the provided offline data. Because of this, additional online fine-tuning is a necessary step to be able to learn to fully solve it. We hope that our benchmark can provide a good way to evaluate methods for offline-to-online fine-tuning, where the available offline data is insufficient to fully solve a task, but can provide a good starting off point for initializing a policy for online learning. We carry online fine-tuning experiments on a number of our environments, results are shown in Table 2.

Some of these challenges could be already observed in previous benchmarks. Evaluating temporal compositionality seems possible using previous benchmarks (D4RL Maze2D, AntMaze, FrankaKitchen-Mixed, Calvin).

We agree that these environments test temporal compositionality to some degree. However, performance on the D4RL maze tasks is becoming saturated. While the existing Franka and Calvin environments still pose a challenge, they do not test other scenarios, such as the multimodality of the data distribution (Franka kitchen is composed entirely of demonstration data, and CALVIN is composed entirely of play data). In our “play” datasets tasks can be executed partially and in arbitrary order and the agent needs to learn how to compose these together, which proves to be quite challenging.

The proposed benchmark has less diversity than D4RL, and different levels of data distribution are also discussed in D4RL.

In this proposed benchmark we focus on realistic robotics tasks. While it is true that D4RL considered different levels of data quality, it did so by using different parts of an online RL replay buffer. Instead in this work we focus on more realistic data distributions, such as sub-optimal demonstrations and play data.

The text does not explicitly claim why using this new benchmark is better than using previous benchmarks for each challenge. To encourage using this benchmark, the paper should address questions like: when to choose this benchmark? Are previous works successful on prior benchmarks likely to fail on this new benchmark specifically due to one of specific challenges?

We cover the difference between d5rl and other benchmarks in our Related Works section. In particular we believe it addresses the following challenges:

1. Uses models of real robot systems.
2. Uses realistic datasets, such as demonstrations with varying quality provided by haman tele-operators, play data and scripted planners.
3. Addresses realistic challenges, such as learning from high-dimensional observations and robustness to distribution shifts.
4. Compared to other works we believe it strikes a better balance addressing the above challenges and time/complexity/computational requirements.

Addressing offline-to-online learning adaption appears to be incremental. As almost all previous offline RL benchmarks are based on simulation environment and rule-based reward function, implementing online fine-tuning on the top of exisiting benchmarks is feasible.

We do not claim that our work proposes or formalizes the online fine-tuning problem. Instead we aim to demonstrate that our environments and datasets are suitable for that application.

More information or resource should be provided to challenge the community to solve the image observation version of Franka Kitchen. Is the failure of existing works attributed to the incompleteness of offline RL algorithms or the lack of image representation learning? Can this task be intuitively completed solely with the provided offline data? If not, is there an alternative data source that the community can leverage to address this challenge?

The kitchen environments are quite challenging due to a variety of factors. As far as we are aware at the time of submission no offline RL or online fine-tuning approach had fully solved even the fully observable low-dimensional state versions. However, the standard (non-randomized) environments are feasible to solve, since a recent model-based work provided a complete solution [1].

Even if the benchmark doesn't specifically address one of those challenges, it can still be beneficial to have a domain with a collection of challenges if it is important to solve the domain. However, the provided domains are not significantly more realistic or different compared to existing tasks, making them less of an important target domain that the community aims to solve.

We believe that the challenges posed by the proposed environments are significant and indicative of challenges of real world robot learning. In particular issues such as visual diversity and robustness remain quite challenging for real world robot systems.

[1] MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning Rafael Rafailov, et. al, Conference on Robot Learning, 2023