Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations

We thank reviewer awMg for the thoughtful comments. We are grateful to see that the reviewer finds our work novel and finds our real-life experiments and analyses as strengths. Below we address their comments and questions and we hope they can reconsider our score:

1. Concern on having a more general approach for O2ORL rather than robotic grasping

We thank reviewer awMg for raising this concern. We agree that our method does not explicitly leverage properties of robotic grasping, e.g., invariance, equivariance, or some other inductive biases. Our work is inspired by the real-life challenges encountered in robotic manipulation—we address the question of why there is so limited real-life robotic research done with reinforcement learning rather than imitation learning. Real-life robotic manipulation tasks have very unique properties that other common RL benchmarks fail to capture, e.g., sparse reward and challenging exploration in continuous action space. As shown in D5RL (Rafailov et al., 2024), the image-based tasks are very challenging and have not been explored widely in the RL literature. We also appreciate that reviewer awMg has pointed out that our approach is agnostic to robotics—we believe that a more general approach than imposing domain-specific knowledge enables better applicability and adoption.

2. More benchmarks to show effectiveness.

This is a fair concern. To address reviewer awMg’s concern on a wider range of tasks, we conduct offline RL experiments in robomimic (Table 1), also suggested by reviewer cV3D. Based on the simulated results we can observe that Simplified Q is better than CQL on majority of the tasks, but similar to our original findings none of the offline RL algorithms is competitive against BC.

3. Use of vacuum to grasp being relatively simple

We agree with reviewer awMg that using parallel grippers will be more challenging. However, our item can easily crumble and deform, making it difficult to create a vacuum seal at times—the agent will need to identify smoother areas to increase the chance of a vacuum seal. We further demonstrate that four other existing algorithms fail to successfully grasp the items (Figures 2 and 3), perhaps this suggests that the problem is not as easy as it seems. Finally, we hope our simulation experiments in the previous response address this concern as well.

References:

Rafael Rafailov, Kyle Beltran Hatch, Anikait Singh, Aviral Kumar, Laura Smith, Ilya Kostrikov, Philippe Hansen-Estruch, Victor Kolev, Philip J. Ball, Jiajun Wu, Sergey Levine, and Chelsea Finn. D5RL: Diverse datasets for data-driven deep reinforcement learning. Reinforcement Learning Journal, 5:2178–2197, 2024.

Dear Reviewer awMg,

We thank you for the time spent reviewing our submission.

As the main discussion phase is ending soon, we wanted to send this gentle reminder. We have done our best to answer the comments you raised, as well as incorporate your suggestions. We would love to hear back from the reviewer and whether we have addressed their concerns.

References:

Younggyo Seo, Jafar Uruc¸, and Stephen James. Continuous control with coarse-to-fine reinforcement learning. In Conference on Robot Learning (CoRL), 2024.

Tim Hertweck, Martin Riedmiller, Michael Bloesch, Jost Tobias Springenberg, Noah Siegel, Markus Wulfmeier, Roland Hafner, and Nicolas Heess. Simple sensor intentions for exploration. arXiv preprint arXiv:2005.07541, 2020.

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In The Tenth International Conference on Learning Representations (ICLR), 2022.

Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 60247–60277, 2023.

We thank reviewer L1Gj for the detailed feedback. We are grateful to see that reviewer L1Gj finds our work to be well motivated and well presented. Below we address their comments and questions and we hope they can reconsider our score:

Not meaningfully build on prior work.

While we agree with reviewer L1Gj that there are image-based manipulation tasks in previous work, we are not aware of any O2ORL that uses purely RL objectives (i.e. no BC regularization term). We also agree that DrQ and follow-up work have worked on MuJoCo continuous-control tasks with success. However, these environments are often in the dense-reward setting. Existing image-based RL algorithms that appear to work in the sparse-reward setting, as far as we know, are ones included in our related work such as SAC-X (Hertweck et al., 2020) and CQN (Seo et al., 2024)---the latter also uses BC regularizer and appears to fail without it. We are happy to include any extra references if reviewer L1Gj knows of any new result.

The absolute performance numbers relative to the BC baseline seem quite low.

If we understand correctly, we believe reviewer L1Gj refers to offline RL performing worse than BC baseline. We agree with this observation, but we note that in the online phase, the Simplified Q eventually outperforms BC with half of the total data (50 demonstration episodes + 200 interaction episodes vs. 500 demonstration episodes). This result is consequential as we no longer require human teleoperators to gather a large amount of demonstrations, which can be costly in both time and resources.

The Ma et al 2023 paper uses an L1 loss… Not very convinced on novelty or effectiveness and single hard-to-reproduce robot environment.

We thank reviewer L1Gj for pointing this out, indeed we had made a mistake on referring to this paper in the statement. We have removed the reference in the updated manuscript. We highlight that both existing works are different to Simplified Q and still leverage the target network. Notably, DR3 uses consecutive state-action pairs as its implicit regularization is motivated differently from ours. Ma et al. (2023) introduces an extra OOD policy, which increases the number of parameters and hyperparameters. Our goal instead is to minimize Q-divergence and further simplify learning the Q-function through reducing previously mentioned quantities. We have added a paragraph, lines 198-203, in our updated manuscript to describe the differences.

Our novelty lies in identifying that the Q-function learning dynamics with gradient descent can result in a simplification of Q-learning. Previous work ignores this possibility and introduces extra components by having an extra policy (Ma et al., 2023) or maintaining more copies of model parameters (Tang and Berseth, 2024). Our approach reduces the number of model parameters and the number of hyperparameters related the target network into a single scalar which is also easy to tune (see Table 2c). This enables practitioners to run the algorithm in real-life applications due to less hyperparameters to tune and potentially lessen the compute burden. As far as we know we are also one of the first to successfully perform O2O RL on real-life image-based manipulation, with purely RL objective without any BC regularizers, e.g., TD3-BC (Fujimoto and Gu, 2021) and CQN (Seo et al., 2024). Once again, we would be happy if reviewer L1Gj is aware of existing work that has already accomplished this.

Regarding effectiveness, we thank the reviewer for pointing out the lack of experiments. We have increased the number of runs and included additional baselines to demonstrate the effectiveness of our approach in the real-life grasping task (Figures 3 and 4). Specifically we include two more algorithms, DR3 (Kumar et al., 2022) and SAC + LayerNorm (Yue et al., 2023), with three seeds per algorithm. The results remain consistent with our original claim although DR3 appears to perform better than other RL baselines except for Simplified Q. Finally, we conduct offline RL experiments in robomimic (Table 1), as suggested by reviewer cV3D and reviewer awMg. Based on the simulated results we can observe that Simplified Q is better than CQL on majority of the tasks, but similar to our original findings none of the offline RL algorithms is competitive against BC. We hope these extra results address the reviewer’s concern on the effectiveness as we add more statistical analyses and on more tasks.

Could the authors explain why they remove the target network when including the NTK regularizer?

Thank you for asking this question. The target network is initially introduced to mitigate the temporal correlation between consecutive state-action pairs, which is already decorrelated using our regularizer. As a result, including the target network shows no obvious benefits anymore. The main benefits are reducing the amount of compute resources and ease of choosing hyperparameters.

We thank reviewer cV3D for the thoughtful feedback. We are grateful to see that reviewer cV3D finds our work and results are presented and formulated in a sound manner. We also appreciate that they recognize the importance of real-life robot applications. Below we address their comments and questions and we hope they can reconsider our score:

Weaknesses

The clarity of the paper can be improved in terms of readability and crispness of contributing statements.

We thank reviewer cV3D for the feedback on the clarity of our results. We have updated the caption of (previously Figure 3, now Figure 5) to specify the different image encoders. Specifically, we compare (1) trained end-to-end, (2) randomly initialized frozen image encoder, and (3) pretrained image encoder using HILP (Park et al., 2024).

​​While it is nice to see comparisons with BC and SAC, it would be interesting to see performance of conservative Q learning as well, …

We agree that we should include more experiments such as more baselines, environments, and tasks. Regarding the baseline comparisons, we apologize for potential confusion. Our compared RL baselines, including the newly added baselines, all include the CQL regularizer as we always pretrain the actor and critic. What is surprising here is that even with CQL the Q-values do end up diverging (Figure 6b). We again thank reviewer cV3D on suggesting further experiments on simulated environments and/or data. As an alternative we have conducted offline RL experiments in robomimic (Table 1), as suggested by you and reviewer awMg. Based on the results we can observe that Simplified Q is better than CQL on majority of the tasks, but similar to our original findings none of the offline RL algorithms is competitive against BC.

Questions

In which cases does your method stop outperforming others? for example, is there a dependency on the number or quality of demonstrations?

This is a great question, we have included relevant discussions in Appendix A but will further describe them here. Simplified Q can still diverge (not necessarily the Q-values) when trained with higher update-to-data (UTD) ratio. Indeed, UTD ratio has been a challenge (D’Oro et al., 2022). We have also observed that the dormant neuron problem is still prevalent (Sokar et al., 2023). We expect algorithms such as LOMPO (Rafailov et al., 2021), REDQ (Chen et al., 2021), and DroQ (Hiraoka et al., 2022), possibly in combination with our regularizer, can leverage higher UTD ratio to further improve sample efficiency.

Another limitation comes from the exploration challenge in our environment. We have observed that none of the compared algorithms can grasp the item even with a larger number of episodes. We believe that more effective exploration can outperform our approach. Finally, we believe that using less data will result in narrower coverage of the dataset—in offline RL literature this appears to be a necessary condition (Zhan et al., 2022, Tkachuk et al., 2024). Unless we allow pretraining from multi-task datasets (Padalkar et al., 2023) or impose inductive biases such as equivariance (Tangri et al., 2024), it is unclear to us whether we can do any better.

Can you clarify what aspect in particular of the NTK helps most? could other kernel be used/compared? if not, why?

We thank the reviewer for the questions. As suggested by our derivation in Equations 2 and 3, the influence of the current Q-update on any Q-values is dictated by the similarity (i.e. dot product) of the state-action pairs. The magnitude of the dot product positively correlates with the change of the Q-value on the other state-action pair. Therefore orthogonalizing (i.e. decoupling) the features will mitigate Q-value changes, at the cost of restricting generalization. On the other hand, this decoupling will mitigate the impact of using the Q-network itself for bootstrapping, thereby allowing us to drop the target network altogether.

There are still other limitations however. First, this requires that the model itself is parameterized such that gradients can be computed—non-parameterized models like decision trees for example will no longer be applicable. Second, the theory on neural tangent kernels is still restricted to the infinite-width neural network setting despite aligned empirical observations. We believe that there are other necessary conditions for effective policy learning since SAC and other existing algorithms can still learn a good policy under Q-divergence. To that end perhaps other kernels can be developed due to the dependence of this policy learning component.

Did you test your algorithms on other baselines/tasks?

We thank reviewer cV3D for this question. We have updated our manuscript to include more baseline algorithms including DR3 (Kumar et al., 2021) and SAC + LayerNorm (Yue et al., 2023). As mentioned in the previous response, we have also conducted offline RL experiments in robomimic (Table 1).

References:

Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024.

Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS, 2022.

Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning (ICML), 2023.

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for dynamics and control, 2021.

Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. In International Conference on Learning Representations (ICLR), 2021.

Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout Q-Functions for Doubly Efficient Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2022.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory (COLT), 2022.

Volodymyr Tkachuk, Gellért Weisz, and Csaba Szepesvári. Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear $q^\pi$-Realizability and Concentrability. Advances in Neural Information Processing Systems (NeurIPS), 2024.

Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.

Arsh Tangri, Ondrej Biza, Dian Wang, David Klee, Owen Howell, and Robert Platt. Equivariant offline reinforcement learning. arXiv preprint arXiv:2406.13961, 2024.

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In The Tenth International Conference on Learning Representations (ICLR), 2022.

Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 60247–60277, 2023.

Thank you very much for addressing my (and other reviewers') comments and providing the clarifications. I find the updated manuscript to be much clearer overall, with well-presented and improved figures.

References:
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning (ICML), volume 202, pp. 1577–1594, 2023.

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In The Tenth International Conference on Learning Representations (ICLR), 2022.

Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research, 2023.

Rafael Rafailov, Kyle Beltran Hatch, Anikait Singh, Aviral Kumar, Laura Smith, Ilya Kostrikov, Philippe Hansen-Estruch, Victor Kolev, Philip J. Ball, Jiajun Wu, Sergey Levine, and Chelsea Finn. D5RL: Diverse datasets for data-driven deep reinforcement learning. Reinforcement Learning Journal, 5:2178–2197, 2024.

Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 60247–60277, 2023.

We thank reviewer TcT9 for the thoughtful feedback and we are delighted to see that reviewer TcT9 is impressed by our experimental results and the applicability of our method. Below we address their comments and questions and we hope they can reconsider our score:

discussion on the relationship between O2O RL and RLfD

We thank reviewer TcT9 for noting the connection between RLfD and O2ORL—indeed RLfD and O2ORL are closely related. We have updated our related work section to include RLfD and also the provided references. Regarding the specifics of the references [1, 2, 3, 4, 5], we thank the reviewer for pointing these papers out. We note that besides [2] and [3], the other references operated in state-based observations. Our finding is that many existing algorithms conducted in state-based environments failed to transfer to image-based environments, as pointed out by Lu et al. (2023) and Rafailov et al. (2024), as well as a simple reaching task in Appendix B. Regarding [3], we thank the reviewer for providing this new result. We believe [3] is an orthogonal direction—if we understand correctly, SERLfD can also suffer from Q-value divergence provided that the underlying RL agent is Q-learning based. We further note that our approach includes minimal domain knowledge (i.e. no predicates) which improves the applicability of our approach. Nevertheless, we believe combining both SERLfD and Simplified Q for more complex tasks might be required.

experimental comparisons with established RLfD approaches

We totally agree with reviewer TcT9 that comparing against RLfD baselines is important in demonstrating the robustness of our algorithms. We have attempted a similar approach to DQfD [2], replacing the DQN algorithm with Simplified-Q algorithm. The result can be found in Table 2a, w/o symmetric sampling (SS). We have noticed that simply including demonstrations in the offline buffer performs worse than using SS in both success rate and P-stop rate, corroborating the findings in RLPD (Ball et al., 2023). Therefore we have decided that RLPD is a more competitive baseline over DQfD (or SACfD in this case). In particular we have compared against RLPD built upon SAC, CrossQ, and two new additional algorithms SAC + LayerNorm (Yue et al., 2023) and SAC + DR3 (Kumar et al., 2022), all of which include a CQL regularizer due to the extra offline RL step. The results can be found in Figures 2 and 3—we have run each algorithm across three seeds and obtained the bootstrapped confidence interval—we can see that SimplifiedQ is better than all compared algorithms.

Finally, while we want to conduct more experiments to further strengthen our claim, we emphasize that these are real-life robotic experiments—due to the cost of running the experiments in both time and robot damages, comparing against older baselines before RLPD might be unsuitable. As an alternative we have conducted offline RL experiments in robomimic (Table 1), as suggested by reviewers cV3D and awMg. Based on the results we can observe that Simplified Q is better than CQL on majority of the tasks, but similar to our original findings none of the offline RL algorithms is competitive against BC.

insights on the number of demonstration trajectories required for effective training

We thank reviewer TcT9 for this question. For BC we have surprisingly found that it needs more than 500 demonstrations (10x) with data augmentation in order to achieve above 60% grasp success rate (Figure 2, right). Regarding offline RL, we have found that having at least 50 is sufficient for the policy to start approaching the bin, albeit missing the item of interest occasionally—we believe this is mostly due to the coverage of the space. Behaviourally, even with 50 demonstrations the offline RL agent is unable to go towards a specific corner of the bin initially, not until the online RL phase does the agent end up learning to grasp from said corner.

We thank reviewer TcT9 for their detailed response, we greatly appreciate the discussions on O2ORL and RLfD.

C1

Thank you for the discussion. We would like to express that O2O RL and RLfD are very similar except for the fact that RLfD usually includes near-optimal data while potentially removing the assumption of having reward access, as mentioned in the previous response. In robotic settings, demonstrations are often provided as suboptimal data (e.g. RoboMimic and our real-life experiments), not random motion like Brownian motion, as that is harmful to the robot. Secondly, we do not claim all demonstrators to be equally good—some demonstrators can result in significantly lower performance in practice—while all trajectories were successes, we cannot claim that the sole demonstrator is optimal. Nevertheless, due to very limited time constraints, we have only conducted experiments with mixed data on D4RL’s relocate and door environments. According to D4RL’s results, CQL and Simplified Q will perform slightly better than BC due to the mixture of data, furthermore Simplified Q does better than CQL. We again emphasize Simplified Q being better has been consistent across all benchmarks that we were able to test on and we truly do not believe this can be due to luck.

C2 and C3

Thank you for the extra references. We sincerely appreciate the reviewer’s effort on providing these results. While we are not motivated by RLfD, we would like to raise that none of these works were evaluated in real-life environments. In real-life there are considerations including inference time—in practice many of existing papers that leverage larger models including diffusion policies/transformers inference at <10Hz frequency (e.g. diffusion policy, RT-X). In fact we did try OpenVLA as well and it showed subpar performance and not any better than our BC baseline with a standard ResNet model.

Finally, the surprising result and its significance is that although D5RL has demonstrated existing RL methods designed on state-based environments fail on image-based environments, specifically robotic manipulation, our method can still succeed without any image-specific nor robotic-specific techniques (we note that D5RL has not yet open-sourced their dataset, hence we were unable to directly compare). While the line of work using data augmentation or pretraining with large image-dataset are promising, our result suggests that these domain-specific techniques are unnecessary. This further enabled us to run real-life image-based robotic manipulation with RL in less than few hours, which is not common at all except for limited work such as CQN (QT-Opt and SAC-X they have significantly larger scale experiments). We emphasize once again that running on real-life environments is significantly more challenging than in simulation due to cost and scalability.

C4

Please see appendix C.2 for ablation on whether our regularizer makes an impact and other design decisions including $N$-step return and symmetric sampling. Secondly, all of our baselines leverage the same algorithmic techniques such as symmetric sampling and 3-step returns as mentioned on lines 255-257. We note that all of these experiments are conducted on real-life environments.

The result on robomimic is also presented here (Table 1 in updated manuscript). Here we can see that Simplified Q is generally better than CQL except for lift, PH, where we observe a divergence on one seed. Nevertheless, similar to the general message of the original paper, we have found that offline RL is inferior to BC with the only given demonstrations, and Simplified Q is better than CQL.

As the discussion period is coming to an end, we would like to thank all reviewers for their discussions and suggestions.

We have received extra feedback regarding using mixture data to demonstrate further robustness (reviewer TcT9), and have conducted experiments on D4RL's relocate-cloned and door-cloned environments which are 50% human data and 50% policy data. We note that the human data is significantly worse than the policy data in this case. We ran CQL and Simplified Q on three seeds and have observed that Simplified Q is still consistent better (please see table below).