Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning

5. Ablations on hyperparameter H:

We would like to clarify that we used H=1 for SGFT-SAC and H=4 for SGFT-TDMPC in the experiments in the initial submission. For the rebuttal, we have additionally performed a more extensive ablation on the effects of H on two standard sim-to-sim benchmark experiments: Walker and Rope Peg-in-Hole [5] (Figure 13). Here, the sim2real dynamics gap is proxied as perturbations in simulator dynamics. Since the Rope Peg-in-Hole Environment requires significantly more precise motions than Walker to solve, this enables us to tackle the question: do tasks that require higher precision control require higher H, which in turn makes the policy transfer more difficult? We find that the performance of SGFT is relatively insensitive to the choice of H on Walker. However, SGFT requires larger H to achieve high-performance on the more precise Rope Peg-in-Hole task. We believe these experiments motivate the need for the more general framework. Please refer to Figures 8 and 9 for details.

6. Question 1: “When H=1…”

This is a very interesting point, and an interesting intuition! We have addressed this in the next comment. 

7. Question 2: “I’m not convinced by the statement…”

Thank you for drawing this to our attention; we agree we can improve the wording here. What we meant to say here was:

“For small values of H, optimizing Equation (3) guides policy optimization algorithms towards policies that increase the value of V_sim over H-step windows. In the extreme case where  H=1, SGFT greedily attempts to increase the value of V_sim each step. Conceptually, in this special case we have reduced the policy search problem to a contextual bandits problem, which is much easier to solve than the original infinite horizon objective. Intuitively, this is because the critic is frozen and no bootstrapping in the real-world is required. More generally, as H becomes larger the dependence on V_sim is lessened while the dependence on the real returns increase.” 

We have updated the draft accordingly, please let us know if we have cleared this point up! 

Thank you again for your feedback, and please let us know if there are further ways you believe we can improve our submission! 

[1] Tiboni, G., Arndt K., and Kyrki V. "DROPO: Sim-to-real transfer with offline domain randomization." Robotics and Autonomous Systems 166 (2023): 104432.

[2] Tiboni, G. et al. "Domain Randomization via Entropy Maximization." ICLR 2024.

[3] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023.

[4] J. Zhang, M. Heo, Z. Liu, E. Biyik, J. J. Lim, Y. Liu, and R. Fakoor, “Extract: Efficient policy learning by extracting transferrable robot skills from offline data,” arXiv preprint arXiv:2406.17768, 2024.

Dear Reviewer, 

Thank you for your feedback on our work! We have been busy running the additional experiments requested by you and the other reviewers. We have uploaded all of the figures we will reference to a document titled ‘Rebuttal Figures’ attached in the supplementary; we will incorporate these into the paper draft in the coming days.  Please see our common response for a summary of the new experiments, including two new real-world deformable object manipulation tasks with a squishy ball and a towel. These are difficult tasks for sim-to-real methods wherein the benefits of SGFT are substantial. 

We have run the additional baselines you requested on the puck, ball, and cloth pushing tasks, and are planning to also run them on the hammering and insertion tasks tomorrow. However, we wanted to respond as soon as possible to your concerns with our current experiments, and give you a chance to let us know if there are any other points we can address in the coming days. 

1. Adding references to state-of-the-art methods

We thank you for pointers to recent state-of-the-art methods: DROPO [1] and DORAEMON [2]. We have now discussed them in the updated version of the related work, with the following text:

“Another appealing approach is to search for distributions of domain randomization parameters that will lead to maximally robust transfer in a principled fashion. For example, Tiboni et al. (2023a) uses entropy maximization to find simulator parameters which will lead to maximally robust transfer while accurately describing a small number of real world trajectories. This is taken further by Tiboni et al. (2023b), which automatically evolves training distributions purely in simulation to generate a curriculum of environments which leads to policies which are both performant and robust.”

Moreover, we have included these baselines in our experiments for the puck, ball, and cloth pushing tasks. In these three tasks, DROPO and DORAEMON generally perform favorably compared to other direct transfer methods. However, with additional fine-tuning data SGFT is able to improve over the success rates of these methods. Please refer to Figures 1, 2, and 3 for further details. In future work, we would be very interested to see if these methods could provide a better initial policy for fine-tuning. 

2. Clarification on performance of SAC baseline

We use the StableBaselines3 implementation of SAC, which is robust and widely used. Moreover, because SGFT-SAC is built on top of SAC, our method depends on the strength of the implementation, and we carefully tuned SAC for our experiments. This is further highlighted by the performance of SAC in our simulation experiments (Figure 10), where SAC generally performs comparably to SOTA methods such as TDMPC. Thus, we contend that the performance of our SAC implementation is reasonable and that learning in the real world is significantly harder for many methods than learning in simulation. Nonetheless, we ran RLPD [3] on the puck, ball and cloth pushing tasks, and found that it led to only minimal improvements over our SAC implementation.  

3. Additional experiments with asymmetric actor critic

We note that the policy from SGFT can actually be trained with asymmetric actor-critic in simulation to obtain high-performance policies. Then, one could estimate a critic for this policy using only available observations (i.e. no privileged information) to use with SGFT. Nonetheless, we agree that comparing to asymmetric actor-critic domain methods is important, and we have added this as a baseline to our hardware experiments. We found that it did not significantly impact performance for domain randomization methods on the puck, ball, and cloth pushing tasks. Please refer to Figures 1, 2, and 3 for details.

4. Evaluating all baselines for real-world pushing task

We evaluate all baseline methods we missed in the initial submission for the pushing tasks in the real-world: Recurrent Policy + Domain Randomization, ASID, TDMPC2, SGFT-TDMPC2, IQL, DROPO[1], DOREAMON[2], Asymmetric Actor-Critic[3], and RLPD[4]. We observe that both versions of SGFT substantially outperform prior fine-tuning methods, and yield better final policies than direct transfer methods. Please refer to Figure 1 for details.

I thank the authors for the clear responses to all my concerns. The current last revision of the paper displays a thorough experimental evaluation, which makes the paper a valuable addition to the field. I'm therefore keeping the score as six as I think it's worth accepting it. I'm not raising the score higher than six as I believe the method could benefit from a better theoretical explanation of the implication to the RL problem when reducing the finetuning problem to a contextual bandit problem. I'm suspecting there could be connections to existing works and settings that are missing, as the practical implementation of the method is rather straightforward and simple (i.e. frozen critic and setting done=True for each transition). For example, could this approach be connected to "residual learning" [1,2]? E.g. in [2] they mention:

Every time a new task needs to be learned, we can transfer our learned residuals and learn the component of the Q-function that is task-specific, hence, maintaining the task structure from prior behaviors.

Unfortunately I don't have enough theoretical expertise to judge for myself and hence will leave my score as it is.

[1] Klink, Pascal, et al. "Boosted curriculum reinforcement learning." International Conference on Learning Representations. 2021.

[2] Jauhri, Snehal, Jan Peters, and Georgia Chalvatzaki. "Robot learning of mobile manipulation with reachability behavior priors." IEEE Robotics and Automation Letters 7.3 (2022): 8399-8406.

Dear Reviewer, 

Thank you for your constructive comments! We have rewritten the introduction and added a small contributions section to clarify the framing of our approach. We have also updated our figures and writing to fix the errors that you have highlighted. Before addressing these points in more detail, we note that we have also included two new deformable object manipulation tasks. These are very difficult tasks for existing sim-to-real approaches, and the performance gains of SGFT are even more pronounced on these tasks. See the meta-comment above for details, and the ‘Rebuttal Figures’ in the supplementary for plots. We intend to add these new results to the paper draft soon. 

1. Improving introduction

In the new draft of the introduction, the main point of each paragraph is as follows: 

i) Robot learning is a powerful paradigm, but collecting enough real-world data for current algorithms is challenging and expensive.

ii) Physics simulators can provide a scalable, cheap source of off-domain (i.e. inaccurate) data. 

iii) However, due to the domain gap, current transfer techniques struggle for challenging contact-rich tasks.

iv) Prior methods have investigated fine-tuning policies pretrained in simulation using real-world data, but use unstructured exploration strategies which are inefficient and impractical for real-world learning.

v) In contrast, we argue for distilling information from the simulator that will guide and accelerate real-world fine-tuning.

vi) In more detail, we accomplish this by reshaping objectives with value functions. We show this is particularly effective when paired with MBRL algorithms.

Contributions:

1) We introduce a light-weight framework which substantially accelerates existing fine-tuning approaches.

2) We systematically show SGFT significantly outperforms numerous baselines across many real-world tasks.

3) We provide theoretical analysis for the performance gains we observe.

Please let us know if there is anything we can do to make our contribution clearer! 

2. Improving figure illustrations

Thank you for pointing out the typo in Figure 2. We have fixed the notation, also modified the figure to make our framework easier to parse. Please let us know if you have any further suggestions!

3. Clarification on how dynamics model is learned
The dynamics model is learned by fitting a generative model on the dataset, following the approach of conventional MBRL approaches such as MBPO and TD-MPC2. In our implementation, the dynamics model is trained via maximum likelihood to predict the next state given the current state and action. Please refer to Algorithm 2 (Dyna-SGFT) in the paper for further details on how we learn the dynamics model.

4. Typo in abstract

Thank you for pointing out the typo in the abstract. We have fixed “is to inefficient” -> ”is too inefficient”.

We hope these additional explanations and modifications have addressed your previous questions. Please don’t hesitate to let us know for any additional comments or questions.

2. SGFT is a paradigm shift for sim-to-real fine-tuning:

We argue that SGFT, while simple to implement, is a substantial paradigm shift from prior sim-to-real fine-tuning approaches such as [6][7] and not just a minor engineering improvement. In short, these approaches a) pretrain a policy in simulation and b) simply use this as an initialization for real world learning. This approach does not retain the structure of the simulator during fine-tuning. By transferring knowledge with the value function through reward shaping, SGFT is a novel, conceptually distinct paradigm which leverages the structure of the simulator to guide and accelerate real-world learning. We emphasize is a broad algorithmic insight and not simply an engineering trick for a particular setting - it is applicable across fine-tuning algorithms and tasks, and is distinct from what people currently do for sim to real fine-tuning. Moreover, we demonstrate it has theoretical grounding and leads to substantial empirical gains. 

We hope these additional explanations and modifications have addressed your previous questions. Please don’t hesitate to let us know for any additional comments or questions.

[1] Tiboni, G., Arndt K., and Kyrki V. "DROPO: Sim-to-real transfer with offline domain randomization." Robotics and Autonomous Systems 166 (2023): 104432.

[2] Tiboni, G. et al. "Domain Randomization via Entropy Maximization." ICLR 2024.

[3] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023.

[4] Peng, Xue Bin, et al. "Sim-to-real transfer of robotic control with dynamics randomization." 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.

[5] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-toreal transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1290–1296. IEEE, 2021.

[6] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International

conference on machine learning, pp. 7444–7453. PMLR, 2019.

[7] Yunchu Zhang, Liyiming Ke, Abhay Deshpande, Abhishek Gupta, and Siddhartha S. Srinivasa.

Cherry-picking with reinforcement learning. In RSS, 2023

Dear Reviewer, 

Thank you for your feedback on how we can improve our evaluation to better justify our claims. We have performed 7 new real-world experiments and simulated ablations to address your concerns. Then we seek to clarify how SGFT, while simple to implement, is a substantial conceptual departure from prior fine-tuning methods such as [6] and [7]. 

We have uploaded all of the figures we will reference to a document titled ‘Rebuttal Figures’ attached in the supplementary; we will incorporate these into the paper draft in the coming days.  

1. More extensive real-world and simulation experiments

We have included 5 new hardware experiments, including deformable object manipulation tasks, and 2 new sim-to-sim experiments to further highlight the capabilities of SGFT. 

1.1. Real-world deformable object manipulation tasks

We add two additional real-world tasks, where the objective is to push deformable objects (a towel and a squishy toy ball) to a desired location (Figures 11 and 12). Deformable objects are notoriously difficult to model, and a significant challenge for sim-to-real methods. To achieve this, we train a pushing value function in simulation on a small range of objects with varying physical parameters (size, mass, etc). Even though the distribution of simulated objects is extremely different from the real-world deformable objects, SGFT achieves a near 100% success rate in 100 trajectories. Meanwhile, for these substantially harder tasks, all other baselines achieve less than 50% success rate for the toy ball and less than 30% for the towel. Details depicted in Figures 2 and 3.

1.2. Additional sim-to-sim experiments

In addition to the two standard sim-to-sim benchmark experiments (Walker and Cheetah) from [5] in Appendix E of the initial submission, we have added an additional environment: Rope Peg-in-Hole. In each of these experiments, SGFT substantially outperforms baseline transfer and fine-tuning methods. Details depicted in Figures 5, 6, and 7.

1.3. Ablations on hyperparameter H

We ablate over the choice of H on two standard sim-to-sim benchmark experiments: Walker and Rope Peg-in-Hole [5] (Figure 13). Here, the sim2real dynamics gap is proxied as perturbations in simulator dynamics. Since the Rope Peg-in-Hole Environment requires significantly more precise motions than Walker to solve, this enables us to tackle the question: do tasks that require higher precision control require higher H, which in turn makes the policy transfer more difficult?

We find that the performance of SGFT is relatively insensitive to the choice of H on Walker. However, as the reviewer conjectured, SGFT requires larger H to achieve high-performance on for Rope Peg-in-Hole. Please refer to Figures 8 and 9 for details. We thank the reviewer for suggesting this sort of experiment! 

1.4. (New) baseline methods for pushing task

To address question (1), we evaluate all baseline methods we missed in the initial submission for the pushing task in the real-world: Recurrent Policy + Domain Randomization, ASID, TDMPC2, SGFT-TDMPC2, IQL. Per request of reviewer TjP503, we additionally run baselines of DROPO[1], DOREAMON[2], Asymmetric Actor-Critic[3], and RLPD[4] on the pushing tasks. As with our previous experiments, SGFT substantially outperforms prior methods for these tasks. Please refer to Figures 1, 2, and 3 for details. We will also add the new baselines for hammering and insertion tomorrow. 

1.5. Additional real-world ablation on domain randomization

Per request from reviewer ZMTP07, we have additionally ablated the effects of domain randomization on our real-world experiments. In particular, we used SGFT to finetune policies which were trained without dynamics randomization. While this substantially reduced the effectiveness of the initial policies, it had a minimal effect on the fine-tuning performance of SGFT, demonstrating our method is insensitive to this hyper parameter. Details depicted in Figure 4.

Dear Reviewer, 

Thank you for your thorough feedback on how we can improve the evaluation of our method. We have conducted hardware experiments for two new deformable object manipulation tasks to further highlight the capabilities of SGFT. Our central claim is that SGFT bypasses the need for an accurate simulator, reducing engineering effort and enabling transfer with substantially less fine-tuning data for tasks which are impractical to model accurately.

For ease of access, we have uploaded all of the figures we will reference to a document titled ‘Rebuttal Figures’ attached in the supplementary; we will incorporate these into the paper draft in the coming days. 

1. Scaling to more complex (deformable) tasks

To address weaknesses (1)(4) and question (1)(3), we add two additional real-world tasks, where the objective is to push deformable objects (a towel and a squishy toy ball) to a desired location (Figures 11 and 12). To achieve this, we train a pushing value function in simulation on a small range of objects with varying physical parameters (size, mass, etc). Even though the distribution of simulated objects is extremely different from the real-world deformable objects, SGFT achieves a near 100% success rate in 100 trajectories. Meanwhile, for these tasks, all other baselines achieve less than 50% success rate for the toy ball and less than 30% for the towel. Details depicted in Figures 2 and 3. This highlights our argument that value functions trained in simulation can provide impactful guidance and massively accelerate real-world learning even when the simulator is extremely inaccurate, reducing the need for engineering effort.

All in all, we now have 5 real-world tasks, which we believe to be a reasonable spread over tasks of interest to robot learning. For example, the peg insertion task is a multi-step task, while hammering displays highly dynamic contact-rich motion. We believe this provides a broad evaluation which provides a more comprehensive understanding of SGFT’s robustness and adaptability.

2. SGFT substantially outperforms prior methods

To address weakness (2), we would like to clarify that our primary claim is that SGFT boosts sample efficiency, not final performance. Across the board, SGFT is more sample efficient than prior methods, upwards of 2-3x on hammering, 2x on puck pushing, and even larger margins on the deformable tasks where prior methods struggle completely. Details are shown in Figures 1, 2, and 3. We view the fact that SGFT has higher final performance on some tasks to be an additional benefit, but is not the primary claim we aim to highlight. 

3. SGFT requires minimal engineering overhead

To address weakness (3) and question (2), we first remark that our core reward-shaping techniques are very simple to implement, and a simple addition to existing sim-to-real pipelines. However, we also argue that it requires substantially less engineering effort when designing simulation environments compared to prior sim-to-real approaches. To further address this point, we additionally ran ablations for the puck pushing tasks where we removed all dynamics randomization from the pretraining phase. As depicted in Figure 4, this had barely any effect on the learning performance of SGFT. Thus, carefully tuning simulation parameters is not a particularly important ingredient of SGFT. Instead the methodology introduced in SGFT actually allows the fine-tuning procedure to overcome significant misspecification during simulator design, making the engineering task easier. 

We hope these additional explanations and modifications have addressed your previous questions. Please don’t hesitate to let us know for any additional comments or questions.

4. Engineering effort of SGFT

Reviewer ZMTP07 raised the concerns over the engineering effort required for our method, specifically:  “SGFT’s approach requires a highly accurate simulation setup to guide policy learning effectively, which can be labor-intensive.” 

Our central claim is that SGFT bypasses the need for an accurate simulator, reducing engineering effort and enabling transfer with substantially less fine-tuning data for tasks which are impractical to model accurately. Moreover, our core reward-shaping insights are very simple to implement, and a straightforward addition to existing sim-to-real pipelines. To demonstrate that SGFT reduces the effort needed to design simulation environments, we additionally ran ablations for the puck pushing task where we removed all dynamics randomization from the pretraining phase. As depicted in Figure 4, this had barely any effect on the learning performance of SGFT. Thus, carefully tuning simulation parameters is not a particularly important ingredient of SGFT. Instead the methodology introduced in SGFT actually allows the fine-tuning procedure to overcome significant misspecification during simulator design, making the engineering task easier. 

5. SGFT substantially outperforms prior methods

Reviewer ZMTP07 raised concerns over the performance gains of SGFT over baselines: “SGFT performs comparably to these baselines in most cases, it does not consistently outperform them by a significant margin, raising questions about whether the added complexity of the SGFT approach is justified by these incremental improvements”

We would like to clarify that our primary claim is that SGFT boosts sample efficiency, not final performance. Across the board, SGFT is more sample efficient than prior methods, upwards of 2-3x on hammering, 2x on puck pushing, and even larger margins on the deformable tasks where prior methods struggle completely. We view the fact that SGFT has higher final performance on some tasks to be an additional benefit, but is not the primary claim we aim to highlight. 

[1] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444–7453. PMLR, 2019.

[2] Yunchu Zhang, Liyiming Ke, Abhay Deshpande, Abhishek Gupta, and Siddhartha S. Srinivasa. Cherry-picking with reinforcement learning. In RSS, 2023