Intelligent Switching for Reset-Free RL

We thank the reviewer for their comments. We are glad the reviewer recognizes our innovative approach to this problem by learning when to switch. We address your concerns and questions below.

[No Significant Theoretical Results and lack of Convergence Proofs] Although we do not provide a proof of the convergence properties of RISC, we do provide a discussion of the role of bootstrapping in the reset-free RL setting in Section 4.1. Reset-Free RL thus far has been a fairly empirical field, with several prior works [1,2,3,4] being accepted to ICLR and comparable conferences without theoretical results. While it would be useful, developing such theory falls outside the scope of this work.

[Too many hyperparameters] We agree that adding hyperparameters adds complexity to the method, but we would like to point out: (1) Our algorithm is fairly robust to hyperparameter changes, and our performance does not degrade drastically even when completely removing a hyperparameter from the method. To demonstrate this, we have added a hyperparameter sensitivity analysis in Figure 10 (Appendix F, new figure) and Figure 11 (Appendix F, new figure) showing the results when only optimizing for 1 or 2 of the 3 parameters. In both cases, the perturbation/removal of hyperparameters reduced performance, but not significantly. (2) Having those hyperparameters is necessary to maintain the flexibility of our method. Unlike our work, several prior methods only work on sparse reward environments with demonstrations, and the ones that also work on dense reward environments without demonstrations, we outperform. Thus, for the cost of the extra hyperparameters, we provide added coverage on different environments.

[Value of success critic's discount factor] This was simply a design choice based on the fact that we wanted the success critic to generally reflect the capabilities and state evaluation of the main agent. A different discount factor could be used, and could potentially result in more or less conservative switching behavior.

[How the method makes learning more stable] The timeout-nonterminal bootstrapping strategy makes learning more stable, as it computes consistent targets for a given state regardless of whether a trajectory ended there. With the other strategy, if a transition is sampled from the replay, and it represents the end of the trajectory but not a terminal state, it still gets assigned a target value of 0. If this transition also exists in the replay buffer as not the end of a trajectory, it gets assigned a target value based on the next state in the transition. Removing this discrepancy by using the timeout-nonterminal bootstrapping strategy leads to more stable learning.

[Relation to CEM and Reward-Free RL] The ideas from CEM and Reward-Free RL seem complementary to the ideas of our work and potentially useful in adapting Reset-Free RL methods to unsafe environments with irreversible states. Our method allows the agent to explore areas it hasn’t mastered yet, and the ideas from CEM can be used to guide the agent and perform safe exploration.

We thank the reviewer for their valuable suggestions and helping improve our work. If your major questions and concerns have been addressed, we would appreciate it if you could further support our work by increasing your score. If there are more questions/concerns, please let us know.

Thank you for your response and for engaging in the discussion process. We respond to your questions below.

[Timeout-nonterminal bootstrapping for traditional RL] Timeout-nonterminal bootstrapping has shown benefits for traditional RL as well in terms of sample efficiency [1]. There is no major obstacle for traditional RL to use timeout-nonterminal bootstrapping other than awareness and getting the community to adopt them. The new versions of gym (Gymnasium [2]) and dm_env [3] have moved to an API that more clearly supports the distinction between termination and truncation. The challenge is now to get the community to update legacy environment code to match these APIs and legacy agent code to use the new API.

[Open-sourcing code] Yes, we will release code with the final version of the paper.

If we have addressed your questions and concerns, we would appreciate it if you could support our work by raising your score. As before, if you have any more questions/concerns, please let us know.

[1] Pardo, Fabio, et al. “Time limits in reinforcement learning.” International Conference on Machine Learning. PMLR, 2018.
[2] Towers, Mark, et al. “Gymnasium.” 2023.
[3] Alistair Muldal, et al. “dm_env: A Python interface for reinforcement learning environments.” 2019.

We thank the reviewer for their insightful comments. We’re glad they found our paper well written, our experiments well executed, our analysis of the bootstrapping strategies valuable, and the idea behind our intelligent switching mechanism to be intriguing. We address your concerns and questions below.

[Practical motivations for timeout non-terminal bootstrapping] The timeout-nonterminal bootstrapping strategy makes learning more stable, as it computes consistent targets for a given state regardless of whether a trajectory ended there. With the other strategy, if a transition is sampled from the replay, and it represents the end of the trajectory but not a terminal state, it still gets assigned a target value of 0. If this transition also exists in the replay buffer as not the end of a trajectory, it gets assigned a target value based on the next state in the transition. Removing this discrepancy by using the timeout-nonterminal bootstrapping strategy leads to more stable learning.

[Hyperparameter Sensitivity Analysis] Our algorithm is fairly robust to hyperparameter changes, and our performance does not degrade drastically when completely removing a hyperparameter from the method. To demonstrate this, we have added a hyperparameter sensitivity analysis in Figure 10 (Appendix F, new figure), and Figure 11 (Appendix F, new figure) showing the results when only optimizing for 1 or 2 of the 3 parameters. In both cases, the perturbation/removal of hyperparameters reduced performance, but not significantly.

[Will code be available?] Yes, we will release code with the final version of the paper.

[Are there any correlations between adding bootstrapping and adding an intelligent switch] The timeout-nonterminal bootstrapping strategy is necessary in order to do more aggressive switching. Since intelligent switching introduces more trajectory ends into the replay buffer, not having timeout-nonterminal bootstrapping could likely result in even more instability. This can be seen with the ablation experiments in section 5.3, where just the intelligent switching isn’t able to learn well, but adding bootstrapping and the switching results in the best agent.

[Why not use the SAC critic instead of training a separate success critic?] We trained a separate success critic because the competency metric we wanted to estimate doesn’t always correspond to the discounted return that the SAC critic is estimating (e.g., when using dense rewards or reward scaling).

[Not using bootstrapping to break the long TD chain, as in EARL] The EARL paper refers to breaking the TD chain for long trajectories when performing standard RL with large episode lengths. The tasks they do these experiments for are infinite horizon tasks without a notion of a task end. For such tasks, it might make sense to periodically use terminal state bootstrapping, since otherwise, trajectories for a single task can grow to be tens or hundreds of thousands of steps long. In our environments, while we are training in a reset-free setting, our target task has a notion of a task end and we get terminal signals naturally from the environment when the agent reaches the goal. Further investigating how the characteristics of the environment interact with the bootstrapping strategy is an interesting direction for future work.

[Is bootstrapping timeout-nonterminal states more important in the reset-free setting] We expect timeout-nonterminal bootstrapping to likely be more important for the reset-free setting. There is both empirical and theoretical evidence for this. Based on the analysis in section 4.1, we expect that having the correct timeout-nonterminal bootstrapping strategy helps stabilize learning in the reset-free setting. Since the state distribution for a reset-free agent can be very different from the forward policy’s state distribution, stabilizing the targets through the timeout-nonterminal bootstrapping is necessary to learn. In standard RL, because the state distribution more closely matches the distribution of the forward policy, the targets will likely automatically be more stable even with the timeout-terminal strategy (although it’s been shown that doing timeout-nonterminal bootstrapping can still help). Empirically, the community has only recently started to shift to timeout-nonterminal bootstrapping for episodic tasks, and still saw much success with the old strategy. In reset-free RL, several tasks cannot be learned at all or are learned suboptimally when using the timeout-terminal strategy.

We thank the reviewer for their comments. We are glad the reviewer appreciates the strength of our results and that intelligent switching can lead to better exploration in a reset-free setting. We address the reviewers' concerns and questions below.

[The paper does not provide a thorough analysis of the theoretical properties of RISC]: While developing such theory would be useful, Reset-Free RL thus far has been a fairly empirical field, with several prior works [1,2,3,4] being accepted to ICLR and comparable conferences without theoretical results, and developing such theory falls outside the scope of this work. Although we do not provide a proof of the convergence properties of RISC, we do provide a discussion of the role of bootstrapping in the reset-free RL setting in Section 4.1.

[Experiments limited to small set of environments] We perform experiments across 3 sparse reward robotic manipulation with demonstrations (tabletop, sawyer door, sawyer peg), 1 dense reward robotic navigation without demonstrations (minitaur), and 1 sparse reward gridworld task without demonstrations (4rooms). Recently published works in the Reset-Free RL have a less diverse evaluation than our work, such as MEDAL [4] (tabletop, sawyer door, sawyer peg) and VapRL [2] (tabletop, sawyer door, hand manipulation), as these are all sparse reward tasks with demonstrations. In fact, we evaluate on every publicly available environment in the EARL benchmark other than Franka Kitchen, which no reset-free method has made any progress on.

[Comparison of RISC to other reset-free algorithms in terms of sample efficiency and generalization] We refer the reviewer to Figures 4 and 5 (main paper, existing figures) for a summary of RISC’s performance in terms of generalization. Our algorithm has the best performance on two of the evaluated environments, the second best performance on the other two, and across all environments, has the highest mean and interquartile mean. We also have added Figure 7 (Appendix D, new figure) to highlight our method’s gains in sample efficiency compared to prior work. The figure shows the area under the curve of the learning curves, and shows that our method is significantly more sample efficient than the other baselines. Thank you for this helpful suggestion.

[Can RISC be extended to handle environments with irreversible states, where the agent could get stuck?] Yes, we believe that RISC could be extended to handle environments with irreversible states. It would likely involve making the exploration more conservative, perhaps incorporating ideas from the risk-averse RL literature [5]. An alternative approach would be to incorporate prior knowledge about the environment and unsafe states into the switching algorithm through the use of large pretrained models.

[Combining RISC with curriculum learning or demonstration-based techniques] RISC by itself does form a type of implicit curriculum, as it switches directions whenever it encounters a state it has already solved, thus focusing on states it hasn’t solved. We agree with the reviewer that finding ways to combine RISC with more sophisticated curriculum learning methods or finding ways of using demonstrations more effectively is an important direction of research and could enhance the performance of our method, but this would require several more significant research contributions and is out of scope for this work.

We thank the reviewer for their valuable comments and helping improve our work. If your major questions and concerns have been addressed, we would appreciate it if you could further support our work by increasing your score. If there are more questions/concerns, please let us know.

[1] Eysenbach, Benjamin, et al. "Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning." International Conference on Learning Representations. 2018.
[2] Sharma, Archit, et al. "Autonomous reinforcement learning via subgoal curricula." Advances in Neural Information Processing Systems 34 (2021): 18474-18486.
[3] Sharma, Archit, et al. "Autonomous Reinforcement Learning: Formalism and Benchmarking." International Conference on Learning Representations. 2021.
[4] Sharma, Archit, Rehaan Ahmad, and Chelsea Finn. "A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning." International Conference on Machine Learning. PMLR, 2022.
[5] Brunke, Lukas, et al. "Safe learning in robotics: From learning-based control to safe reinforcement learning." Annual Review of Control, Robotics, and Autonomous Systems 5 (2022): 411-444.

We thank the reviewer for their insightful comments. We’re glad they found our analysis of the bootstrapping strategies rigorous and easy to follow, and our idea of using competency of the policy to control switching to be cogent. To address your concerns, we provide a discussion of the few hyperparameters our method introduces below. The content of this discussion has also been added to Appendix F of the paper.

[Tradeoff of adding hyperparameters] We agree that adding hyperparameters adds complexity to the method, but we would like to point out: (1) Our algorithm is fairly robust to hyperparameter changes (Figure 10, Appendix F, new figure), and our performance does not degrade drastically even when completely removing a hyperparameter from the method (Figure 11, Appendix F, new figure) (2) Having those hyperparameters is necessary to maintain the flexibility of our method. Unlike our work, several prior methods only work on environments with demonstrations, and we convincingly outperform the methods that can work without demonstrations. Thus, for the cost of the few extra hyperparameters, we provide added coverage on a broader set of environments.

[Hyperparameter Selection] As mentioned in Appendix B, to select the hyperparameters for our experiments, we conducted a grid search over the following hyperparameters: conservative factor $\beta\in\{0.0, 0.9,0.95\}$, minimum trajectory length as a fraction of the maximum trajectory length $\in\{0.0, 0.25, 0.5, 0.75\}$, and switching trajectories proportion $\zeta\in\{0.25, 0.5, 0.75, 1.0\}$.

[Hyperparameter Sensitivity Analysis] We also conducted a sensitivity analysis of each hyperparameter on each environment and added it as Figure 10 (appendix F, new figure). We took the configurations selected for each environment, and varied the value of one hyperparameter at a time. The results show that our results are fairly robust to changes in hyperparameters, with fairly small changes to the final return across all environments.

[Importance of each hyperparameter] We also want to discuss the relative importance of each hyperparameter. In Figure 8 (appendix E, existing figure), we show learning curves demonstrating the effect of removing each modulation on the performance. Continuing that discussion, we add Figure 11 (appendix F, new figure) showing the effect of doing a partial hyperparameter search. Figure 11.a shows the performance achieved when only optimizing one hyperparameter, and disabling the other modulations. The most important modulations seem to be the minimum trajectory length or the proportion of trajectories where the switching happens. They recover most of the performance, but the full method still results in a slight performance gain. Figure 11.b shows the performance when optimizing 2 out of the 3 hyperparameters, and shows that each pair of hyperparameters can get fairly close to the full method’s performance, but still does slightly worse.

We thank the reviewer for their valuable suggestions and helping improve our work. If your major questions and concerns have been addressed, we would appreciate it if you could further support our work by increasing your score. If there are more questions/concerns, please let us know.