Learning variable-length skills through Novelty-based Decision Point Identification

Dear Reviewer Yx7H,

We thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.

1. Its improvement on the two robotic simulation tasks is small, according to Figure 6.

Thank you for pointing out this. We've incorporated Table 1 to highlight the degree of quantitative improvement, especially in complex environments, compared to SPiRL[2]. Please refer to the general response to see the details.

2. In robotic tasks without clearly defined skills it is doubtful whether NBDI can outperform SPiRL.

Thank you for your insightful feedback. In the sparse block stacking environment, the agent needs to stack blocks on top of each other. Since the agent’s sole objective is to stack as many blocks as possible, this environment can be regarded as a robotic task without clearly defined skills (whereas kitchen has clearly defined subtasks). Please refer to Appendix B for more details.

Table 1 shows that NBDI outperforms SPiRL in the sparse block stacking environment (robotic tasks without clearly defined skills) with an improvement rate of 67.16%. Note it is a higher improvement rate compared to robotic task with clearly defined skills (kitchen: 22.33%). The significance of learning variable length skills with state-action novelty increases in environments lacking clearly defined subtasks (sparse block stacking).

3. With non-expert data collection policies with stochasticity, it is doubtful whether NBDI can outperform SPiRL.

Thank you for your constructive advice. We've added a new section (Appendix C) to investigate the extent to which our proposed method depends on the quality of the offline dataset. Please refer to the general response to see the details.

4. How did you adjust the model capacity and training steps of ICM[1]? If ICM overfits on training data, can it still represent state-action novelty, as I discussed in Weakness (2)?

Thank you for raising this important point. We have included a section (Appendix E) to show that the estimated state-action novelty is not much influenced by the number of parameters or dataset used to train the model. We also doubled the number of training steps, but observed that the scale didn't differ significantly.

5. When generating termination labels for the dataset, how to convert the continuous prediction loss of ICM into binary termination labels? Does this require selecting a threshold manually?

Thank you for pointing out this. Empirically, we discovered that state-actions falling within the top 1% of the prediction error percentile serves well as critical decision points, and used the corresponding threshold to learn the termination distribution.

6. The theorem tells that breaking skills into shorter pieces can improve the high-level controller. But the paper studies how to determine the skill termination points, instead of breaking pre-defined fixed-length skills into shorter pieces.

SPiRL basically extracts random fixed-length action sequences from the offline dataset by regarding the fixed-length action sequence in the dataset as pre-defined skills. On the other hand, our approach tries to break long action sequences using state-action novelty to learn better defined skills. Given that we are able to learn optimal skill (high-level) policy, the optimal way of defining skill would be using low-level actions directly as skills. In practice, however, shorter skills will increase the difficulty of learning, leading to poor skill policy. This is why we need a more clever and efficient break down of skills that resembles terminations with optimal improvement as shown in the Theorem [3] and the Figure 4, while obtaining similar temporal abstraction.

[1] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787. PMLR, 2017

[2] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188–204. PMLR, 2021a

[3] Richard S Sutton. Between mdps and semi-mdps: Learning, planning, and representing knowledge at multiple temporal scales. 1998

Dear Reviewer Pqwe,

We appreciate your comprehensive comments. Please find the response to your questions below.

1. Lack of option discovery/learning baselines

We appreciate your constructive advice for the experiment. We added a section (Appendix F) to compare NBDI with different variable length algorithm. Please refer to Appendix F for more details.

We also attempted to incorporate [2] into the comparison; however, we were unable to replicate the results reported in the paper since the dataset used to train [2] was not made available. So we mainly compared our model with [3] in Appendix F.

2. Standard offline RL algorithms should be compared to instead of SAC

As depicted in Figure 5, both NBDI and SPiRL[1] need environment interactions to train the skill policy. As our paper primarily focuses on the online RL setting, we believe SAC serves as a suitable baseline for comparison across various environments.

3. Test the proposed approach on more domains.

Thank you for this suggestion. We've incorporated a section (Appendix D) where we explore the possibility of identifying meaningful decision points in complex physics simulation environments. Please refer to Appendix D for more details.

[1] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188–204. PMLR, 2021a

[2] Shankar, Tanmay, and Abhinav Gupta. "Learning robot skills with temporal variational inference." International Conference on Machine Learning. PMLR, 2020.

[3] Jiang, Yiding, et al. "Learning Options via Compression." Advances in Neural Information Processing Systems 35 (2022): 21184-21199.

Dear Reviewer zT8h,

We thank the reviewer for the detailed feedback. We address your remarks below.

1. To which extent the exact nature of the curiosity signal influences the learned skills.

We acknowledge the concern raised regarding the missing details, and we would like to address this by providing the necessary information in Appendix B. Please refer to the general response to see the details.

2. How dependent NBDI is on a suitable structure being present in the offline dataset. For example, could NBDI work with exploratory data, and if yes, under which circumstances?

We appreciate your constructive advice for the experiment. We've introduced a new section (Appendix C) to explore to what extent our proposed method is depend on the quality of the offline dataset. Please refer to the general response to see the details.

3. How does the complexity of the environment influence the requirements on the data?

We have also added a section (Appendix D) to investigate whether meaningful decision points can be found in complex physics simulation environments. Please refer to Appendix D for more details.

4. In the paragraph “Option Framework”, in the definition $\beta:S^+\to[0,1]$ the symbol is not defined anywhere. What does it stand for?

We apologize for the confusion caused. We initially intended to define $S^+$ as the termination set in accordance with [2], but we deemed it unnecessary and substituted it with the state space $S$.

5. In equation (5) $\tilde{r}$ is added to the objective in each time step even though $\tilde{r}$ itself is already the cumulative discounted reward for the execution of one skill. Could you maybe explain how to reconcile these definitions?

Thank you for pointing this out. To reconcile the definition, we have now defined a set of time steps where we execute skills (Equation 5).

[1] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188–204. PMLR, 2021a

[2] Richard S Sutton. Between mdps and semi-mdps: Learning, planning, and representing knowledge at multiple temporal scales. 1998

Dear Reviewer 6rC9,

Thank you for the valuable feedback. We hope we can address your concerns below.

1. Clarity of proposed approach (state-action novelty estimation)

We apologize for the missing details. We've included a section (Appendix B) that details our application of the Intrinsic Curiosity Module (ICM)[1] in calculating state-action novelty. Please refer to the general response to see the details.

2. The quantitive gains over SPiRL[2] seems very marginal in the more complex environments (sparse block stacking, kitchen).

Thank you for pointing out this. We've included Table 1 to highlight the extent of quantitative improvement over SPiRL (including complex environments). Please refer to the general response to see the details.

3. Can the authors include some analysis of how their discovered skills differ qualitatively in these more complex environments, as compared to [2]? What are the specific cases in these environments that variable length skills are actually helpful/needed?

Thank you for the valuable suggestion. We've included Figure 11 (Appendix B) to illustrate states in the sparse block stacking environment that NBDI would learn to terminate (A, B, C) and states it would not learn to terminate (D, E, F). NBDI would learn to terminate in states where the agent has multitude of plausible actions (A, B, C) (e.g. choose to lift the block below or to move towards other blocks), while SPiRL would terminate in random states (A, B, C, D, E, F). As indicated by the improvement rate in Table 1, the significance of learning variable length skills with state-action novelty becomes increasingly crucial in environments lacking clearly defined subtasks (sparse block stacking), or in more complex settings (maze 40x40).

[1] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787. PMLR, 2017

[2] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188–204. PMLR, 2021a