NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations

Thank you for your review and constructive suggestions. We address your questions below.

Q1: Details in skill termination and execution

Thank you for your feedback. To clarify, the variable-length skill embedding space $z$ is learned offline during the skill extraction phase using a deep latent variable model. During this phase, the low-level policy (i.e., the skill decoder in Figure 12) receives a randomly sampled experience from the task-agnostic demonstration, along with a termination condition vector $\beta$ provided by the pre-trained state-action novelty module. The latent model is trained to reconstruct the corresponding action sequence and its length (i.e., the termination point) by maximizing the evidence lower bound (ELBO) (Appendix B.1).

During the execution phase (or when solving downstream tasks), the pre-trained low-level policy (i.e., the skill decoder) takes as input a latent skill vector $z$ from the high-level policy and reconstructs action and termination signal $\beta$ at each step, without needing to observe or predict future states or actions (Appendix B.2). Thus, while future trajectory segments are used during skill extraction to learn variable-length skill embeddings and terminations, they are not required at execution time, ensuring our approach remains practical and deployable.

Q2: Task-Agnostic Dataset Size and Coverage

Thank you for your feedback. The number of task-agnostic trajectories collected for each environment were as follows: maze (85,000 trajectories), block stacking (37,000 trajectories), and kitchen (400 trajectories). Since we used image-based observations in the maze and block stacking environments, and the downstream tasks involve significant configuration changes (e.g., entirely new maze layouts or a larger-scale block stacking environment with more blocks in random positions), we require a larger set of task-agnostic demonstrations compared to the kitchen environment, where the overall layout remains consistent across tasks (Appendix F).

Since our goal in maze and block stacking environment is to solve downstream tasks with significantly different environment configuration, we do not assume that task-agnostic demonstrations fully cover the state space of the downstream tasks. However, it is important that they provide a good coverage of the observation space. Note we are using cropped image centered around the agent from task-agnostic demonstrations as observations. This way, we can extract consistent structural information that can be extracted across multiple task-agnostic trajectories, which then can be used to detect state-action novelty based decision points even in downstream tasks with significantly different environment configurations (Appendix F).

Q3: Impact of model capacity and dataset usage

Thank you for the insightful feedback regarding the experiments. We agree with the reviewer that simply reporting the mean and standard deviation of state-action novelty across 25 trajectories may not sufficiently demonstrate the robustness of our method with respect to model capacity or dataset size. The concern that machine learning models can trivially capture mean values under i.i.d. sampling is valid, and we appreciate the opportunity to clarify our intent.

To address this, we revised the analysis in Figure 8 to present box plots that visualize the full distribution of state-action novelty scores across 25 sampled trajectories. The revised version of Figure 8 is available at: https://imgur.com/a/9gQ8IrU. This allows us to assess the variance, range, and consistency of state-action novelty estimates across different network sizes (left) and dataset usage levels (right). From the revised plots, it can be observed that the spread (variance, median and interquartile range) of state-action novelty remains relatively stable across different model sizes and dataset usage.

Q4: Paper structure rearrangement

Thank you for your thoughtful suggestions regarding the structure of the paper. We agree that the current presentation of Sections 5.1 and 5.2 may limit accessibility for readers who are less familiar with skill extraction from task-agnostic demonstrations. To address this, we will incorporate concise summaries of Appendix B.1 and B.2 into Sections 5.1 and 5.2. Additionally, we will expand the caption of Figure 3 to briefly explain the visualization procedure used to generate the figure. This added context will help readers better interpret the figure and understand its significance.

To accommodate these changes within the page limit, we will move Figure 2(b) and its accompanying explanation (currently in Section 6.1) to the appendix. Furthermore, we will relocate Figure 2 to Section 6, where it is first referenced, to improve the flow of the paper.

Thank you for your valuable feedback. Please find our responses to your concerns below.

Q1: Diversity of the task demonstration set

Thank you for your insightful suggestion. In our experiments, we found that the initial task-agnostic demonstration set needs to be diverse enough to cover the observation space (i.e., cropped images centered around the agent (Figure 2)), rather than fully covering the state space of downstream tasks. This diversity is especially important in environments like maze and block stacking, where downstream tasks involve significant changes in configuration (e.g., unseen maze layouts or random block arrangements in a larger-scale block stacking environment). To ensure sufficient diversity, we used 85,000 trajectories in the maze environment and 37,000 trajectories in the block stacking environment. In contrast, the kitchen environment, where the scene layout is consistent across tasks, required only 400 trajectories.

Q2: Details in the average length of skills learned

Thank you for your feedback. In Appendix A.4, we explored the impact of fixed skill trajectory lengths by evaluating whether NBDI still outperforms SPiRL [1] when SPiRL is configured to use the average skill length observed in NBDI (referred to as SPiRL (avg-fixed)). We found that NBDI consistently outperforms SPiRL even when SPiRL uses these average fixed skill lengths. Interestingly, SPiRL (avg-fixed) performed worse than SPiRL with a fixed skill length of 10 in the block stacking environment, but better in the maze environment. However, across all fixed skill lengths tested (ranging from 10 to 30), there was no instance where SPiRL outperformed NBDI. These findings support our claim that NBDI can effectively leverage critical decision points in the environment compared to fixed length approaches.

Furthermore, in Table 2 of Appendix A.4, we report the skill lengths of high-level policy for each environment during downstream reinforcement learning. The results show that our method maximizes temporal abstraction by frequently executing longer skills (lengths 21–30). At the same time, it retains the flexibility to terminate earlier (lengths 1–10) when critical decision points are encountered. This adaptability enables more efficient exploration of the state space and enhancing the transfer of knowledge across various tasks.

Q3: More informative figure captions

Thank you for your suggestion. We will revise the captions for Figure 1 and Figure 2 to provide additional context about the tasks and domains depicted. Additionally, we will expand the caption for Figure 3 to briefly explain the visualization procedure, thereby helping readers better interpret the figures and their significance.

[1] P., Karl, et al. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

Thank you for your thoughtful comments. Please find the responses to your questions below.

Q1: Details in implementation of termination conditions

Thank you for the suggestion. To improve clarity, we will revise Line 232 (second column) to include a more detailed explanation of the termination condition implementation, which is currently described in Appendix B.1. The revised sentence will read: “During the training of the low-level policy, the deep latent variable model receives a randomly sampled experience from the training dataset, along with a termination condition vector $\beta$ provided by the state-action novelty module. The model is trained to reconstruct the corresponding action sequence and its length (i.e., the termination point) by maximizing the evidence lower bound (ELBO).”

Q2: Adaptive thresholds for environments

Thank you for your insightful suggestion. In practice, the thresholds we tuned through experiments (Appendix A.1) approximately correspond to the 97th percentile of the novelty values computed over task-agnostic demonstrations—e.g., kitchen = 0.3 (97.47th percentile), maze = 50 (96.86th percentile), and block stacking = 40 (96.12th percentile). We will include this percentile-based guidance in the paper to help readers more easily apply our method across new environments.

Q3: Comparison NBDI to other methods on discrete state/action space

Thank you for your valuable suggestion. To provide an intuitive comparison between NBDI, Relative Novelty[1], and LOVE[2] in discrete settings, we conducted a case study using a grid-based maze environment. This allowed us to directly visualize and compare termination points across methods in a controlled, discrete domain. To align with our task-agnostic setup, we collected diverse expert-level demonstrations from random start and goal positions, and used these datasets to extract skills and termination points. The figure is available at: https://imgur.com/a/aRFrZ30.

[1] defines novelty at a given state as the ratio between the average novelty of future and past states, measured within a fixed-size sliding window (n_lag). A high value indicates that the agent transitions from a familiar region to a less familiar one. Following the original formulation, we only evaluated states with sufficient trajectory context on both sides of the window. As shown in the visualization, Relative Novelty is highly dependent on transition history, often identifying only a subset of bottleneck states.

[2] uses a variational inference framework to extract skills based on the Minimum Description Length (MDL) principle in that its objective is to “effectively compress a sequence of data by factoring out common structure”. While [2] employs a variational inference framework to implement the MDL principle, we used Byte Pair Encoding (BPE) to extract skills in discrete setting as BPE is a specific formulation of MDL [3] and it offers a more intuitive and interpretable formulation of compression. Termination points were visualized based on where the segmented skills terminated during 100 goal-reaching tasks. The results show that terminations vary significantly with the number of trajectories used to extract skills, as [2] focuses on capturing common structure rather than consistent bottlenecks.

NBDI terminates skills based on both conditional action novelty and state novelty (Section 4.1). The visualization demonstrates that NBDI consistently identifies key bottleneck states and exhibits robustness to the number of trajectories collected in contrast to other methods. Moreover, states with high state novelty often correspond to regions that are rare or hard to reach within the task-agnostic dataset. By increasing the decision frequency in such unfamiliar states, NBDI promotes more effective exploration of the state space.

[1] Ş., Özgür, and A. Barto. "Using relative novelty to identify useful temporal abstractions in reinforcement learning." Proceedings of the twenty-first international conference on Machine learning. 2004.

[2] J., Yiding, et al. "Learning options via compression." Advances in Neural Information Processing Systems 35 (2022): 21184-21199.

[3] G., Matthias. "Investigating the effectiveness of BPE: The power of shorter sequences." Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.

We thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.

Q1: Description of the train/transfer environments

In Appendix E, we introduced the train/transfer domain similarity metrics for the environments used in our experiments. To demonstrate the effectiveness of our method in challenging downstream tasks involving significant configuration changes, we tested on a maze with an unseen layout and a larger-scale block stacking environment with more blocks in random positions —both of which differ significantly from the task-agnostic offline datasets. The significant performance improvements reported in Section 6 clearly demonstrate the effectiveness of our method.

Q2: Confusing expression

We apologize for the confusion caused by our wording. We agree that the term "finding" may have been misleading in this context. We will revise the word to “a novel method”.

Q3: Applying NBDI to other skill-based approaches

Thank you for your constructive suggestion regarding the experiments. Our work focuses on extracting meaningful termination points from task-agnostic demonstrations. To explore the broader applicability of our approach, we investigated its compatibility with a skill-based approach that leverages task-agnostic demonstrations to solve challenging long-horizon, sparse-reward meta-RL tasks [1]. Specifically, we applied our method during the skill extraction phase of [1] to learn variable-length skills in place of fixed-length skills.

In the maze environment, we randomly sampled 10 goal locations for meta-training. The table below compares the success rate of meta-policies trained with [1] (fixed-length skills) and our method during the meta-training phase across episodes. The result shows that the extracted variable-length skills allows the meta-policy to better promote knowledge transfer between different tasks, helping the meta-policy in combining the skills to complete complex tasks. We report mean success rates on the 10 meta-training tasks across 3 different seeds with standard errors.

Furthermore, during the target task learning phase, the meta-policy learned through our approach leads to significantly better sample efficiency on the unseen target task. These results indicate that our approach can be effectively integrated with a broader class of skill-based methods that leverage task-agnostic demonstrations (3 seeds).

Q4: The trajectory collecting policy

As our work primarily focuses on learning termination conditions from task-agnostic demonstrations, we will revise Line 264–268 and 318 to clarify that we assume access to task-agnostic, expert-level demonstrations. In Section 6.5, we analyzed the impact of dataset quality on our method’s performance. The results indicate that the level of stochasticity and the suboptimality of the dataset influence the effectiveness of our approach, which remains a limitation of our work.

Q5: Skill length impact on baselines

Due to space limitations, we kindly refer you to our response to Reviewer bZqs’s Q2, where we provide a detailed discussion.

Q6: Details in environment settings

We apologize for not including information about the environments (Appendix E, F) in the main paper. In the kitchen environment, the agent completes an unseen sequence of object manipulation subtasks, like opening the cabinet or turning on the oven. In the maze environment, it receives a binary reward for reaching the goal. In the sparse block stacking environment, the agent is rewarded based on the height of successfully stacked blocks at the end of the episode. We will include a brief introduction to the environments in the experiment section.

Q7: Other comments and suggestions

(Line 185-190) We apologize for the confusion. Since the statement in Line 185–190 is based on Figure 3, we will move it to the paragraph where Figure 3 is discussed in detail.

(Figure 5) Figure 5(a) and 5(b) illustrate the performance of the baselines and our method on a challenging downstream maze task, which involves reaching the goal in a maze with an unseen layout (Appendix E). Due to the complexity of this task, performance is highly sensitive to random seeds, leading to generally lower success rates compared to previously reported results (Appendix F). In the sparse block stacking environment, the maximum number of blocks that can be stacked is five.

[1] N., Taewook, et al. "Skill-based meta-reinforcement learning." ICLR, 2022.