SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions

We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our idea of learning skills to induce diverse interactions between state factors novel and our evaluation sound.

Please find below our responses to address your concerns regarding our work:

Regarding reorganizing Sec 3

We appreciate the reviewer's suggestions, and we will rearrange these sections for greater clarity.

We will also add more detail to the high-level description at the beginning of the section to make it clear what the high-level and low-level inputs are:

• the high-level policy outputs graphs and diversity parameters.
• the low-level skills take in both and output primitive actions.

Regarding details on dynamics model training, when and how are diversity rewards applied, what exactly are the input/output features of the graph selection policy?

We appreciate this comment and will add the pseudocode to the main paper (it is also attached to the global response). We will also add the appendix section addressing the following components,

• Loss functions for dynamics model: line 167 mentions that the loss is the prediction error.

  Specifically, we measure it as cross-entropy loss for discrete state space and mean squared error for continuous state space.

• When and how are diversity rewards applied: as shown in Eq. 3, when the induced graph matches the desired graph, the diversity is applied as an additional reward to the graph reward.

• The input/output features of the graph selection policy:

  • As shown in Fig 2 and line 186, the input is the current state
  • The output feature is a categorical distribution over N, where N represents the number of unique graphs in the history of seen graphs.

High-level details about environment assumptions (learned/given dynamics, what is given in the graph factorization, etc.) in section 4.1 should be given, even if details are already in the appendix.

We agree with this constructive feedback and have added this information to Sec 4.1.

Why do all methods need so many timesteps?

Though mini-behavior looks simple, the environments are challenging because of the following factors:

• The state factors are highly sequentially interdependent, making skill learning and task learning challenging: for example, in cleaning car environments, the agent can’t clean the car until it picks up the rag, turns on the sink, and soaks the rag. These interdependencies between state factors pose great challenges to the agent’s exploration ability.
• During the skill-learning stage, we would like the agents to learn all possible skills (e.g., manipulating all objects) rather than learning to manipulate a single object. Hence, it requires more samples. Furthermore, Fig. 4 shows that, with 10M timesteps, baselines still fail to learn to manipulate all objects.
• During task learning, we use sparse rewards, and thus it is further challenging for agents to explore. Our results match the results in the mini-behavior paper, which shows that it takes millions of timesteps to learn the task with sparse rewards.
• In addition, we use primitive actions, and many actions have no effect if it’s not applicable in the current state. So the task is especially challenging for exploration.

Regarding hyperparameters tuning.

We apply grid search on the following hyperparameters for the following methods to ensure their best performance. We will add these details to the paper Appendix. Due to space limits, we leave the values we searched in the Appendix.

• SkiLD (ours):
  • skill-learning:
    • low-level policy: exploration noise, skill horizon
    • high-level policy: entropy coefficient
  • task learning: skill selection policy’s entropy coefficient
• DIAYN:
  • skill-learning: low-level policy exploration noise
  • task learning: skill selection policy’s entropy coefficient
• CSD:
  • skill-learning: low-level policy exploration noise, intrinsic reward coefficient, learning rate
  • task learning: skill selection policy’s entropy coefficient, skill horizon
• ELDEN:
  • dynamics model: regularization lambda and its annealing, gradient cutoff threshold
  • task policy: entropy coefficient
• COInS:
  • Dynamics model:
    • Passive and active model cutoff thresholds
    • Interaction state reweighting ratio
  • Granger-causal score threshold
  • Skill learning:
    • Goal sampling and relative action sampling distance
    • Hindsight reweighting ratio
• Vanilla RL (PPO): entropy coefficient, learning rate, batch size

Regarding meaningfulness of learned skills.

Fig 4 in the paper (full version: Fig 1 of the global response) shows the semantic meaningfulness of skills acquired by different methods (in terms of which interactions they can induce).

Specifically, for each method, we randomly sample its learned skills (see details of # of skills below) and we record which interactions they induce in each episode. From the global response Fig 1, we can see that

• SkiLD induces all inducible graphs, suggesting that it learns skills that cover all possible interactions with objects (i.e., at least 15 skills are semantically meaningful), though there may also exist redundant or less meaningful skills.
• In contrast, for DIAYN, with a similar number of skills, most interactions it induces are agent moving (i.e., agent, action -> agent) and agent getting blocked by other objects (e.g., agent, car -> agent). It suggests that a large portion of skills are navigation skills, and they are less meaningful when solving tasks requiring inducing object interactions.

the number of skills:

• SkiLD (ours): in the Cleaning Car environment, we uniformly sample from the history of 15 seen graphs and 4 discrete diversity variables: leading to 60 discrete skills
• DIAYN: we uniformly sample from 64 discrete skills.

Regarding typos

We appreciate the detailed feedback and have corrected these typos.

Thanks for the detailed response.

Due to the neurips rebuttal format, I can't verify the suggested changes will be made, so I can only partially consider those in whether to increase the score. However, the plan makes sense. I can reconsider if the authors explicitly list out quotes for what will be changed.

Though mini-behavior looks simple, the environments are challenging because of the following factors:

This is a reasonable description, thank you. This information however is not presented in the actual paper, with the only hint at this being: "While conceptually simple, this domain has been shown to be extremely challenging for Vanilla RL."

Fig 4 in the paper (full version: Fig 1 of the global response) shows the semantic meaningfulness of skills acquired by different methods (in terms of which interactions they can induce).

This doesn't fully answer my question (though it is a helpful visualization) of "what proportion of skills end up being meaningful" as it's easier to induce these interactions in a discrete environment over a long horizon of 500 steps. Randomly sampling skills and visualizing their behavior, truly at random, in the Gibson environment (cont. control) would be a better comparison, or perhaps plotting a graph of the state distribution of the skills (again sampled at random) against x-y positions in the minigrid environments to get more information about what the policy is doing with each skill.

I have currently raised my score in response to the other parts of the rebuttal.

We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our method well-motivated and our presentation clear.

Please find below our responses to address your concerns regarding our work:

Regarding the use of overexaggerations and mismatch in some claims and with respect to the experimental results

We appreciate the reviewer’s observation and will make sure to eliminate all overexaggerations. We have made adjustments to the language of the paper to better match the empirical results. In particular, we have removed any mention of “large” number of state factors or “robustness”, as well as other possible exaggerations such as “object-rich” or “realistic”, and replace with more precise language such as object-factored and 3D real-world-based simulation.

Regarding Q1 evaluation

We appreciate this insightful comment and will incorporate the following clarifications into the paper.

Due to space limitations, we deliberately only show task-relevant interactions in Figure 4, since these interactions are more informative. That being said, we agree with the reviewer that a general metric comparing all found interactions against all inducible interactions will better showcase the diversity of interactions for different methods. To this end, we additionally show all inducible interactions and whether each interaction is induced, in the global response Fig 1.

The diversity metric would be (# induced interactions) / (# induciable interactions). As shown in global response Fig 1, SkiLD induces all inducible interactions, while CSD only induces 80% and DIAYN only induces 60% of all inducible interactions.

Regarding the number of skills, we will add the following clarification:

• SkiLD (ours): in the Cleaning Car environment, we uniformly sample from the history of 15 seen graphs and 4 discrete diversity variables: leading to 60 discrete skills
• DIAYN: we uniformly sample from 64 discrete skills
• CSD: CSD uses continuous skill space, so there are an infinite number of skills.

Regarding efficiency in Q2

We measure sample efficiency in terms of performance after a certain number of environment transitions. We appreciate this comment and will add this description to the paper.

Regarding details, modules, and methods

We appreciate this comment and will add the following details to the manuscript. We also attach the pseudocode in the global response Alg. 1 to improve the clarity of the workflow.

• Regarding the events of interest, it is known from the collected transitions.
• Regarding the target dependency graph, it is sampled from the high-level policy and known.
• Regarding HER, following the original paper, we apply it to 87.5% of sampled trajectories, and we will add this hyperparameter to the paper.
• Regarding the diversity variable,
  • Fig 1 mentions that the skill has two goals: (1) inducing the target dependency graph, and (2) inducing the graph in diverse ways (i.e., reaching different states).
  • Lines 173 - 175 mention that the diversity variable fulfills the second goal, in the same way as DIAYN – the agent will be rewarded to visit states if the diversity variable can be distinguished from the states. Hence, the agent is trained to visit different distinguishable states under different diversity variables.
• Regarding the flow of the skill learning phase: we describe it in the pseudocode in the global response and will include it in the paper.

Regarding other metrics

We appreciate this insightful comment and have added IQM scores to the global response.

Regarding varying numbers of objects

Please note that for the scope of this paper, we only consider MDPs, which are fully observable by definition, with a known state space (and therefore a known number of objects)).

To handle varying numbers of objects, one possible extension is to pre-assign a large number of state factors and use placeholders for objects that are not present. The assumption is that objects that are not present do not involve interactions, and their states do not change.

Regarding allowing for ‘infinite’ number of skills and scaling to it

A simple way to extend to an ‘infinite’ number of skills is to have the diversity variable be continuous instead of discrete. As the number of skills increases, the capacity of the agent’s behavior grows proportionally, allowing for more diverse skills.

In the meantime, it will also be harder for the skill policy and the discriminator to process these skills, potentially hampering skill learning. Finding the “right” number of skills is often a domain-specific problem

Regarding two learning phases

Each phase is done once.

Regarding the action space

It is defined over the full state space.

Regarding the factorized lower-level policy

Each skill is realized by one policy network.

• As described in Appendix Sec A, there are N (N: # of state factors) parameterized low-level networks, one per state factor.
• When the target dependency graph is specified, we will identify which state factor should be influenced and use its corresponding policy to sample primitive actions
  • As a result, only one of the N networks is activated at every time step.
• The task policy still selects the target graph from the history of seen graphs and thus is unaffected by this specific design of low-level policy.

Regarding Fig 4 notations,

$x,y \\rightarrow x$ represents that $x_{t+1}$ locally depends on $x_t$ and $y_t$.

Regarding ‘state-specific’ dependency graph

A general dependency does not necessarily happen in every state. For example, though a knife can cut fruit, when they are far away, the cutting will not happen. The state-specific dependency graph encodes this state-specific dependency between factors, the edge represents the dependency happening in a given state.

Regarding Fig 6

It is also mean and standard deviation, and we will add that clarification to the final paper.

We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our presentation clear and our evaluation sound.

Please find below our responses to address your concerns regarding our work:

The method's reliance on accurately detecting and modeling local dependencies adds a layer of complexity that may limit its applicability.

With respect to detecting and modeling local dependencies, two possible avenues offer promising directions:

• Relaxing the dependence of precise local-dependency identification by using a window of states rather than a per-state reward when incorporating dependency information.
• Since many state-specific dependencies fall into broad categories (contact, co-occurrence, etc.), employing a meta-learning strategy for identifying local dependencies in a lifelong setting can improve the effectiveness of dependency identification.

The effectiveness of SkiLD hinges on the availability of a factored state space, which may not always be available or easily obtainable.

With respect to factored states, we agree that SkiLD requires a factored state space. In the meantime, we believe that advances in vision, especially related work such as SAM [1] and other object-centric modeling [2] offer a path forward for extracting a factored state space from a dense state such as pixels, thereby extending the application scope of SkiLD.

[1] Ravi, Nikhila, et al. "SAM 2: Segment Anything in Images and Videos." arXiv preprint arXiv:2408.00714 (2024).

[2] Aydemir, Görkay, Weidi Xie, and Fatma Guney. "Self-supervised object-centric learning for videos." Advances in Neural Information Processing Systems 36 (2024).

Regarding the usage of symbols

We appreciate the detailed feedback and will make sure we use the correct symbol in the paper!

Regarding the definition of state-specific dependency graphs

We appreciate this insightful comment and will add definitions of state-specific dependency graphs to section 2.2 to help it better integrate into section 3.

We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our idea of learning skills to induce diverse interactions between state factors novel and our evaluation sound.

Please find below our responses to address your concerns regarding our work:

Assumption of Factored State Space: SkiLD assumes access to a factored state space, which may not always be available or easy to obtain in real-world applications.

With the recent advances in object segmentation and object-centric representation techniques, such as SAM [1] and SOLV [2], one potential way to construct the factored state space is to use these models to extract factored representation from real-world image observations.

We agree that testing the effectiveness of SkiLD with learned factored representations would be an interesting next step, and will add it to the future work section of the next version of this paper.

[1] Ravi, Nikhila, et al. "SAM 2: Segment Anything in Images and Videos." arXiv preprint arXiv:2408.00714 (2024). [2] Aydemir, Görkay, Weidi Xie, and Fatma Guney. "Self-supervised object-centric learning for videos." Advances in Neural Information Processing Systems 36 (2024).

Evaluation on Limited Domains: The evaluation domains are somewhat restricted. Broader evaluation across more commonly used environments could emphasize the method's performance. Evaluating environments like Crafter, 2D Minecraft, or manipulation environments with multiple objects would better illustrate the benefits of focusing on local dependencies. Scalability: There is a need to explore how SkiLD scales with an increasing number of state factors, which is crucial for practical applications.

We appreciate the reviewer’s suggestions and agree that further empirical testing in other commonly used or larger-scale settings is an important direction for future work.

Meanwhile, we emphasize that, while exploring scaling further is valuable, we believe that it will not fundamentally change the scientific insights we introduced – compared to prior works in unsupervised skill discovery, by focusing on local dependencies between state factors, SkiLD learns skills that induce more diverse interactions and enable more sample-efficient downstream task learning, for the following two reasons:

1. The environments used in the paper are also challenging and baselines fail to learn diverse skills and solve downstream tasks.

   • Specifically, the mini-behavior environments have the same inter-dependency between state factors as 2D Minecraft (e.g., in Cleaning Car, the agent cannot clean the car until it soaks the rag in the sink).
   • For the iGibson environment, despite the four task-relevant factors, it also contains many intractable objects such as jars, microwaves, garbage cans, etc.

   In contrast, most prior works in unsupervised discovery are only evaluated in environments with one or two state factors (where skills are limited to moving the agent to different locations). As a result, as shown in Fig 4 and 5, when evaluated on Mini-behavior and iGibson, baselines fail to learn diverse skills and solve downstream tasks. Compared to them, SkiLD learns to induce those complex inter-dependency graphs and solves tasks successfully.

2. In principle, nothing in our method prevents it from scaling to more state factors.

   • For methods that focus on reaching diverse states (like DIAYN), due to the exponentially growing state space, it would be more challenging to learn to induce meaningful interactions.
   • In contrast, the number of inducible interactions typically increases much more slowly. As a result, by focusing on inducing diverse interactions, our method in principle has larger advantages than existing methods.