Learning Transferable Sub-goals by Hypothesizing Generalizing Features

Thank you for your review. We are glad to see that you deeply understood our paper. We plan to revise the paper with a mathematical definition of a hypothesis as well as clarify some of our experiment decisions.

I am very interested in your formulation of “hypothesis” but at the same time was disappointed for not seeing enough explanation or theorem that covers your idea. How did you exactly formulate “hypothesis” in your code? Can you provide any theorem and/or math notations that support your formulation?

We plan to update the paper with a formal definition. Given a dataset D we can define a space of functions f that accurately fit the data D. Our hypotheses are sampled from this space which is very large as there are many ways to fit a finite collection of points. Because a neural network is approximating a function, we represent each hypothesis as an individual neural network.

In addition, no pseudocode was provided nor any clear statements about the connection with The Diversity-By-disAgreement Training (D-BAT) algorithm. More discussion on these areas would be much appreciated.

D-BAT uses unlabeled data to sample a set of diverse functions from our hypothesis space. This is done by requiring that each function disagree on a set of unlabeled data. This disagreement ensures each hypothesis is varied in a significant way while maintaining consistency with the labelled dataset which is why we chose to leverage D-BAT.

In what respect does your use of representation learning differ from using dimension reduction?

When performing dimensionality reduction you are always discarding information. A good reduction would remove irrelevant information while a bad reduction would also discard pertinent information. If you have a single subgoal to generalize from, it is not clear what features should be removed and which should be kept. By learning a diverse ensemble, each ensemble member discards different parts of information and so is performing a kind of dimensionality reduction.

How do you exactly provide the high-level policy?

Our high-level policy is a PPO algorithm that receives the full state and must select an action, each of which is a low-level policy that is tasked with satisfying a single subgoal from the ensemble of classifiers. In the MiniGrid experiment, we started with 5 options and used a 3 head ensemble for each. The high-level policy had an action space of 15 actions, one action for each ensemble member for all 5 ensembles.

In Figure 3, you mentioned that “this performance is on a collected set of data and does not fully encompass all the possible states an agent may encounter during policy learning.” How would the result differ from not using a collected set of data or any suggestions to overcome this?

In the Montezuma’sRevenge experiment we use a state composed of a stack of the previous 4 frames. As a result our state is now a factor of the previous 4 actions, atari has 18 possible actions, so there are a large number of possible states. We used expert collected trajectories for our data and so did not cover all possible states for every ClimbDownLadder subgoal. If this data was coming from an existing option, it is likely that there would be larger variation in these states as the policy executes random actions during its initial training. However, this cannot be guaranteed if we are allowing the agent to autonomously discover and train different skills and subgoals.

Why did you use the PPO and DQN agents to compare with? Similarly, why did you choose the CNN classifiers to compare with? A bit more discussion on these choices would be helpful.

We agree that more discussion on these choices is needed and will be adding our motivations in the paper revision. Our ensemble agent uses DQN for the lower-level option policies and PPO for the higher-level policy. Our motivation for testing against these algorithms was to definitively show that the performance of the ensemble agent was not due to the power of these deep RL methods but instead our transferred subgoals.

For the Montezuma’sRevenge experiments we choose to test against a CNN because this is traditionally how subgoals would be defined and so clearly shows how our method performs against current techniques as well as highlighting the difficulties for current CNN based methods.

We would again like to thank the reviewer for their thoughtful review and will address their concerns in our revision. We are happy to address any further questions or concerns.

Dear reviewer,

We have updated all experiments to be averaged over 10 seeds.

We believe we have addressed all your concerns and are eager to engage with you on any further points or questions you may have!

Thank you for critical and insightful comments that expose weakness in our communication. It has been clearly stated that

The goal is to provide a scalable method to learn transferable options, and the claims are that no HRL method does this

We agree that we have not achieved these goals. However, this was not the goal of our research, which we will make clear in our revision of the paper. Rather, our goal is to generalize subgoals from one previously given option instance. We agree that we do not perform goal discovery (c and f) because we intend to work with existing option discovery methods, which identify single, unique subgoals which are not reusable outside their discovering instance. This is a result of two problems a) the subgoal is defined by a collection of states, labeled as either inside or outside the subgoal which can only be sampled from the initial discovered subgoal leading to a very small dataset with very little diversity/coverage, and b) this dataset of samples does not provide the ability to discern features important for the subgoal as this is fundamentally underspecified (we argue this point with an illustrating example in section 3.2).

The review correctly points out existing works that tackle skill generalization. These useful references will be added to our Learning transferable skills related works section. The works mentioned (a,b,d,e,g) as well as the works already referenced, approach skill generalization retroactively; these works either require large libraries of skills that can be compressed or changed to form generalized skills (b) or assume sampling from a distribution of tasks which implicitly assumes an expert has curated a collection that encompasses what the agent must learn (a,d,e,g). Our goal is instead to approach skill generalization in the forward direction, i.e. if we are given one example of the skill (which is what most subgoal discovery algorithms provide) how do we generalize this skill to future tasks which are likely in never-before seen states? If we wish to make use of the rich body of work on subgoal discovery, it is important to be able to generalize these skills from the single instance in which they are discovered by these algorithms.

The representation learning work again differs from our scenario as it uses full episode trajectories to learn representations that allow for better learning by removing irrelevant features. However, from one example of a subgoal it is not possible to decide which features are truly irrelevant. Especially when considering the fact that what may be irrelevant to an agent (such as the color of an object during object manipulation) may not be irrelevant to a subgoal classifier. Thank you for bringing these works to our attention and will add them to our related works section.

There is no mechanism to discover the goals. The images used to train DBAT were hand-picked. How can this process work in more general settings?

We hand-picked subgoals since our aim is to evaluate how well our method can generalize which requires knowing what our ensemble must generalize to. However the data used to train the D-BAT classifiers are what would typically be generated by an off-the-shelf skill discovery algorithm.

It is necessary to train preliminary options to generate rich enough data to train the ensemble classifiers. Where do these options come from?

Since we intend to work in tandem with existing subgoal discovery methods, these preliminary options can come from any subgoal discovery method that generates subgoals.

How can we guarantee that they will generate rich enough data?

The core problem is that we cannot guarantee this. In fact, we can almost always guarantee the opposite: since we are gaining data from a small collection of states, it is not rich enough to effectively train a traditional classifier to generalize. This is why we leverage multiple diverse classifiers which each attend to separate features from the provided data.

Having those options already, why bother to use a classifier to learn options that carry out the same task as the original options?

These subgoals all represent options that we want the agent to continue to use. We define these subgoals only for seed=0, a single possible Minigrid DoorKey configuration. If we change the seed however, we want the agent to continue to use these options.

The way in which the option policies are obtained from the classifiers is unclear and possibly does not scale to more general settings

We learn a policy for each ensemble member. The policy is given a reward of 1 if the relevant ensemble member returns 1 and 0 otherwise.

Dear reviewer,

We have updated all experiments to be averaged over 10 seeds.

We believe we have addressed all your concerns and are eager to engage with you on any further points or questions you may have!

First of all, I want to thank the authors for replying to all the questions posed in my reply. They gave me some clarity.

Second, I will explain my current decision regarding the score I assign to the paper and then provide some actionable suggestions.

Decision:

In my previous review, I identified a goal and two claims in the paper and I explained why in my opinion this goal is not satisfied and why the claims are not supported by compelling arguments or evidence. In response, the authors' reply that the identification of the goal is incorrect. They state that the goal is not to provide a method to learn transferable options, but instead a method that generalizes “subgoals from one previously given option instance”. However, I fail to clearly see the distinction between the two goals and, most importantly, I do not think the revised version makes any distinction between these different goals. In particular, the text says that “However, existing HRL generalization methods are unable to generalize an option from a single context to another.” This sentence seems to point out that the problem with previous techniques is that options learned are not transferable. This, to me, is exactly the goal that I previously identified and, given that neither the arguments nor the experiments changed, my score cannot change.

That being said, I feel that I need to explain myself better. In particular, I will explain in what follows why I think the two goals are essentially the same, why the paper provides an interesting but neither novel nor complete solution, why it does not do any relevant comparison with similar techniques, and why it misses important details of previous work.

Thank you for your thoughtful review. We agree that our presentation can be improved. In particular, we should first clarify that our work is not a new method for skill discovery. It is a method to generalize subgoals and so should be used in tandem with a skill discovery method.

I don't think that three seeds per curve is sufficient to reasonably compare the methods

We are actively running more seeds and will update all graphs with additional seeds.

can you explain the baseline how the accuracy is calculated in section 4.1?

For evaluation, we collected data from across Montezuma’sRevenge and labeled which states lie within and outside the subgoal. We calculate our accuracy against these expert labeled samples.

I'm also not sure I agree with the claim that the performance when only one ladder instance has been seen is the most important given how low the accuracy is for all three models at that point compared to the two-ladder case.

The goal of our work is to generalize a subgoal from a single example, which is the expected output of an off-the-shelf subgoal discovery algorithm. As such, illustrating how well our method performs for the one-ladder case indicates how well the method works if used along-side a subgoal discovery method. Although 70% accuracy may appear to be low at first glance, especially when compared to the near 100% accuracy achieved after two input examples, it means that we detect previously unseen states that fall inside our ClimbDownLadder subgoal—each detection means a new skill need not be discovered from scratch, but can simply be instantiated as a generalization.

This environment is not so large that DQN/PPO should be unable to succeed when trained over 1.5 million steps, so it makes me think there is an error in the evaluation.

The failure of the baselines is due to the episode length of 500 steps, which makes it a more challenging exploration problem. If we increase the episode length to 1000 steps, both DQN and PPO solve the task. We chose these baselines because our hierarchical agent uses DQN for its lower level policies and PPO is used for the higher-level policy. Our intention is to convincingly show that our agent learned the task not because of the power of these deep RL methods but because of the subgoals provided by our ensembles. We will clarify this argument in the revision to our paper.

can you better explain the training process for the MiniGrid environment in section 4.3?

We collect labeled data from a single seed (seed=0) and unlabeled data is collected from seeds 1 and 2. We then evaluate the agent in unseen seeds (10, 11, 12). When training starts, the ensembles have seen only data from seeds 0, 1 and 2. The additional states seen in the testing seeds are not shared across the ensemble agents. This was not clearly communicated in the current paper. We can address this completely with a careful experiment description, in revisions we will implement.

In Figure 6, why is the standard error shown instead of standard deviation?

Standard error is standard deviation divided by the square root of the number of runs; this is a standard metric, but we can report standard deviation if you think that will substantially increase the quality of the paper.

Thank you again for your thoughtful review. The points you raise will be reflected in our revisions and we welcome any additional questions.

Dear reviewer,

We have updated all experiments to be averaged over 10 seeds.

We believe we have addressed all your concerns and are eager to engage with you on any further points or questions you may have!

Thank you for your comments which expose a weakness in our presentation which we will address. Specifically, the goal of the paper could be misconstrued as applying D-BAT to the hierarchical reinforcement learning setting. This would indeed limit the novelty significantly to the D-BAT method which would not justify publication. We want to emphasize that this is not the main goal of our work. Current subgoal discovery methods (which are the vast majority of skill discovery methods) identify subgoals which are considered standalone and unique. Current skill generalization methods take a retroactive approach, by either assuming a large library of learnt skills which are then compressed into a small collection of generalized skills or assuming the agent has access to a distribution of tasks to sample from during training. This latter case implicitly assumes an expert has informed the agent of what skills it must have. This retroactive view means current skill generalization techniques are not compatible with the rich collection of skill discovery algorithms the community has developed.

We take a forward looking approach, generalizing a unique subgoal, which can be provided by any subgoal-based skill discovery method, to unseen states. This leaves two main problems 1) we have a subgoal described using a small labeled dataset, but with very low diversity/coverage, and 2) based purely on one instance, the true intention of the subgoal is unclear (we argue this with an illustrating example in section 3.2). To overcome these challenges, we propose learning an ensemble of classifiers, each achieving high accuracy on the training dataset, but attending to separate features of the data. We make use of D-BAT because it is the only method we found capable of creating multiple classifiers that vary meaningfully on out-of-distribution data with their use of unlabeled data. So, while D-BAT can be regarded as a robustness algorithm, we are not using it to increase robustness of a provided subgoal, but instead to create a collection of subgoals based on a sufficiently descriptive subset of the data. As such, our method is not tied to the D-BAT method and any algorithm that produces a set of diverse classifiers can replace D-BAT.

Each experiment was carefully selected. For our subgoal classifier evaluation, we compare to a CNN because this is traditionally how subgoals are represented in the vast majority of GCRL and HRL works. In the MiniGrid-DoorKey experiment, we show two things: (1) giving the agent skills that are not aligned with the overall task will not irreparably damage policy learning, and (2) the strong performance of our ensemble agent cannot be explained by our use of DQN (for learning the policy over options) and PPO (for learning low-level option policies).

As the review report correctly indicates, this is not communicated clearly in the paper. We did not compare to state-of-the-art skill discovery methods because our methodology can be applied with any subgoal based skill discovery method. We did not compare against any skill generalization techniques because these methods either assume prior knowledge of the collection of tasks the skill must generalize to or they assume a large library of prior learnt skills. These would therefore not be a fair comparison.

We will improve the current draft of the paper by a) removing the overgeneralizations of HRL b) better communicating the goal of our work and our algorithm c) better explaining the selection of comparisons in our experiments.

Thank you again for a thoughtful review, we are happy to address any additional concerns.

Dear reviewer,

We have updated all experiments to be averaged over 10 seeds.

We believe we have addressed all your concerns and are eager to engage with you on any further points or questions you may have!

Looking over the updated paper, while the addition of seeds is important, the limitations remain. In particular, there is still an absence of rigor related to the hypothesis formulation and how it is precisely applied to HRL. It is still entirely unclear because of this lack of formalism what the inputs or outputs are, and these must be interpolated from the related work, rather than from the paper itself. Speaking of this, the paper continues to not be self-contained, without a formal description of D-BAT prior to it's inclusion in the methods. Next, the notable absence of competitive HRL baselines (The argument that these methods are not directly comparable does not mean that they cannot be compared, and further suggests that a comparable baseline should be used. Furthermore, if the claim is that this method is strictly tangential, then experiments should be run to indicate that it can actually augment existing methods, and a comparison against the un-augmented version). Finally, the edits to the paper are not sufficiently notable to improve the quality of the paper---it remains difficult to parse.

For these reasons I will maintain that the paper should be rejected.