A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

Dear Reviewer,

Thank you for the time and effort put into this review, as well as the suggestions for improving the paper. It seems that the main inquiries relate to further analysis of skill learning. To address this concern, we have added new analysis on policy rollouts (see new Fig. 15) as suggested by the reviewer. We have also highlighted unique skill learning in a new, highly-difficult environment (see new Fig. 14) and added additional analysis on representation probing (see new Fig. 17). Do the new analysis and experiments address the reviewer's concerns about the paper? We look forward to continuing the discussion!

analysis of emergent skill learning: perform clustering on policy rollouts data to identify distinct exploratory behaviours as learning progresses.

Thank you for the suggestion! We have performed this clustering and added the visualization to Appendix D (see new Fig. 15). We observe that the action and state rollouts for early checkpoints (2,3,4) tend to be more spatially separated in the PCA and T-SNE plots, indicating unique exploratory behavior and skill learning, whereas the rollouts for later checkpoints (8,9,10) have more overlap, indicating the agent has converged to similar behavior patterns once the single-goal has been reached.

Would like to see further analysis on why this strategy works

As suggested by the reviewer, we did further analysis of the representations to understand what state features they were picking up on, and how this changed throughout the course of learning. We find that the representations are more sensitive to perturbations along “easier” state dimensions during early training and more sensitive to perturbations in “harder” state dimensions during later training stages (see new Fig. 17 in Appendix F), suggesting that the method automatically develops a reasonable training curriculum.

While our paper is empirical, we are also very interested in, and have started investigating, the possible theoretical reasons why this method works so well. We speculate that there might be something to do with the connection between CRL and the graph Laplacian [1, 2], which could allow CRL representations to capture the large-scale geometry of the environment and drive directed exploration [3]. If the reviewer has suggestions on theoretical approaches, we'd love to continue the discussion!

Would be interesting to see if this approach results in exploration strategies that are capable of solving more complex tasks (where naive subgoal design may not work well)

Thank you for the suggestion! In addition to the current tasks, which are already quite challenging (see SAC+HER and RIS in Fig. 9), we have added a new environment (sawyer box hard) in which the agent must place a block in the box and then place the lid on top of the box. This environment is especially difficult because the required behavior is non-linear and requires multi-step planning. While single-goal CRL was not able to consistently solve this environment (success rate < 0.05), we observed unexpected emergent skills during training which were not observed in the simpler environments. We showcase a few of these emergent skills in a video uploaded to the anonymous code repository, and we have added a section about this environment in Appendix C (see new Fig. 14). We believe that this behavior further supports our main contribution that single-goal CRL drives exploration and skill learning, though it has not solved this difficult, multi-step task.

It would be interesting to understand how well this approach can be used to learn value representations that generalize well across task environments … Can the learned representations be leveraged to address the issue of sample efficiency when learning multiple tasks?

To address this question, we are currently running an experiment where we train the agent using single-goal CRL to place a block in the box. Taking that trained model, we will then train it on the sawyer box (box closing) task and observe whether there is improvement in sample efficiency. If so, this experiment would indicate that skills learned by single-goal CRL are transferable to new tasks.

Were there any exceptions to the skill learning trends identified in (1) (e.g. checkpoints with purely random behaviours) and if so how frequent were exceptions?

For all 15 random seeds that we have visualized, we always see a similar progression of skill learning. The time at which each skill is learned differs across seeds. We did not observe any seeds that behaved randomly all throughout training. Additionally, some seeds converge to different ways of solving the task (see Fig. 7). We have uploaded training videos for each environment to the anonymous code repository. These training videos correspond to one seed for each of the three sawyer environments, but we are happy to provide training videos for more seeds if requested.

I also wished to ask when you mention fixed sampling intervals for checkpoints, what fixed interval was used?

The checkpoints were saved at fixed time intervals (e.g. every 15 minutes), which corresponded to ~3,300 trials in the environment. We have added this detail to Appendix D. The videos we uploaded also have trial numbers to give a sense of how far along in training each skill emerges.

References:

[1] Wu, Y., Tucker, G., & Nachum, O. (2018). The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv:1810.04586.

[2] Erraqabi, A., Machado, M. C., Zhao, M., Sukhbaatar, S., Lazaric, A., Ludovic, D., & Bengio, Y. (2022, August). Temporal abstractions-augmented temporally contrastive learning: An alternative to the Laplacian in RL. In Uncertainty in Artificial Intelligence (pp. 641-651). PMLR.

[3] Machado, M. C., Bellemare, M. G., & Bowling, M. (2017, July). A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning (pp. 2295-2304). PMLR.

Dear Reviewer,

Thank you for the time and effort put into the review. It seems like the main concern is about the difficulty of these tasks. We would like to emphasize that state-of-the-art prior methods not only fail to achieve good performance on these tasks, but often fail to ever reach the goal during training, highlighting the high difficulty of these tasks (see "SAC+HER" and "RIS" in Fig. 9). Second, we would like to clarify that we use "single goal" to mean that the only human supervision to the algorithm is a one goal observation; the reviewer is correct that the algorithm internally uses automatically-sampled goals for training the actor. Together with the new analysis (see new Fig. 16) and discussion below, does this fully address the reviewer's concerns about the paper?

Do the authors have any statistics on eval performance over the same distribution of goals used to generate the replay buffer?

We have run a new experiment to study this question. For single-goal CRL, we generate the replay buffer by commanding the agent with only the single-hard goal during data-collection. For multi-goal CRL, we command the agent with goals along the path to the single-goal state. We have added evaluations for this distribution of goals in Appendix section E (see new Fig. 16), and we find that performance of single-goal CRL on this distribution of multiple goals remains high for the sawyer box and sawyer peg environments. This additional evaluation suggests that single-goal CRL retains its ability to accomplish multiple tasks without being explicitly guided to learn those tasks.

maybe these environments are too easy

The robotic locomotion environments are very difficult for the strong baselines we compared to: SAC and RIS (state-of-the-art at the time of publication). Given the task difficulty, we were also surprised that multi-goal CRL could achieve higher performance with fewer assumptions. We have also run a new experiment to test the limits of the single-goal CRL method. We created a new environment, which we call sawyer box hard, in which the agent must place a block in the box and also place the lid on top of the box (see new Fig. 14). This environment is especially difficult because the required behavior is non-linear and requires multi-step planning. While single-goal CRL was not able to consistently solve this environment (success rate < 0.05), we observed unexpected emergent skills during training which were not observed in the simpler environments. We showcase a few of these emergent skills in a video uploaded to the anonymous code repository, and we have added a section about this environment in Appendix C. These results further supports our main contribution that single-goal CRL drives exploration and skill learning while lifting the burden of sub-goal/reward design.

Not sure how "single goal" the final method is… Figure 10's caption was confusing because the actor loss uses multiple goals, rather than a single goal.

Thanks for the suggestions for improving the presentation, which we have incorporated by revising section 3.2 (orange text). We think that this new presentation does a more effective job of clarifying that the single-goal is used for data-collection and evaluation, but multiple goals sampled from future states in the replay buffer and used for actor training. We have also updated Figure 2 to reflect this clarification. We found that using the single-goal for actor training decreased performance (see Fig. 10), indicating that multi-task learning during actor training is a key component of the method. Using the single-goal for data-collection, however, leads to exploratory behavior without requiring any rewards or curriculum of subgoals.

Dear Reviewer,

Thank you for the detailed review and constructive feedback on the paper. It seems like the main concerns relate to the conclusions from our experimental findings. We believe that the key experimental conclusions are correct; we have added a new experiment (see new Fig. 16) to verify these results, and have also revised the paper (see Sec 3.2, Sec 4.3). Together with the responses below, does this fully address the reviewer's concerns? We look forward to continuing the discussion!

Are there other experiments done which would alleviate concerns that this method is simply overfitting to solve for this one goal?

To study this question, we did additional analysis where we evaluated the policy trained by our method on a range of goals. In particular, we evaluated our method on reaching goals sampled uniformly along the path from the starting point to the single, hard goal state. We find that single-goal CRL retains the ability to accomplish multiple tasks without being explicitly guided to learn those tasks, especially in the sawyer box and sawyer peg environments; the previous multi-goal CRL method required these subgoals to be defined explicitly. We have added this new analysis in appendix section E (see new Fig. 16). This result, in combination with the ablation experiment (Fig. 10), suggest that single-goal CRL is not overfitting its policy parameters on a single task. Instead it performs multi-task learning, which surprisingly improves performance on the single task as well compared to prior methods.

Thank you also for pointing out the discrepancy in our wording. We have updated section 4.3 (orange text) to clarify that single-goal CRL “does not just overfit the policy parameters to reach the single goal, but also learns useful representations for states along the path from the starting point to the goal.” While we don't expect single-goal CRL to succeed consistently on out-of-distribution (OOD) goals (the perturbation experiment shows that encountering OOD states does degrade performance), we find that it can reach in-distribution goals that it has seen during training.

Unclear how the authors selected or defined “difficult goals”

We have revised section 3.2 (orange text) to clarify how we select the one goal per task. Our environments all have natural semantically meaningful goal states — closing a box, inserting a peg into a narrow hole, or reaching the end of a maze. For each task, we defined the difficult goal as the respective environment coordinates for those end states. Previous methods reach this difficult goal with the help of rewards, demonstrations, or subgoals, but our method does not require those assumptions.

Figure 11 (right) might contradict the assertion that the separate representations are important?

We have revised the wording of Sec 4.3 to clarify that this experiment shows that representations could be important for driving exploration, specifically for the single-goal CRL method; this claim does not necessarily apply to the multi-goal CRL. Note that this clarification does not affect the main claim of our paper: that using a single goal for exploration is surprisingly effective and can result in the emergence of skills. We also find the the difference between monolithic critic CRL (Range of Difficulties) in Fig 11 (right) and CRL (Range of Difficulties) in Fig 3 (top right) is not statistically significant (p = 0.096).

I appreciate the care and time from the authors' side to address my concerns, as well as the additional clarity on goal selection. However, my concerns regarding sampling the single hard goal state remain.

It is worth noting that MetaWorld provides a distribution of goal states which can be sampled at the beginning of each episode (e.g. see MT1 [1]), not just a single goal state. The original CRL implementation (which the provided anonymous code seems to build on) seems to implement this as well via self._freeze_rand_vec=False [2].

From the authors' response, it seems this paper uses the environment's "default" goal state for their single goal, rather than sampling from a distribution of possible goal states (i.e. train different single-goal CRL policies on different — but still just one — end goal state).

This paper heavily relies on the assumption that single-goal CRL is better than alternatives for reaching a "single difficult goal." However, in such environments, the default goal state is often not the most difficult goal state from the provided goal state distribution (or even a generally difficult goal state compared to the rest of the goal state distribution).

So, as I mentioned previously, since each task evaluates on only the default environment goal state, it is unclear if single-goal CRL will outperform standard CRL when selecting any possible goal state from the environment, including goal states which might be more difficult than the environment's default goal state.

Finally, in the original CRL paper, standard CRL evaluated on all goal states in the Sawyer Bin environment achieves ~50% success rate after 3e6 steps [3]. Meanwhile, in this work, standard CRL evaluated on the Sawyer Bin default goal state achieves <20% success rate on all seeds after 1.5e7 steps (5x the number of steps). This indicates to me there are certain goal states — which may be more or less difficult than the default goal state — where standard CRL would perform similarly (or potentially better) compared to single-goal CRL.

Although the authors provide an experiment in Fig. 19 which shows similar success rate on an 'easy' goal, this does not assuage my concerns that we do not know the general performance of single-goal CRL on the environment's provided goal state distribution, including potentially more difficult goal states.

---

In summary, I believe the authors' choice to use the default environment goal state as the goal state used for training/evaluation to be flawed because 1) this is not necessarily the most difficult goal state (or even a generally difficult goal state compared to the rest of the goal state distribution) for each MetaWorld environment, and 2) it is difficult to know how single-goal CRL would perform compared to standard CRL when trained on any particular goal state sampled from the environments' provided goal state distribution (including more difficult goals).

While these experiments on the default environment goal state show promise compared to other RL methods, I believe using only the default goal state for training/eval rather than training/evaluating single-goal CRL policies on multiple goal states from the environment's provided distribution (including ones more or less difficult than the default) is a fundamental flaw of this paper. Since this paper is mostly an empirical analysis, I find this flaw in experimental methodology quite important. Hence, I am choosing to decrease my score to 3, reject.

[1] https://meta-world.github.io

[2] https://github.com/rafapi/contrastive_rl/blob/main/env_utils.py

[3] Eysenbach, Benjamin et al,. "Contrastive Learning as Goal-Conditioned Reinforcement Learning." NeurIPS 2022.

I thank the authors for their detailed response.

We do not intend to imply that training using a single goal teaches the agent to reach any state

I would like to clarify that I understand the purpose of the method is to train one policy to reach exactly one goal state. My concerns are regarding the evaluation methodology for these policies.

As noted in my original review, "if this method is effective on one particular goal, but not others, then it would not be effective in practice." Since this work only uses the default goal state from the given environments, which is usually not the most difficult success state, the experiments provided do not show whether single-goal CRL can indeed outperform standard CRL when reaching the more difficult (or easier) goal/success states provided by the MetaWorld environments.

However, when we dig into the metaworld codebase [...] self._freeze_rand_vec actually corresponds to randomizing the starting position

I apologize for my confusion regarding the standard CRL implementation and the self._freeze_rand_vec attribute. However, it is not accurate to claim that the MetaWorld environments effectively use a single goal. As I am unsure on best practices for sharing videos on OpenReview, I have instead attached code below which is adapted from MetaWorld's README.md to run MT1 on peg-insert-side-v2. In the generated video, one can see the box (goal) and peg (object) are both randomized far more than $1\text{mm}^3$.

Standard CRL performs data collection and evaluation using goals sampled uniformly along the goal path from the starting point to the single goal state (+- 5 mm) used in Metaworld.

Are the authors certain of this? The CRL paper defines success for each environment in Appendix E.1. For Sawyer Bin, the paper states, "sawyer bin (image, state) – This task is taken from Yu et al. [139]. This task involves using a robotic arm to pick up a block from one bin and place it at a goal location in another bin. The benchmark specifies success as reaching within 5cm of the goal" [1]. The last two sentences seem to indicate that the goal locations are indeed in the other bin.

However, if the authors can indeed confirm that the CRL implementation also uses goals states along the path to success to determine success rate during evaluation, then I am happy to reevaluate this portion of my review.

[1] Eysenbach, Benjamin et al,. "Contrastive Learning as Goal-Conditioned Reinforcement Learning." NeurIPS 2022.

# to run:
# git clone https://github.com/Farama-Foundation/Metaworld.git
# cd Metaworld
# git checkout cca35cff0ec62f1a18b11440de6b09e2d10a1380
# pip install -e .
# python [file-name-here]

import metaworld
import random
import imageio

print(metaworld.MT1.ENV_NAMES)  # Check out the available environments

env_name = "peg-insert-side-v2"

mt1 = metaworld.MT1(env_name)  # Construct the benchmark, sampling tasks

env = mt1.train_classes[env_name](
    render_mode="rgb_array"
)  # Create an environment

# Record initial renders for 10 episodes
all_renders = []
for episode in range(10):
    task = random.choice(mt1.train_tasks)
    env.set_task(task)  # Set task

    # Reset environment and record initial render
    obs, _ = env.reset()  # Reset environment
    all_renders.append(env.render())

    # Record renders for 10 steps
    for step in range(10):
        a = env.action_space.sample()  # Sample an action
        obs, reward, terminated, truncated, info = env.step(a)
        all_renders.append(env.render())

# Save all renders as one video
imageio.mimsave("all_episodes.mp4", all_renders, fps=20)

print(f"Saved {len(all_renders)} frames to video")