Guiding Skill Discovery with Foundation Models

Thank you for your review!

Studies such as [1] have already researched generating reward functions using foundation models. While there is some differentiation in applying a similar idea to skill discovery, the novelty appears limited compared to previous research that generated entire rewards through foundation models. The proposed idea only utilizes foundation models in the restricted aspect of 'applying weights to rewards proposed in other studies'.

We agree that previous work already leverage foundation models to generate reward functions to train RL agents, such as [1, 3]. However, most of them only work for state-based input. For pixel-based tasks, previous work [4] shows that foundation models can only output very noisy rewards which can not be used to directly train RL agents. In our work, we actually show that such “noisy” signals can come into play in the skill discovery field.

It seems that skills that cannot be generated by METRA also cannot be generated through FoG, raising concerns about whether the performance of the proposed framework is dependent on METRA.

Yes, we think this might be the case. However, for example, in Fig. 6, we did see that FoG learns to move different directions with METRA, suggesting that FoG might have potential to discover different skills with the underlying used skill discovery algorithms.

Although the paper explains why [2] was not used as a baseline, the lack of comparison with other studies that utilize foundation models appears to be a weakness of the paper.

Thanks for pointing out. We now added two more baselines, METRA+ which uses human-defined score functions, and DoDont+ which uses foundation models annotated demonstrations for training DoDont. Please see updated Fig.4 and Fig.5.

The paper mentioned in the Weakness section, [2], claims to be able to discover useful skills in environments utilizing robotic arms. In contrast, FoG appears to have failed to discover useful skills in the Franka Kitchen environment. I'm curious what differences between the two methods led to these contrasting results.

First, [2] is using state-based input, while FoG takes pixel-based input, which makes them not directly comparable. Meanwhile, FoG’s performance largely depends on the performance of the underlying used visual foundation models. For Franka Kitchen, it appears that CLIP cannot distinguish different visual states well, resulting in poor performance. With stronger foundation models that are more accurate, we believe the performance of FoG can be enhanced.

---

[1] Xie, Tianbao, et al. "Text2reward: Automated dense reward function generation for reinforcement learning." arXiv preprint arXiv:2309.11489 (2023).

[2] Rho, Seungeun, et al. "Language Guided Skill Discovery." arXiv preprint arXiv:2406.06615 (2024).

[3] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv: Arxiv-2310.12931

[4] Adeniji, Ademi, et al. "Language reward modulation for pretraining reinforcement learning." arXiv preprint arXiv:2308.12270 (2023).

Thank you for your review!

The methodology is the same as DoDont.

FoG is not only closely related to DoDont, but also close to LSD, CSD, METRA. As can be seen in Table 1, all of these methods (including FoG) use the same objective (Distance-maximizing Skill Discovery objective proposed by CSD), but leverage different underlying distance metrics, which have different properties.

For state-based RL, FoG utilizes ChatGPT to "write" a function to indicate if a certain dimension of the state is larger than a threshold; for instance, f(s) = lambda s: s[1] > threshold. This function is a trivial implementation of the score function, i.e., I(s'[1] > threshold), i.e., "s'[1] > threshold" is classified as "desired" and "s'[1] <= threshold" labeled as "undesired".

The intention of FoG is to show that foundation models can be used to guide skill discovery in both state and pixel-based tasks, instead of showing how complicated score fn that foundation models can propose. Meanwhile, such a score function might seem too trivial to design for humans, it actually requires foundation models to understand 1) our intention; 2) identify the important state dimensions; 3) propose the logic of implementation, etc. which is not trivial for computers to do.

For pixel-based RL, FoG also utilizes the CLIP similarity score to implement a trivial score function CLIP(s', "desired") > CLIP(s', "undesired"). For instance, FoG utilizes CLIP(s', "ground is blue") > CLIP(s', "ground is orange") to encourage the halfcheetah to appear in the blue ground. However, one can easily obtain the visual state of the halfcheetah in the blue ground and the visual state of the halfcheetah in the orange ground and label the former as the "desired" state and the latter as the "undesired" state to train an image classifier for DoDont. The training process of the image classifier in DoDont can be regarded as a distillation from the pre-trained CLIP or human knowledge to a lightweight neural classifier. Similarly, the descriptions of "flip over" vs. "stand" and "twist" vs. "stretch" can easily be distilled from CLIP, other foundation models, or even human annotators.

Thanks for pointing out. We added the DoDont trained by CLIP annotated demonstrations as a baseline (DoDont+). Turns out DoDont+ does not really work very well. See updated Fig. 5, DoDont with expert demonstrations can learn to avoid bottom-left area in Quadruped. Then we annotate these demonstrations using CLIP (what FoG uses) and use the annotated demonstrations to train DoDont+. Since CLIP is not able to 100% annotate “dos” and “donts”, the demos end up with ~70% of accuracy. Then such inaccuracy is injected to the classifier of DoDont+, which consistently provides inaccurate signals, resulting in poor performance. Whereas FoG uses CLIP on the fly, it has ~70% accuracy, the overall trend is positive, leading to a self-correct system and ending up with better performance.

What's more, when there is no color in the ground or the color distribution in the left plane is the same as the right plane, the CLIP image encoder would fail since it does not capture any feature related to the motion.

We totally agree with this point, and this is a problem of all these Distance-maximizing Skill Discovery methods [1]. In such a case, none of FoG, DoDont nor METRA would work using pixel-input. We don't think it's suitable to use a vision-based foundation model when there are no salient features in the image to distinguish between states. CLIP is best used in settings with clear visual features, like Cheetah, Quadruped with colored planes. Note that we used these environments out of the box from prior work [1], and did not do any additional modifications to make CLIP better.

Additionally, utilizing the vision language model to capture the human intention has been proposed by Juan et al.^1.

[2] indeed shows that vlm can be used to capture the human intention to provide step-wise reward signals to RL agents. But they are too noisy to directly train the agents (36% success rate). In order to make them work, authors need to 1) change the texture of the environment background to be more realistic; 2) change the camera view. In contrast, FoG provides an idea of how we can leverage vlm and avoid outputting such noisy step-wise signals to better guide skill discovery without any specific modifications of the original environments.

---

[1] Park, Seohong, Oleh Rybkin, and Sergey Levine. "Metra: Scalable unsupervised rl with metric-aware abstraction." arXiv preprint arXiv:2310.08887 (2023).

[2] Rocamonde, Juan, et al. "Vision-language models are zero-shot reward models for reinforcement learning." arXiv preprint arXiv:2310.12921 (2023).

Thanks to the authors for their clarifications. However, these clarifications cannot fully address my concerns.

1. Methodology and Novelty: Previous methods, such as LSD, CSD, METRA, and DoDont, introduced novel components like Lipschitz constraints, controllability-aware distances, metric-aware abstraction, and intrinsic reward weighting, respectively. In contrast, FoG only substitutes score function implementation in DoDont with pre-trained foundation models. Moreover, the implementation of the score function has been demonstrated by [2].

2. Use of Foundation Models: Foundation models, such as ChatGPT and Claude, are proficient at generating complex code based on human instructions. Using them to develop a predefined score function, as in FoG, is unsurprising and arguably represents a straightforward application. Although the authors' response emphasizes the complexity of understanding intentions, identifying critical state dimensions, and implementing logic, these tasks fall well within the capabilities of these models. This diminishes the originality claimed in this aspect.

3. Comparison to DoDont+: Although the authors added DoDont+ as a baseline, its implementation may only partially match that of FoG. Specifically, if demonstrations are "automatically" annotated by CLIP, then DoDont+ should similarly leverage CLIP to annotate all past training samples during training dynamically. An on-the-fly version of DoDont+ would align better with FoG's setup and provide a more equitable comparison. The current implementation may not adequately reflect the potential of DoDont+ with an automatic annotation model like CLIP, which undermines the validity of the performance comparison.

4. Overlap with Related Work: The difference between FoG and prior work such as [2] remains unclear. Both approaches utilize Vision-Language Models (VLMs) to capture human intentions utilizing cosine similarity between visual and textual embeddings. I disagree with the authors that FoG does not modify the "original" MuJoCo environment since the environment used in FoG, followed by previous work like METRA, has already been modified. For instance, the floor color has been changed. Moreover, [2] utilizes the OpenAI MuJoCo environment while FoG adopts the DeepMind Control Suites. The noisy level of the gymnasium MuJoCo environment and DeepMind Control Suites is certainly different. It is well-known in the RL community that gymnasium MuJoCo and DeepMind Contro Suites have different dynamics. If the author argues that FoG is more stable than [2], they at least need to compare the reward function of FoG and [2] under the same simulation environments.

Thanks for your response.

1. Methodology and Novelty: Previous methods, such as LSD, CSD, METRA, and DoDont, introduced novel components like Lipschitz constraints, controllability-aware distances, metric-aware abstraction, and intrinsic reward weighting, respectively. In contrast, FoG only substitutes score function implementation in DoDont with pre-trained foundation models. Moreover, the implementation of the score function has been demonstrated by [2].

The reviewer seems concerned about novelty, i.e. we are not contributing a novel component to skill discovery, or coming up with a new score function for VLMs. However, our core focus is not on skill discovery in isolation, rather we are interested in investigating how foundation models can improve skill discovery. Our solution is to weight the skills using a distance metric defined by the foundation model, and we believe this is a simple and effective way to incorporate foundation models into skill discovery, as shown by our results.

2. Use of Foundation Models: Foundation models, such as ChatGPT and Claude, are proficient at generating complex code based on human instructions. Using them to develop a predefined score function, as in FoG, is unsurprising and arguably represents a straightforward application. Although the authors' response emphasizes the complexity of understanding intentions, identifying critical state dimensions, and implementing logic, these tasks fall well within the capabilities of these models. This diminishes the originality claimed in this aspect.

As mentioned above, while it is possible for foundation models to solve tasks, they come with their own drawbacks, like prompting, resource efficiency at inference time, etc. We don’t think it’s a fair argument to claim that because foundation models could directly solve tasks, this diminishes the potential of alternative approaches.

3. Comparison to DoDont+: Although the authors added DoDont+ as a baseline, its implementation may only partially match that of FoG. Specifically, if demonstrations are "automatically" annotated by CLIP, then DoDont+ should similarly leverage CLIP to annotate all past training samples during training dynamically. An on-the-fly version of DoDont+ would align better with FoG's setup and provide a more equitable comparison. The current implementation may not adequately reflect the potential of DoDont+ with an automatic annotation model like CLIP, which undermines the validity of the performance comparison.

If we do what the reviewer described, we would think it is an entirely different algorithm, instead of a variant of DoDont. We think the core idea of DoDont is to use a pre-trained classifier that is trained on pre-collected demonstrations, instead of 1) update the classifier on-the-fly; 2) annotate the demonstrations using foundation models on-the-fly.

4. Overlap with Related Work: The difference between FoG and prior work such as [2] remains unclear. Both approaches utilize Vision-Language Models (VLMs) to capture human intentions utilizing cosine similarity between visual and textual embeddings. I disagree with the authors that FoG does not modify the "original" MuJoCo environment since the environment used in FoG, followed by previous work like METRA, has already been modified. For instance, the floor color has been changed. Moreover, [2] utilizes the OpenAI MuJoCo environment while FoG adopts the DeepMind Control Suites. The noisy level of the gymnasium MuJoCo environment and DeepMind Control Suites is certainly different. It is well-known in the RL community that gymnasium MuJoCo and DeepMind Contro Suites have different dynamics. If the author argues that FoG is more stable than [2], they at least need to compare the reward function of FoG and [2] under the same simulation environments.

In our previous response, we mentioned that “we used these environments out of the box from prior work [1], and did not do any additional modifications to make CLIP better.” We never claimed that we did not modify the “original” MuJoCo environment, what we meant was we did not modify environments we took from [1], which all other baselines are used, including DoDont (part of), METRA.

—-------

[1] Park, Seohong, Oleh Rybkin, and Sergey Levine. "Metra: Scalable unsupervised rl with metric-aware abstraction." arXiv preprint arXiv:2310.08887 (2023)

Thank you for your review!

1. How does the method related to papers [2] and [3]? Does FoG provide any specific advantages over these methods? If so, would it be fair to consider these are relevant baselines for comparison?

Thank you for pointing out [2] and [3]. They are indeed quite related, as they both integrate human intentions into skill discovery. However, both of them only work in state-based tasks, while FoG works in both state-based and pixel-based tasks. Both [2] and [3] require humans to design either safety-rule or provide preferences, while FoG leverages foundation models to do so, which is scalable.

[2] integrates a safety-indicator function into the Critic learning of the unsupervised skill discovery method to achieve learning safe behaviors. However, the safety-indicator function needs to be pre-defined. It is similar to our newly added baseline METRA+, which uses a predefined score function. The problem with these kinds of pre-defined score functions / reward functions is that: if it is perfectly defined, the agent will learn well (see updated Fig.6, METRA+ can learn to perfectly avoid hazardous areas). If not, which is generally the case, (see updated Fig.5, METRA+ cannot learn to avoid flipping using a simple predefined score function), the agent might struggle.

[3] leverages preference-based RL to integrate human’s preference, it requires human-in-the-loop to continuously provide preference on trajectory segments. In more complex tasks where the training time is long, this might be infeasible.

2.In lines 251-252, how did the foundation models develop this insight about the angle of the cheetah’s front tip? On a related note, can we guarantee the score functions are always appropriate?

For state-based tasks, we input the task description and state space (basically most information you can see on the Gymnasium webpage, for example: https://gymnasium.farama.org/environments/mujoco/ant/) to the foundation model as the context. But we do NOT tell foundation models which dimension they should focus on or look at specifically.

In state-based tasks, we cannot 100% guarantee that the score functions are always appropriate. However, during the experiments, we tried different foundation models (ChatGPT, Claude) on different state-based tasks (ant and half-cheetah), and it seems both of them can give us reasonable score functions on both these two tasks. Our intuition is that since all the context is quite clear and presented, and the task is relatively easy, the state-of-the-art foundation models can easily recognise the needed information. If the task gets more complicated, we might need to take an iterative way to gradually guide function models to output the appropriate score function like what Eureka [1] does.

3. Is it assumed that the score can be captured purely by virtue of observing s’ (Eq 6)? Do these scores not consider the manner in which a task is achieved? For example, if the task is to cut a fruit, this could be done with a kitchen knife or a hacksaw. Although the final outcome (the fruit cut into pieces) may be identical, the way it is done also involves some alignment signals – does the proposed score function capture such process-based alignment (which would be based on state trajectories, and not just a single state) as opposed to state/outcome based alignment?

Indeed, FoG aims to provide step-wise reward signals, thus, the score fn is based on states. Current FoG does not capture the process-based alignment, but in future work, we mentioned FoG has the potential to integrate multiple intentions simultaneously, in your case, such process-based alignment could be potentially achieved by using two intentions, for example, “the agent is using the knife” and “the fruit is cut into pieces”.

—---------------

[1] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv: Arxiv-2310.12931

[2] Kim, Sunin, et al. "Safety-aware unsupervised skill discovery." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[3] Hussonnois, Maxence, , et al. "Controlled Diversity with Preference: Towards Learning a Diverse Set of Desired Skills." Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 2023.

Thank you for your review!

Though learning diverse skills while avoiding undesirable behaviors is a valid motivation, using foundation models simply as an undesirable behavior recoginizer seems to be a weird use case. In the case that we already use foundation models, it seem to me a more natural way is to let foundation models propose some tasks and safety reward (if applicable), examples include but are not limited to [1] and [2]. This allows skill discovery to directly target at learning more semantically-meaningful skills, compared to target at visiting diverse states.

FoG works with both state-based and pixel-based input. In state-based tasks, we let foundation models output a score function that is similar to the “safety reward” that you mentioned. Similar to works you mentioned [1] and [2] both only work on state-based tasks.

However, using foundation models output the reward for pixel-based tasks is not trivial, for example, [3,4] show that step-wise reward output by the foundation models are too noisy to directly train a RL agent. Thus, FoG leverages foundation models as “behavior recognizers” in unsupervised skill discovery instead of using them for providing step-wise reward signals directly.

Could authors providing the reasoning for this design choice? (though a bit off-topic, though the authors mentions that Eureka requires multiple iterations of reward designs, I am not fully conviced. Learning a skill for each reward during the iterations could also lead to diverse skills).

Eureka [5] is set up under the conventional RL setting, where there is a reward fn and the agent optimizes the reward fn. At the end of the day, the agent learns to achieve the task that the reward fn is defined. In order to learn diverse skills by optimizing a predefined reward fn, it requires 1) knowing all skills in advance; 2) defining a reward function for each skill. FoG is under an unsupervised RL (URL) setting, where there’s no task-specific predefined reward function. In URL, 1)we don’t assume having access to the downstream tasks, all we want is to learn diverse skills, which could possibly include some downstream tasks we are interested in; 2) all skills are learned by optimizing a single unsupervised objective, which defines how diverse / distinguishable the skills are.

—---------------

[1] Ma, Y. J., et al. "DrEureka: Language model guided sim-to-real transfer. arXiv 2024." arXiv preprint arXiv:2406.01967.

[2] Zhao, Xufeng, Cornelius Weber, and Stefan Wermter. "Agentic Skill Discovery." arXiv preprint arXiv:2405.15019 (2024).

[3] Adeniji, Ademi, et al. "Language reward modulation for pretraining reinforcement learning." arXiv preprint arXiv:2308.12270 (2023).

[4] Rocamonde, Juan, et al. "Vision-language models are zero-shot reward models for reinforcement learning." arXiv preprint arXiv:2310.12921 (2023).

[5] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv: Arxiv-2310.12931