Skill Discovery using Language Models

In equation 1, wouldn’t optimizing \phi_{i-1} and \varphi_{i-1} potentially lead those variables to be out of distribution for the frozen policy \pi_{i-1}?

When training the $i$-th skill $\pi_i$, optimizing the parameters $\phi_{i-1}$ and $\varphi_{i-1}$ from the previous $(i-1)$-th skill does not lead to initial states outside the distribution of the frozen policy $\pi_{i-1}$. In L2S, the skill policies are parameter-conditioned, meaning that, during training, they are trained to handle a range of parameter values well (similar to goal-conditioned RL). When optimizing $\pi_i$, these parameter values $\phi_{i-1}$ and $\varphi_{i-1}$ from $\pi_{i-1}$ are adjusted within the same range, maintaining stability during training.

How are the “user-selected” variances for the skill parameters determined?

As we provided information about the environment and additional knowledge that connects the semantics of real-world instructions to the robot environment and specifies the task's successful conditions (see Appendix E), the LLM gains some understanding of the environment's scale and selects reasonable (though not necessarily optimal) parameter mean values. By default, we set the variance of the parameters to be twice the maximum mean value generated by the LLM for the current task (a heuristic). We conducted experiments with varying alternative parameter value variances, while keeping the parameter mean value fixed:

LORL-Turn faucet left

For this task, we examined three different parameter variance combinations—variants 1, 2, and 3—to analyze their effects on reward and termination parameters. The first skill in the task is trained to position the end effector around the faucet handle, with the reward function parameter defining the acceptable distance to the handle. The termination condition parameter specifies how close the end effector must be to the target position to transition to the next skill.

• Variant 1: The variance of the termination condition parameter is increased.
• Variant 2: The variance of the reward function parameter is increased.
• Variant 3: The variance of both parameters is increased.

The results were obtained using three different random seeds and demonstrate that L2S consistently achieves optimal parameter values across the default setting and all tested variants.

ManiSkill2-Open drawer

For this task, the parameter in the termination condition of the first skill specifies the required proximity of the robot's end effector to the target position above the drawer handle. In the variant, we increase the variance of this parameter from the default value of 0.02 to 0.05 to evaluate its impact on performance.

The results, obtained using three different random seeds, consistently demonstrate the robustness of L2S in handling various parameter variance configurations while maintaining effective performance.

The abstract was somewhat difficult to understand.

We will update the paper to reflect the role of parameters in policies in the abstract.

The critique of existing skill chaining literature

We will clarify in the paper that we do not assume the system can be reset to any state, as this would be a strong assumption for the learning algorithm in the real-world setting.

No error bars in figure 5 and no statistical analysis for any of the results.

Thanks for pointing out the inadequacy of result analysis! We will add error bars in Fig. 5 and update the statistical analysis for Fig. 4 and Fig. 5 in the article as follows:

“For the performance on the task sequence of 6 tasks, as shown in Fig. 4: 1) In LORL, L2S solved an average of 5.44 tasks in a total of 1.1e7 time steps, while L2S-fixed solved 4.98 tasks, and Text2Reward solved 4.9 tasks in a total of 1.5e7 time steps. 2) In ManiSkill2, L2S solved an average of 5.33 tasks in a total of 1e7 time steps, while L2S-fixed solved 4.14 tasks, and Text2Reward solved only 2.68 tasks in a total of 1.5e7 time steps. Overall, L2S showed an improvement of 11.0% in LORL and 98.8% in ManiSkill2, while requiring 26.7% and 33.3% less training cost compared to the baseline Text2Reward.For the performance on single simple or complex tasks, as shown in Fig. 5, L2S outperformed the baseline Text2Reward by 18.7% and 98.7% on average success rate in LORL and ManiSkill2 environments, respectively, demonstrating a significant performance improvement with L2S.”

I'm concerned that the trial-and-error process required to create task-specific prompts might be more difficult than directly writing the reward and termination functions.

The meaning and intention of this part of the prompt seem to have been misunderstood. The user-provided “additional environment knowledge” is meant to bridge the semantics of real-world instructions to the robotics environment and define the task's overall success conditions. Specifically, it involves: 1) mapping the real-world directions (e.g., Cartesian coordinate axes) to the task description, and 2) specifying the success criteria for the task, which can vary based on human preferences. For example, in the LORL environment, the human instruction "moving an object forward" should correspond to moving the object along the positive y-axis in the simulation, and the task would be considered successful if the object is moved 0.1 meters along the positive y-axis.

Thank you for your dedication to reading our work thoroughly and providing thoughtful feedback. Your insights have been instrumental in helping us refine and improve our research. Below are our responses to the weaknesses and questions you raised. We provide our essential code base at https://anonymous.4open.science/r/L2S_basecode-F5D4.

Questions

Few-shot prompting with examples

Thank you for pointing out the missing part. We provide the content of our few-shot examples here and will include them in Appendix E. These few-shot examples serve two purposes: 1) to provide workflows for task decomposition, and 2) to constrain the LLM from generating overly fine-grained or meaningless skills that contribute little to the task.

Instances of Few-shot Examples:
1.Task to be fulfilled: Turn an object with a handle left.
Corresponding skills and sequence of skills for accomplishing the task:
  Skill 1: Align the robot arm end-effector to a 3D position on the right of the object handle with some offset.
  Skill 2: Move robot arm end-effector and turn the object handle left.
  The sequence for accomplishing the task could be: Skill 1 -> Skill 2.
2.Task to be fulfilled: In the MuJoCo PickAndPlace environment, pick up a box and move it to the 3D goal position and hold it there.
Corresponding skills and sequence of skills for accomplishing the task :
  Skill 1. Navigate gripper to the box.
  Skill 2. Grasp the box and move the box to the goal position and hold it.
  The sequence for accomplishing the task could be: Skill 1 -> Skill 2.

Policy Termination

As is common in reinforcement learning methods, a policy terminates either when the termination condition is met or when the maximum number of action steps is reached. The state distribution after the policy terminates, regardless of the termination reason, is then considered as the initial state distribution for training the next skill.

Why make the assumption that the parameters for each skill only depend on the skill after them?

Repeatedly optimizing the parameters of previously learned skills can be inefficient, as it enlarges the search space. More importantly, optimizing the parameters of all prior skills simultaneously can degrade the performance of some skills, as each skill is trained based on an initial state distribution conditioned on the optimized parameter values of the preceding skills. Updating these parameters for earlier skills may lead to suboptimal behavior.To support this explanation, we conducted additional evaluations using the "Open Drawer" task from LORL and the "Pick Cube" task from ManiSkill2, each of which involves three decomposed skills. In these evaluations, we optimized the parameters of both the first two skills while training the third skill. Results indicate that optimizing all prior parameters incurs higher training costs to maintain a similar success rate:

Training of third skill in LORL-Open Drawer

Training of third skill in ManiSkill2-Pick Cube

Given a subtask sequence of 1) move to button, 2) close fingers, and 3) push button, we assume that the reward and termination functions for skill 2 developed by the LLM encourage and require the end effector to be fully closed around the button.

Why optimize the parameters and the policy separately in an iterative process rather than together?

When the parameters and policy are optimized together, changes to the policy might invalidate the parameters, leading to potential suboptimal behavior. Optimizing them separately allows for the preservation of previously learned behaviors while still refining the system.

Thanks to the authors for answering my questions, providing the codebase, and including error bars in their experimental results, and including the missing appendix section.

Regarding the assumption that each skill only depend on the skill after them, it is clear from the experiments that the assumption holds for the tasks that were evaluated on. However, I don't think this assumption holds in a lot of other tasks. The fact that "optimizing the parameters of all prior skills simultaneously can degrade the performance of some skills" is a limitation of this particular method on tasks where the assumption doesn't hold. This is maybe a direction for future work, but should probably be listed as a limitation.

The user-provided “additional environment knowledge” is meant to bridge the semantics of real-world instructions to the robotics environment and define the task's overall success conditions

This is reasonable for some of the "additional environment knowledge" and not as reasonable for others. For example "3. Tasks about moving mug are considered successful when mug is moved at least 0.1 meter towards the correct direction compared with the object's initial position." This is an extremely task-specific piece of context. I think the extensive prompt engineering required to get this system to work should be listed as one of the limitations.

Lastly, the abstract remains unchanged and continues to contain somewhat vague terminology such as "uncertainty surrounding the parameters used by the LLM agent" that is hard to understand without first understanding the paper.

Overall, I think the paper has improved and I have increased my score accordingly.

Thank you once again for your thoughtful response, suggestions, and the score increase! We have included “optimizing the required previous skills (not just the immediate one)” as a limitation and potential extension in the paper.

Regarding the concern that some additional knowledge appears highly task-specific—such as the example, “Tasks involving moving a mug are considered successful when the mug is moved at least 0.1 meters in the correct direction compared to its initial position”—this type of success condition can be provided as input alongside the task description. That said, Algorithm 1 (line 7-9) in the paper can utilize sparse reward feedback from the environment to automatically infer such termination conditions, even if they are not explicitly specified in the prompt. We hope this explanation helps mitigate the concern.

We will update the abstract of our paper as follows:

"... Each skill is defined by an LLM-generated dense reward function and a termination condition, which in turn lead to effective skill policy training and chaining for task execution. However, LLMs lack detailed insight into the specific low-level control intricacies of the environment, such as threshold parameters within the generated reward and termination functions. To address this uncertainty, L2S trains parameter-conditioned skill policies that performs well across a broad spectrum of parameter values. ..."

Thanks for the response. With more explanations and experiments, the current version should be better now. I would like to consider increasing my score.

For this version, I still have some questions and suggestions:

1. For Text2Reward[1], based on my personal experience, the one-shot reward generations are usually even not usable (the generated values and components are unresonable without human-in-loop prompting.) I understand that subtask requires simpler reward functions, making them easier to define accurately, but it would be helpful if you can provide more analysis, examples, or even ablation studies between them.

2. I think adding more details for implenting Eureka[2] for fair comparison in the draft is important, since it has different training progress rather than L2S and T2R[1].

3. I think adding more comparsion/analysis between L2S and League++[3] is helpful. I might be curious about the following aspects:

   a. League++[3] combines symbol operators with LLM for skills decompositions and terminal conditions generation, which might be able to increasing the planning success rates but requires more human efforts. I think comparing it with the method used in L2S is also needed. Add some experiments to test the success rates of skill decomposition with L2S is also helpful.

   b. Adding more analysis about the trade-off with higher success rate in generating rewards with human-defined metrics functions (maybe helpful for long-horizon task) or with free-form reward generation with lower success rate in generating reward is helpful. Also, in League++[3], actually the "skills" shown in the L2S are similar to the "metrics functions". Adding some analysis regarding this choice is helpful.

   c. Since the ideas have a lot of similarity, change the story and highlight the contribution will enhance the novelty.

4. For only optimizing the last skill, it's impressive to see that optimizing all prior parameters incurs higher training costs to maintain a similar success rate. However, with my example shown before (you have a task with three skills: open the cabinet, pick a hammer, and place the hammer to the cabinet. If the cabinet is not opened far enough, you cannot place the hammer successfully. At the moment, you require to update the parameters and termination of the first skill, not the last skill), I think it's necessary to figure out which skill required to be optimized. To let the framework figure out which skills need to be optimized might be a great extension

5. I am not quite understand the details about the reward generation ablation. For reward generation experiment in LORL and ManiSkill2 environments, what are the tasks you use? How many stage/skills do they have? The success rate is for single skills or the entire tasks? Also, looks like the shape/syntax errors in T2R is only 3%. How about the other errors shown in T2R?

[1]. Xie et al., TEXT2REWARD: REWARD SHAPING WITH LANGUAGE MODELS FOR REINFORCEMENT LEARNING, 2024

[2]. Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models, 2024

[3]. Li et al., LEAGUE++: EMPOWERING CONTINUAL ROBOT LEARNING THROUGH GUIDED SKILL ACQUISITION WITH LARGE LANGUAGE MODELS, 2024

We sincerely appreciate the time and effort you have dedicated to carefully reading our paper, and we sincerely thank you for your valuable feedback. Below are our responses to the weaknesses and questions you raised.

Questions

How can you generate usable reward codes in one shot?

L2S generates usable reward functions in one shot by leveraging task decomposition into subtasks (skills) through the LLM. Each subtask requires simpler reward functions, making them easier to define accurately compared to the more complex reward functions needed in Text2Reward, which must handle entire tasks and are more challenging to get right.

How do you compare the training efficiency with Eureka directly? Why the performance of a single skill like OpenDrawer is pretty low in Eureka? (It shows great performance in much more complex tasks.)

Eureka incorporates a feedback loop where successful RL reward function samples from the previous step are used as input to generate new samples from the LLM. For our benchmarks, we ran Eureka for 3 rounds with 8 samples per round. This process resulted in significantly higher training costs compared to L2S, measured by the environment steps required for agent training. While increasing the number of rounds or samples per round could potentially improve Eureka's performance on our benchmarks, we observed that L2S remains more sample-efficient overall. For Eureka, we conducted multiple runs and reported results only from those that had at least one successful sample in each round. Eureka achieves strong performance in the ManiSkill2 OpenDrawer environment but struggles in the LORL OpenDrawer environment. In contrast, L2S performs consistently well in both environments, demonstrating its robustness across a diverse range of tasks and settings.

How can we manage skills with distinct semantic descriptions but similar motion patterns (e.g., "place" and "insert")? Additionally, how can we effectively utilize such skills for reuse across tasks?

L2S decomposes complex tasks into subtasks, enabling skill reuse. For instance, it may break down tasks like "place" and "insert" into common subtasks such as "moving closer to the object," "grasping the object," and "moving closer to the target." The final steps, however—such as "putting it on the target position" for "place" and "putting it into the target position" for "insert"—are task-specific and less reusable. Nonetheless, L2S can effectively reuse the shared first 3 subtasks across different tasks, demonstrating its strength in leveraging compositional structures.

Weaknesses

Comparison with League++

Thank you for highlighting the related work we missed. In League++, the reward functions are generated by the LLM through selecting and weighting pre-defined metric functions provided by human experts. This approach simplifies the problem and minimizes irrelevant outputs by constraining the solution space. However, L2S differs from League++ in the following key aspects:

1. Free-form Reward Generation: Unlike League++, L2S enables the LLM to generate free-form reward functions directly from the environment knowledge, rather than being limited to a predefined set of metric functions.

2. Reduced Human Expert Effort: L2S requires less human effort, as it only needs natural language examples of task decomposition. Providing these examples is significantly easier than designing specialized metric functions.We will include a discussion of League++ in the related work section of our paper to address this comparison.

As shown in Table 7 of Text2Reward, the success rate of generating executable reward functions is just 90%. Since L2S generates several reward functions for different skills, the failure rates grow exponentially.

L2S achieves a higher execution success rate for each generated skill compared to the whole-task reward function generated in Text2Reward. This is because generating free-form function code for individual skills is inherently simpler than generating a single function for the entire task. Each skill represents only a portion of the overall task, reducing complexity. To substantiate this claim, we evaluated the LLM's performance in generating correct function code for both LORL and ManiSkill2 environments more than 100 samples each:

The results highlight the effectiveness of L2S in breaking down complex tasks into manageable components and improving the reliability of code generation.

It's not convincing that only optimize the last skill parameters is enough.

Repeatedly optimizing the parameters of previously learned skills can be inefficient, as it enlarges the search space. More importantly, optimizing the parameters of all prior skills simultaneously can degrade the performance of some skills, as each skill is trained based on an initial state distribution conditioned on the optimized parameter values of the preceding skills. Updating these parameters for earlier skills may lead to suboptimal behavior. To support this explanation, we conducted additional evaluations using the "Open Drawer" task from LORL and the "Pick Cube" task from ManiSkill2, each of which involves three decomposed skills. In these evaluations, we optimized the parameters of both the first two skills while training the third skill. Results indicate that optimizing all prior parameters incurs higher training costs to maintain a similar success rate:

Training of third skill in LORL-Open Drawer

Training of third skill in ManiSkill2-Pick Cube

More experiments for long-horizon tasks.

To address the reviewer’s concern, we provided additional results on complex, meaningful tasks in both the LORL and ManiSkill2 benchmarks. Notably, we prompted GPT-4 in both L2S and Text2Reward to reuse policies learned from prior single tasks whenever possible, ensuring a fair comparison between the two approaches.

Thanks for giving more detailed explanations and more experimental results. Although I would like to see more comparsion in reducing LLMs Hallucinations with limited symbolic operaters & metrics functions and using freeform generations for planning and long-horizon tasks, I understand it might not be the main contents of this paper. I don't have any other questions now and would like to increase my score.

Thanks for the adding more details and your reminding. Since the current version has addressed my questions and added some related works and experiments, I would like to increase my score to boardline accept now. Please remember to include the complete comparison into the paper accordingly

How is the parameter range chosen for the experimental results in Section 4?

As we provided information about the environment and additional knowledge that connects the semantics of real-world instructions to the robot environment and specifies the task's successful conditions (see Appendix E), the LLM gains some understanding of the environment's scale and selects reasonable (though not necessarily optimal) parameter mean values. By default, we set the variance of the parameters to be twice the maximum mean value generated by the LLM for the current task (a heuristic). We conducted experiments with varying alternative parameter value variances, while keeping the parameter mean value fixed:

LORL-Turn faucet left

For this task, we examined three different parameter variance combinations—variants 1, 2, and 3—to analyze their effects on reward and termination parameters. The first skill in the task is trained to position the end effector around the faucet handle, with the reward function parameter defining the acceptable distance to the handle. The termination condition parameter specifies how close the end effector must be to the target position to transition to the next skill.

• Variant 1: The variance of the termination condition parameter is increased.
• Variant 2: The variance of the reward function parameter is increased.
• Variant 3: The variance of both parameters is increased.

The results were obtained using three different random seeds and demonstrate that L2S consistently achieves optimal parameter values across the default setting and all tested variants.

ManiSkill2-Open drawer

For this task, the parameter in the termination condition of the first skill specifies the required proximity of the robot's end effector to the target position above the drawer handle. In the variant, we increase the variance of this parameter from the default value of 0.02 to 0.05 to evaluate its impact on performance.

The results, obtained using three different random seeds, consistently demonstrate the robustness of L2S in handling various parameter variance configurations while maintaining effective performance.

We sincerely thank you for your valuable feedback and thoughtful comments! We truly appreciate the time and effort you have dedicated to reviewing our submission. Below, we provide detailed responses to your questions. We have updated the flaws mentioned in Minor Corrections in paper and will upload it shortly.

Weaknesses

Who determines the semantic meaning of the parameters?

L2S leverages LLMs to decompose a task into subtasks and generates reward and termination functions for each subtask. We prompt the LLM that when it lacks high confidence in determining a task-specific numerical threshold, it introduces a parameter to represent the uncertainty. For instance, in the task "turn faucet left," one subtask involves positioning the end effector near the faucet handle. The threshold for how close the end effector should be to the handle is task-specific, so the LLM introduces a parameter to define this distance. These parameters are tied to specific concepts within the reward and termination functions generated by the LLM. Consequently, optimizing these parameters directly enhances the overall reward performance, as it aligns with the structure and objectives defined by the generated reward functions.

Thank you once again for your time and thoughtful review of our work. As the deadline of discussion phase approaches, please don’t hesitate to let us know if there are any additional questions we can address to further strengthen our submission.

Thank you for addressing my questions. I have no further inquiries and am happy to stick with my original score!

Thank you for your dedication to reading our work and providing valuable feedback. Below are our responses to the weaknesses and questions you raised.

Questions

How is the baseline being compared？

We compare our approach, L2S, against two baselines, Text2Reward and Eureka, using all available environment-related information. This includes an environment description, additional environment-specific knowledge, and few-shot examples. The "additional environment knowledge" component serves to bridge the semantics of real-world instructions with the simulated environment and clarify task success criteria. Specifically, this involves:

1. Mapping the 3D coordinate system of the environment to the human-provided task descriptions.
2. Defining the success conditions of tasks, which may vary based on human preferences.

For example, in the LORL environment, a human instruction such as "move an object forward" is interpreted as:

1. Moving the object along the positive y-axis in the simulated environment.
2. Considering the task successful if the object is moved at least 0.1 meters along the positive y-axis.

How to generalize to new environments？

To generalize to new environments, all environment-specific information in the prompt must be updated to align with the new environment, as the assets, supported tasks, and corresponding success conditions differ across environments. However, the "Instruction Hint" section remains consistent. For further details, please refer to the prompt blocks provided in Appendix E.

Weaknesses

L2S depends heavily on the accuracy of LLM-generated reward and termination functions. No evolutionary search

The primary contribution of L2S lies in its framework for leveraging LLMs to perform task decomposition and facilitate the efficient reuse of subtask policies across related task domains. In this work, we use few-shot examples to guide the LLM in generating accurate and stable free-form code for reward and termination functions. This design choice is inspired by related work, such as Text2Reward, which, however, does not address task decomposition or composition. Alternatively, we could replace the few-shot examples in our prompts with evolutionary search to guide the generation of reward and termination functions, as in Eureka. We believe that L2S would support skill discovery and reuse under Eureka's design strategy. However, our core contribution focuses on enabling skill discovery and policy reuse, rather than on the specific method—whether few-shot examples or evolutionary search—used for reward and termination function generation for subtask policies.

The experimental scenarios are simple.

Our experiments prioritize tasks with compositional structures rather than the complexity of environment dynamics, as prior work (e.g., Text2Reward) has already demonstrated LLMs' capability to generate rewards in such environments. To this end, we compositionalized environments from Text2Reward, including robotics manipulation tasks in LORL and ManiSkill2, to highlight L2S's effectiveness in leveraging task compositionality for continuously presented tasks. For example, one of the long-horizon tasks in our experiments involves opening a drawer, placing a cube from the table into the drawer, and then closing the drawer. During the rebuttal phase, we further showcased an even more complex scenario where the agent must place three blocks into the drawer as part of the process (see our response to Reviewer 1Mn6). L2S not only solves these tasks significantly faster but also achieves higher success rates compared to Text2Reward.

The approach has only been evaluated in state-based environments.

Our results demonstrate that L2S effectively decomposes complex tasks and generates accurate reward and termination functions for subtasks, which is the main focus of this paper. These reward and termination functions can principally be used by RL agents to train control policies using visual inputs and to terminate when specific visual conditions satisfy the termination criteria. Extending L2S to vision-based environments is left for future work.

Dear Reviewer pFie, we hope this message finds you well. We hope the previous response would address the concerns and whether there is anything further we could clarify or elaborate on. Thank you.