Language-conditioned Multi-Style Policies with Reinforcement Learning

W1：Thank you for your feedback. We acknowledge that designing meta-behaviors is currently a heuristic process that requires a certain level of understanding of the environment or task. During our exploration in this study, we have developed two design principles to guide this process:

●Minimal Meta-Behaviors: Use as few meta-behaviors as possible to cover the widest range of agent behaviors within the environment.

●Facilitate Reward Shaping: Ensure that meta-behaviors are designed to facilitate easy reward shaping.

This guidance is akin to the "Occam's Razor" principle, where we aim to describe the majority of an AI agent's behaviors using fewer and simpler elements. This can be achieved through various combinations and degrees of these elements, aligning with the concept of multi-style policy training based on meta-behaviors.

In line with these principles, researchers can design corresponding meta-behaviors based on the desired policy styles for different environments. Currently, there is no totally automated method for designing meta-behaviors, much like reward shaping process in RL, which requires designing different rewards for different environments. It can be likened to "labeling" in supervised learning and is essential for the AI research. Fortunately, recent studies have begun exploring the use of LLMs for automated reward shaping. Following the reviewer's suggestion, we believe leveraging LLMs to design meta-behaviors is a promising research direction, which can increase the automation of our method.

Additionally, typical language-conditioned RL typically requires designing rewards for behaviors corresponding to each instruction. For complex instructions like "Counter Attack," it is challenging to design rewards that assess how well an agent performs this instruction. However, through the combination of meta-behaviors, such complex behaviors can be trained. Decomposing complex instructions into meta-behaviors inherently reduces the burden of designing rewards. This approach allows for the emergence of sophisticated behaviors without the need to explicitly define rewards for each complex instruction.

W2: We sincerely appreciate the reviewer's insightful suggestion. In response, we have conducted two new experiments to evaluate our method's performance on ambiguous instructions and cross-lingual instructions.

For the ambiguous instructions experiment, we designed a protocol to simulate typographical errors that commonly occur during human input. Specifically, for each character in the instruction, we randomly replaced it with an adjacent character on the keyboard with a 3% probability like in [1][2]. This approach mimics realistic scenarios where users may make typing mistakes.

To create the cross-lingual instructions dataset, we uniformly translated the original instructions into Spanish, Chinese, or French, while maintaining some in English. This diverse linguistic mix allows us to test our method's robustness across multiple languages.

Both of these modified instruction sets were based on the GRF single-player instruction set. The results of our method's are presented in Table R1:

Table R1: Alignment accucary of LCMSP-GPT4o on ambiguous and cross-lingual instruction sets with GRF single-player scenario.

As evident from the results, our method demonstrates only a slight decrease in performance when tested on these more challenging datasets. This minimal degradation in effectiveness underscores our approach's generalizability and adaptability to diverse scenarios, including ambiguous and cross-lingual instructions.

W3: We have indeed considered recent LLM-based RL approaches in our related work section, which can generally be categorized into two types:

1.Methods like LLaRP[3] and TWOSOME[4] use RL to fine-tune LLMs to output pre-written macro-actions, such as moving to a specific location or grasping an object. However, for complex environments like GRF, defining such macro-actions is challenging. For instance, creating a "past the defender" action in football requires considering the positions and velocities of the player, the ball, and opponents, as well as anticipating opponent reactions. Moreover, in complex environments like GRF 5v5 scenario, training a policy requires tens of millions of samples, making it prohibitively expensive to fine-tune LLMs directly with these samples.

2.Methods such as SayCan[5] utilize LLMs as high-level planners to select appropriate lower-level RL policies. These approaches also necessitate explicit skill definitions for the lower-level RL policies and require human annotation to determine the success of instruction execution for training. They are also limited to a finite set of behaviors (e.g., 551 in SayCan's case). Consequently, these methods are not easily applicable to complex environments.

To the best of our knowledge, existing LLM-based RL methods are not well-suited for executing instructions in complex environments like GRF. While direct comparisons with other LLM-based RL methods in our complex environments are not feasible due to their limitations, we believe our results provide strong evidence of its effectiveness and uniqueness in instruction alignment for complex RL tasks.

Additional : We have created a new anonymous project page that showcases our policy's capabilities through videos. The page can be found at: https://lcmsp-webpage.github.io/LCMSP/.

Q1：Please see response to W1.

Q2：Please see response to W2.

Q3：Please see response to W3.

Reference

[1] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382, 2020.

[2] Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. Combating adversarial misspellings with robust word recognition. arXiv preprint arXiv:1905.11268, 2019.

[3] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2024.

[4] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.

[5]Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287–318. PMLR, 2023.

W1: We sincerely regret the inconvenience caused by the video link not working. We have provided a new link with the video demonstration included. We hope you can view it and share your feedback. https://lcmsp-webpage.github.io/LCMSP/.

W2: Thanks for your comments. The contributions of our paper can be summarized as below:

1.Novel Paradigm for Language-Controlled RL: We have introduced a novel paradigm for controlling RL policies using natural language. Unlike previous language-conditioned RL (LC-RL) methods, which are limited to simple tasks such as object manipulation and navigation, our approach is the first to enable agents to execute highly complex natural language instructions, such as tactics, which encompass abstract concepts, in intricate environments. Traditional LC-RL methods typically define tasks represented by instructions and use rule-based judgments (like reward shaping) to guide the training of these specific tasks. In contrast, our approach trains predefined meta-behaviors with different style parameters. By combining various meta-behaviors and style parameters, our method can accomplish instructions that are challenging to guide using rule-based judgments. For example, in a football game, we might want a strategy that embodies "Counter Attack." For traditional LC-RL methods, it's challenging to evaluate how well the agent performs the "Counter Attack" instruction in order to provide rewards for training. However, our method automatically learns various complex behaviors during training by combining meta-behaviors, as shown in Figure 5 and Figure 6.

2.Multi-Style RL Training Based on Meta-Behaviors: One of the key contributions of our work is the multi-style RL training framework based on meta-behaviors. Previous multi-style RL approaches, such as [1], have only been tested in simple grid environments, while [2] and [3] were limited to basic game environments like Mario and Sonic, and with a very limited number of styles. For the first time, we have extended multi-style training methods to complex environments like GRF, which involve multiple agents, cooperation and competition, and long horizons. This showcases a wide variety of policies that can be controlled by natural language and resemble real-world tactics. Our work demonstrates the feasibility and effectiveness of multi-style RL methods in such scenarios, significantly expanding the application boundaries of multi-style RL, which previous research has not achieved.

3.The Effectiveness of the DTP method: Our method's effectiveness is not solely dependent on the quality of LLMs. Rather, it derives significant value from the Degree-to-Parameter (DTP) approach, which aligns the LLM with RL policy. To illustrate this, we conducted an experiment comparing the performance with and without the DTP method. In our experiment, we compared prompts that directly output style parameters without using DTP to generate style parameters for a "Counter Attack" tactic. We then used the trained multi-style policy with these newly generated parameters against the same opponents mentioned in the experiment of Figure 5. Tweleve in-game metrics of these two comprason methods are shown in Table R1 and R2.

Table R1: In-game metrics for Counter Attack tactic with DTP prompt.

Table R2: In-game metrics for Counter Attack tactic without DTP prompt.

Q1: Thank you for your insightful comments. To ensure fairness, we selected two of the latest language models with parameter sizes comparable to T5-large and GPT-2 for testing. The comparason results are shown in Table R4 and Table R5. In the GRF single-player and two-player scenarios with simple instructions, the performance of the 8B open-source LLM (Llama3.1-8B) is close to that of the best closed-source LLMs (GPT-4o). However, smaller language models tend to show a decline in performance. This decline is partly due to alignment errors and partly because smaller LLMs are more prone to generating responses in incorrect formats, which cannot be parsed into style parameters. The BERT model, after training, achieves a performance level similar to the few-shot capabilities of a 1B parameter LLM (Llama3.2-1B). Given BERT's rapid inference time, methods based on BERT, such as TALAR, are superior for scenarios requiring quick decision-making.

Table R4: Alignment accucary of six baselines on five types of instructions within GRF Single-player scenario

Table R5: Alignment accucary of six baselines on five types of instructions within GRF Two-player scenario

Q2: Thank you for your meticulous review. We will correct the citation for TALAR to reflect its publication in NeurIPS 2023.

Reference

[1] de Woillemont Pierre Le Pelletier, Remi Labory, and Vincent Corruble. Configurable agent with reward as input: A play-style continuum generation. In 2021 IEEE Conference on Games (CoG). IEEE, August 2021. doi: 10.1109/cog52621.2021.9619127.

[2]Siddharth Mysore, George Cheng, Yunqi Zhao, Kate Saenko, and Meng Wu. Multi-critic actor learning: Teaching RL policies to act with style. In International Conference on Learning Representations, 2022.

[3]Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.

W1: Thank you for your comments. The contributions of our paper can be summarized in the following four points:

1. We have introduced a novel paradigm for controlling reinforcement learning (RL) policies using natural language. Unlike previous language-conditioned RL (LC-RL) methods, which are limited to simple tasks such as object manipulation and navigation, our approach is the first to enable agents to execute highly complex natural language instructions, such as tactics, which encompass abstract concepts, in intricate environments.

2. Traditional LC-RL methods typically define tasks represented by instructions and use rule-based judgments to guide the training of these specific tasks. In contrast, our approach trains predefined meta-behaviors with different style parameters. By combining various meta-behaviors and style parameters, our method can accomplish instructions that are challenging to guide using rule-based judgments. For example, in a football game, we might want a strategy that embodies "Counter Attack." For traditional LC-RL methods, it's challenging to evaluate how well the agent performs the "Counter Attack" instruction in order to provide rewards for training. However, our method automatically learns various complex behaviors during training by combining meta-behaviors, as shown in Figure 5 and Figure 6.

3. Previous multi-style RL approaches, such as [1], have only been tested in simple grid environments, while [2] and [3] were limited to basic game environments like Mario and Sonic, and with a very limited number of styles. For the first time, we have extended multi-style training methods to complex environments like GRF, which involve multiple agents, cooperation and competition, and long horizons. This showcases a wide variety of policies that can be controlled by natural language and resemble real-world tactics. Our work demonstrates the feasibility and effectiveness of multi-style RL methods in such scenarios, significantly expanding the application boundaries of multi-style RL, which previous research has not achieved.

4. We conducted extensive experiments, testing our method with instructions of varying meanings across different environments. Our results demonstrate that our approach can understand abstract and high-level instructions and their application contexts in a few-shot setting, executing reasonable strategies. We also tested various instruction types, proving the generalizability of our method.

W2: Thanks for your suggestion.

1. To eliminate network latency interference caused by calling closed-source LLMs, we tested the inference time of open-source LLMs of different sizes under various scenarios and instruction sets, deployed on our own servers. For Llama3.2-1B, Qwen2.5-0.5B, and BERT, we used an Nvidia A10 GPU for inference, while for the larger Llama3.1-8B, we used an Nvidia L40 GPU. As shown in the Table R1, the inference times for LLMs in the single-player and two-player scenarios are fast. In the most complex 5v5 environment, to reduce inference latency, we optimized the Chain of Thought (CoT) in the prompt, requiring about 4 seconds of inference time, while still ensuring tactical style parameters that conform to the instruction.

Table R1: Inference times of different LLMs and BERT.

2. In practical applications, the multi-style policy and the alignment process of instructions run in parallel, which helps maintain user experience to some extent. Additionally, this method is more suited for scenarios where a strategy needs to be sustained over a period, rather than those requiring immediate responses. Therefore, the impact of latency on user experience is minimal. For example, in a football game, a coach issues a tactical instruction and observes its effectiveness over time before making adjustments.

3. In our previous demo video, part of the observed delay was due to network latency from API calls rather than actual inference latency. We have re-recorded a version using open-source LLMs deployed on our own servers with low inference latency for demonstration, which can be found in: https://lcmsp-webpage.github.io/LCMSP/

W1: Thank you for your feedback. We acknowledge that designing meta-behaviors for an environment is a heuristic process that often requires a certain level of understanding of the environment or task itself. During the exploration process of this study, we have summarized two design principles as guidance:

(1) Use as few meta-behaviors as possible to cover the widest range of agent behaviors within the environment.

(2) Ensure that meta-behaviors facilitate easy reward shaping.

This guidance is akin to the "Occam's Razor" principle, where we aim to describe the majority of an AI agent's behaviors using fewer and simpler elements. This can be achieved through various combinations and degree of these elements, aligning with the concept of multi-style policy training based on meta-behaviors.

We recognize that this requires a certain degree of understanding of the environment, similar to the process of reward shaping in RL. This process can be likened to "labeling" in supervised learning and is essential for the AI study.

W2: Thank you for your valuable feedback. We think your question can be summarized into two topics: the first concerns the generalization of the trained style parameters, and the second concerns the data efficiency of the method. We will address each of these important issues separately.

(1) Our poposed method exhibits excellent generalization for different style parameters, as demonstrated in Appendix D.4.2 and Figure 13. Figure 13 shows the in-game metrics as functions of the style parameters. It can be observed that adjusting the value of each style parameter results in a nearly linear and smooth change in the corresponding metric. The style parameters tested in the figure may not have been trained, yet the results show strong consistency in game metrics as the style parameters change. This indicates that LCMSP possesses the property of 'smooth linear interpolation for generalization.' Furthermore, although we only sampled style parameters within the range [0, 1], we found that setting a style parameter to 1.1 resulted in further changes in the related game metrics. This suggests that our method not only has 'interpolation generalization' but also some 'extrapolation generalization' capability. We appreciate the reviewer's suggestion and have decided to include the results and discussion of these findings in the next version of our manuscript.

(2) Our approach does not involve training a new set of style parameters from scratch each time. Instead, the style generator simultaneously produces multiple different style parameters, allowing the agent to generate new training data based on these parameters. Once the amount of training data is sufficient for a batch, we perform a training iteration using data from various style parameters. Through multiple training iterations, when the change in training metrics (such as episode returns) stabilizes, we consider that the different style parameters within the sampling range have been effectively captured.

We conducted an experiment to preliminarily demonstrate the efficiency of this approach. The training environment involved drawing shapes like triangles, quadrilaterals, and circles, with three meta-behaviors and corresponding style parameters: one controlling the shape, another controlling the radius, and the third controlling the angular velocity. The specific environment setup can be referenced in [1] section 5.1. Four experimental conditions were established :

Experiment A: One group trained with one set of fixed style parameters.

Experiment B: Three groups, each trained by varying one style parameter while keeping the other two fixed.

Experiment C: Three groups, each trained by varying two style parameters while keeping the other one fixed.

Experiment D: One group trained by varying all three style parameters.

We recorded the episode return curves as the number of used samples increased, with three runs per group using different random seeds. The results showed that, with the same number of samples, the episode return for models in Experiment D was only slightly lower than models in Experiment A and was comparable to models in Experiment B and C. The results indicate that as the number of training styles increases, there is a modest decrease in sample efficiency, though the decline is not severe.This training curve can be found in the last part of our project demonstration: https://lcmsp-webpage.github.io/LCMSP/. These results indicate that the LCMSP method is training efficient.

[1] Mysore, Siddharth, et al. "Multi-critic actor learning: Teaching RL policies to act with style." International Conference on Learning Representations. 2022.