YOLO-MARL: You Only LLM Once for Multi-agent Reinforcement Learning

For weakness 1 and 2: So the description for each environment can be directly found on the environments’ website(github for example). We simply copy and paste them to feed into the LLM. And for your concern, this is also why we use LLM to generate strategies for itself instead of writing the strategies by our own. This means that for a brand new environment, all you need to do is to go to the website, copy and paste the description and then let LLM do all the work.

For weakness 3: Thank you for your insightful suggestion regarding the use of multi-modal foundation models to autonomously analyze environments through raw observations like images. We agree that this approach has the potential to make solutions more general and impactful by reducing the effort required from designers.

In our current pipeline, the primary objective is to generate Python functions, which demands the advanced code generation capabilities of large language models (LLMs). We did consider incorporating multi-modal foundation models at the beginning of our project. However, at the time of our research, leading models such as GPT-4 and Claude-3.5 did not support image inputs, limiting our ability to process raw visual observations directly.

Additionally, relying on image inputs would necessitate interacting with the foundation model at every step to continuously analyze the environment. This frequent interaction could introduce significant computational overhead and latency, which runs counter to our goal of developing an efficient and streamlined pipeline.

We wholeheartedly believe that multi-modal foundation models represent a promising direction for future research. As these models continue to advance and better support image inputs, we are excited about the possibilities they offer and plan to explore their integration in our future work.

Thanks for your reply.

1. Regarding the environment description, we want to clarify that the input required for our framework is simple and straightforward. Details about the prompt can be found in Appendix C. For each environment, the description consists of only one or two sentences. Even without prior knowledge of a new environment, users can easily create a brief description using common sense.

2. The following previously published works also rely on some form of prior knowledge or information available on the internet as initial prompts for LLMs:

   a. EUREKA[1]: Utilizes unmodified environment source code and language task descriptions as context to zero-shot generate executable reward functions using a coding LLM.

   b. TEXT2REWARD[2]: Provides a compact representation in Pythonic style, including environment abstractions and functions that serve as background knowledge for the LLM.

   c. Reward Design with Language Models[3]: Supplies the LLM with a task description, a user’s objective description, an episode outcome converted into a string, and a question asking if the episode outcome satisfies the user’s objective.

   d. ProgPrompt[4], Code as Policies[5], GENSIM[6]: These works also require human involvement for prompting, such as few-shot prompting. They often include example trajectories within the prompt to illustrate how tasks should be completed. These works highlight that human labor is commonly required for designing prompts in prior works, whereas our framework minimizes this effort by simplifying the process.

3, The environments used in our study are based on “Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks”[7]. Detailed descriptions of these environments are already provided in the original paper and its accompanying documentation, where available. Additionally, all other components, such as strategies, are automatically generated by the LLM, eliminating the need for users to design them manually—a process that can be time-consuming and often requires expert knowledge. As a result, our framework significantly reduces the effort and time needed for these tedious tasks.

[1] Y. J. Ma et al., “EUREKA: HUMAN-LEVEL REWARD DESIGN VIA CODING LARGE LANGUAGE MODELS.” arXiv preprint arXiv:2310.12931, 2024.

[2] T. Xie et al., “Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning,” Sep. 21, 2023, arXiv: arXiv:2309.11489.

[3] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward Design with Language Models,” Feb. 27, 2023, arXiv: arXiv:2303.00001.

[4] I. Singh et al., “ProgPrompt: Generating Situated Robot Task Plans using Large Language Models,” Sep. 22, 2022, arXiv: arXiv:2209.11302.

[5] J. Liang et al., “Code as Policies: Language Model Programs for Embodied Control,” May 25, 2023, arXiv: arXiv:2209.07753.

[6] L. Wang et al., “GenSim: Generating Robotic Simulation Tasks via Large Language Models,” Jan. 21, 2024, arXiv: arXiv:2310.01361.

Apologies for any confusion. This work focuses on designing a framework that leverages LLM guidance to enhance the training of MARL algorithms, with the training process and curve influenced by the underlying algorithm. To account for variability across random seeds, we report the average mean reward to demonstrate the effectiveness of our framework. By incorporating our pipeline, MARL policies (e.g., MAPPO, MADDPG) outperform baselines with shorter training times and higher mean episode rewards, as shown in [insert specific improvement metric].

Regarding to reward generation, we would like to clarify that the ablation study on reward generation involves using the LLM to generate a reward function that replaces the original reward provided by the environment. The reward generation process is similar to the planning function generation process and occurs only once, during the initial use of a specific new environment as input prompt for the LLM. This stage happens before the MARL policy training begins. We have clarified this point in the revised paper with the following explanation: “For reward generation without feedback, the reward function is generated at the same stage as the planning function to ensure a fair comparison. Specifically, the reward function is generated before the training process begins for each new environment. For reward generation with feedback, the initial reward function is generated in the same way as in the no-feedback scenario. However, the process then becomes iterative: we first complete a full training cycle on the environment, then provide the LLM with feedback based on the training performance. This feedback, combined with the previous prompts, is used to refine the initially generated reward function."

For 10M step, we would like to clarify that the steps we used in our experiments strictly adhere to the methodology outlined in **“Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks.” **. Specifically, as their experiments were conducted in the LBF environment for 2M steps, we adopted the same setting to ensure a fair comparison with the original work. Given that the results clearly demonstrate our framework's superior performance compared to the baseline, we believe it is not necessary to extend the experiments to longer steps at this stage.

Regarding SMACv2, we found its training time significantly longer than SMAC and encountered challenges in accessing core objects and modifying the environment wrapper interface. Integrating our pipeline required incorporating LLM-generated functions, such as the planning function or reward signal, into the training process, necessitating adjustments to components like the critic neural network. These complexities made SMACv2 less suitable for our framework, so we focused on SMAC, which offered a more flexible and accessible environment. This choice highlights why only certain MARL environments, like SMAC, were viable testbeds for our proposed work.

As for the dip observed in YOLO-MARL, we believe it reflects a similar dip in the baseline, as our method builds on the underlying MARL algorithm. For example, MADDPG's inherent instability could contribute to this behavior. The baseline implementation was also directly adapted from “Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks” and its GitHub repository. Since our framework focuses on leveraging LLM guidance to enhance MARL training, the dip is likely due to the training process and the intrinsic limitations of the base algorithm rather than our framework design.

For question about single agent: So our study case is mainly focused on multi-agent fully cooperative environment, which I think can’t be realized in single agent settings. The strategies given by LLMs and the generated planning function are mainly focused on the coordination between multi-agents, and the coordination is also the main idea of this paper. However, I assume our pipeline can be actually implemented in single agent case. But the extra experiment we provided will also focus on multi-agent setting, since this is the topic of our method. For LLM and single agent RL, as summarized in related work, “Reward Design with Language” designs methods that provide scalar rewards based on suggestions from LLMs to guide RL training. (add single agent RL LLM related work here, that we have summarized in our related work; of course, if there is something new we should cite, just revise the related work section). In contrast, as we motivated our work in the introduction and related work sections, multi-agent RL has remaining challenges such as collaboration among agents, and very limited study on whether LLM is helpful or now in different MARL scenarios. Our work would like to explore this direction, starting from what can be a promising architecture and what limitations exist. And we also revised related work in revised version of our paper.

As you can refer to the Figure 1 in page 2, the interaction with the LLM only occurs at the first time you use a specific new environment. We put this in both abstract and introduction: “Notably, for each game environment, YOLO-MARL only requires one time interaction with LLMs in the proposed strategy generation, state interpretation and planning function generation modules, before the MARL policy training process.” To be more specific, let’s say you want to use our pipeline for a brand new environment E, you will only need to follow the pipeline to prompt LLM at the first time to get generated planning function. After this, you can directly use this planning function in your later MARL training, and you don’t even need to interact with the LLM at the beginning of the game when you start a new episode. We will also submit a revised version of this paper to make the parts you mentioned more clear later.