Can Language Agents Approach the Performance of RL? An Empirical Study On OpenAI Gym

Thank you very much for your constructive comments and suggestions. We have revised our paper accordingly. Below, we will provide detailed responses to each point.

Q1: While the idea is timely, the methodology of the paper is unfortunately not very clear. The algorithm the authors propose is not introduced formally, but in charts. This makes it challenging to understand the algorithm, or why it works. Furthermore, the five training scenarios are not described in enough detail either. As a result, it’s not clear to me why L3 does the best out of the setups. It would also help if the authors motivate their algorithm better, as well as the reason why they expect it to work better than previous approaches.

Thank you very much for your valuable advices! We have improved our presentation of the algorithm and $5$ levels.

Q2: It is also strange to me (if I understood this correctly) that the authors used GPT-4 to generate the python code that translates the Gym observations into text. This method seems error prone, and the authors do not demonstrate that this works, or why it’s even necessary in the first place.

Thanks for your helpful comments! We have moved it to the appendix.

Q3: Lastly, the experiments are not presented with enough detail. For instance: the number of seeds used to evaluate the agents is not reported. The learning curves or the converged performance of the PPO/SAC agents are not reported in the main text, but the Appendix. The threshold for saying that a task is solved is not defined in the main text either.

Thank you very much for your questions!

1. we have extended our evaluations to $5$ seeds.
2. PPO and SAC performances are added.
3. The threshold description is moved to the main text.

Q4: If SAC can solve tasks that PPO cannot solve in your setup, why are the PPO scores reported for the majority of the tasks instead of the SAC scores?

Thank you very much for your questions! SAC performances are added. There are also tasks that SAC can not solve.

Q5: I don’t see how some of these environments are partially observable (for instance Acrobot and Cartpole) - this depends on how the states are translated into text. If they are indeed partially observable, it would be good to clarify how in the text. The authors say that Language agents are better in more intuitive and deterministic environments. Aren’t environments like Acrobot and LunarLandar deterministic (after the random first state)? And Blackjack, which is stochastic, can be solved quite well by EXE according to the Radar plot? What do the authors mean when they say an environment is intuitive?

This is a very interesting and important question! It is true that not all of the environments are partially observable. We have summarized the challenges for each environment in Appendix C.1 We have altered the informal description "non-intuitive" to "complex dynamic". For instance, it is easy to reason that the cart will move right if we push right when the cart speed is $0$. We call this dynamic simple. For Acrobot, it is hard to reason how the two links will behave when applying $-1$ torque to the actuated joint.

Q6: I think the paper could be improved a lot by motivating the algorithm and a specific training scenario they recommend better. The evaluation of the standard RL algorithms like PPO and SAC could also be done more thoroughly. Finally, it would be great to understand why EXE works better than the Language Agents.

Thank you very much, it is a really good suggestion. In the revised version:

1. We take CliffWalking as our motivating example.
2. We make further analysis on EXE, especially comparing it to Reflexion.

Thank you very much for recognizing our work, and your constructive comments and suggestions! We have revised our paper accordingly. Below, we will provide detailed responses to each point.

Q1: I would encourage the authors to avoid assertive and absolute claims, such as the first sentence "large language models lack grounding in external environments", and at the bottom of page 1, "we adopt OpenAI Gym, the most classic and prevalent...". These arguments, though eye-catching, are debatable, and such debates are irrelevant to the central points that the authors are trying to make, so why not avoid them? In the final paragraph on page 2, "after sufficient numerical experiments..." I suppose that it should be left for the readers to decide whether the experiments are sufficient. Point 3) at the bottom of page 2 is confusing. Which baseline do the language agents improve upon? I suppose that the authors wanted to say that the performance can be improved via incorporating XXX, but it would be helpful to reorganize the sentence and clarify the point.

Thank you very much for your valuable comments! We have polished our text and avoided the over-claims.

Q2: The Problem Formulation section is somewhat redundant, as none of the math symbols are reused anywhere in the main text.

Thanks for your advice! We have moved it to Appendix.

Q3: What is the main difference between the EXE language agent and other language agents, besides that the learner module provides exploration-exploitation balancing strategies in addition to how to exploit? The reason that I am asking this question is that, if the actor and critic modules are identical to the other agents, the authors don't have to reiterate the descriptions on page 7. The saved space can be dedicated to explaining how the learner module provides guidance on exploration and exploitation.

Thanks you for your comments! We have added pseudocode of EXE and $5$ scenarios in our paper.

Q4: It remains unclear to me (after reading the Appendix) how different levels of agents are implemented. It would be useful to provide at least a brief description.

We are apology for the missing description. We have added pseudocode for them in the revised version.

Thank you very much for your constructive comments and suggestions. We have revised our paper accordingly. Below, we will provide detailed responses to each point.

Q1: First and most importantly, the level of rigor in the empirical evaluation is far from being satisfactory. Performance comparisons are executed using a single seed only and not showing any form of confidence or error bars anywhere in the plots. A comparison of this kind is simply meaningless from the empirical standpoint, regardless of the funding constraints one might have to comply to, and invalidates all the claims and supposed findings contained in the paper. The sequential decision-making scientific community has made significant progress on evaluation rigor in the last few years, and I encourage the author to look at recent work (e.g., https://arxiv.org/abs/2304.01315 , https://arxiv.org/abs/2108.13264) and comply to the standards proposed there. The presence of LLMs or product-oriented APIs does not change the importance of rigorous scientific evaluation.

Thank you very much for your comments. Please refer to Response to all reviewers: Q1 and Q3.

Q2: Some details in the presentation are confusing or unnecessary. As a significant example, why should one highlight that GPT-4 generated the text version of the environment if that one is not a feature that is being scientifically evaluated? As of now, it reads an attempt to yield to GPT-4 the responsibility concerning the correctness of the new environments that are being proposed. But this is totally irrelevant for the matter of the presentation of the scientific evaluation of a system: regardless of whether the code for evaluating the AI agents was generated by GPT-4 writing code, humans writing code, or copied from a random repository on the web, the only thing that matters is its correctness and alignment with the claims that the rest of the paper is trying to make.

Thank you very much for your advices. We have moved it to the appendix and taken a more compact way to present it. However, we also want to clarify that the results that GPT-4 is able to generate a translator based on the documentation and template is very helpful for researchers who want to use language agents as alternatives to PPO.

Q3: I believe a meaningful comparison of pretrained LLM-based systems with RL algorithms trained from scratch might not be as straightforward as the paper is trying to imply. LLMs are pretrained on all the internet, and the neural networks trained with RL start from scratch. It is still interesting to compare the performance of these different approaches, but saying that one approach is more sample efficient than another one does not do justice about their different nature and tradeoffs.

Thank you for your valuable suggestions! We have repositioned our paper as in Response to all reviews: Q1 and motivated our comparisons as in Response to all reviews: Q2.

Q4: Why did you use PPO for some environments and SAC for some others?

Thank you very much for your question! For MountainCarContinuous, PPO fails to generate expert-level performance trajectories due to the exploration challenge in MountainCarContinuous. PPO gets easily to stuck in sub-optimal strategy (not move) even with grid searching hyper-parameters. Although our main focus is the comparison of PPO, the infeasible expert trajectory will make the Lv4 scenario infeasible. Thus we take SAC to generate the trajectories and we take it to construct the L4 scenario. See also in the Response for reviewer DgbS.

Q1: Compute and Time Costs

For detailed information on the compute and time costs, please refer to our Response to all reviewers: Q4.

Q2: Number of Seeds and Range of Environments

We have used 5 seeds to make evauation of Language Agents in our revised manuscript. While hyperparameter tuning is not a concern for our experiments, the decision-making process with LLM-based agents is inherently resource-intensive. This restricts the scale of our experiments. For additional context, see Response to all reviewers: Q3.

Q3: Depth of Analyses in Sections 5.2 and 5.3

We have streamlined the presentation for code generation and relocated the GPT-related content to the appendix to allow for a more concise main text. In Experiment section, we have added more analysis and many case studies are provided for each environment in the Appendix D.

Q4: Problem Formulation Clarification

The issue with the index $i$ in the problem formulation section will be clarified and we now move the formulation section to the appendix.

Q5: Explanation of Scenario Development Effort

We have refined our explanation regarding the design of expert prompts for each environment and language agent. To minimize bias in comparative analysis, we randomized the sequence of language agent prompt design. This approach also addresses potential concerns about the fairness and adequacy of prompt design. However the expert guidance can hard to be controlled abosolutely fair and we can hard to predict how good it will be for a novel task. Thus our paper proposes Lv2, Lv3, and Lv4 scenarios to control the domain knowledge and reduce human efforts.

Q5: Necessity of Game Descriptions

We've included additional scenarios in the Appendix D.2, where EXE's performance in environments such as Cartpole, MountainCar, and MountainCarContinuous was notably lower without the basic game description. This demonstrates that EXE's effectiveness may be constrained when essential game details are not provided.

It's important to note that the game description is not crafted to enhance agent performance artificially; it typically exists as part of the task's framework. If a simulator for a task has been developed, it is reasonable to assume that a description of the game would be available. Human knowledge at Level 5 (L5) differs significantly from a mere game description; it is often constructed to intentionally boost the performance of Language Agents. For instance, in Cliffwalking, humans might outline the cliff locations, the goal, and even provide demonstrations for Language Agents—information that may not be readily available for a new task.

Q6: Release of Code

Our code has been open-sourced.

Q7: Clarity in Figure 3b and Supporting Data

We have revised Figure 3b for better visualization, offering a clearer representation of the performance metrics.

Q1: Scope of the Study

Thank you for your insightful suggestions. We have repositioned our paper to explore the potential of language agents as an alternative to PPO, which is reflected in the updated title: "Can Language Agents Be Alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym". This revision also addresses the concern of "beat RL with RL".
See details in Common Response Q1.

Q2: Omission of Mujoco, Atari Tasks, and More

Please refer to Response to all reviewers: Q2 and Q4. The evaluation costs are significant and cannot be overlooked. We have also made attempts on FrozenLake, which is challenging due to its stochastic dynamics. Even with optimal actions, performance can be negatively impacted. Therefore, we have not pursued further evaluations in this area for now, considering the cost of evaluation.

Q3: A Fairer Comparison Between Language Agents and PPO

We direct the reviewer to Response to all reviewers: Q2 for a more detailed explanation.

For details on PPO, please see Appendix C.4 in our revised manuscript. In our experiments, we utilized Tianshou in almost every environment and Taxi-v3 with stable-baselines3. These were trained and evaluated in OpenAI Gym, not TextGym. Regarding model size, our policy model has 8,964 trainable parameters when there are 3 actions and the input dimension is 1. This figure is the sum of elements across all parameter tensors that require gradient computation.

Q4: Additional Comparison with Reflexion

Firstly, we have provided pseudocode for both EXE and Reflexion in the Appendix to elucidate their differences and the novelty of EXE. There are three main distinctions:

1. EXE can generate suggestions without prior experience;
2. The critic in EXE is a language model that evaluates the trajectory based on suggestions guiding the actor's sampling process;
3. EXE specifically considers insights, exploration, exploitation, and their balance, whereas Reflexion focuses solely on score improvement.

We also added case studies between them in Experiment section 4.4, with a more detailed analysis presented in the Appendix D.

Q12: Termination Condition in TextGym Interface

TextGym utilizes a translator interface wrapped around OpenAI Gym to handle the termination conditions. The termination flag is managed by the gym simulator, and language agents are informed about the termination conditions within the game descriptions. This is particularly highlighted in Appendix B.1.

Q13: Actor's Exposure to Episode Data in Scenarios Lv2, Lv3, and Lv4

In scenarios Lv2, Lv3, and Lv4, the actor does not receive the $5$-episode data for action selection; it only receives the suggestions derived from the data.

Q14: Default Learner Specification

Details about the default learner are now specified in Appendix B.2, highlighted in blue for clarity.

Q15: Mapping of Scores and Raw Performance Data

The raw performance scores are documented in Appendix D.1. The normalized scores used in the experiments are computed as described in the experiment section on page 7.

Q8: Explanation for L4 Results Being Worse Than L3

A8: The GPT model inherently exhibits a degree of stochasticity, which can affect the outcome. When it comes to expert data, a lack of diversity can impede the agents' ability to generalize, a phenomenon observed in contrasts between imitation learning and reinforcement learning. This is supported by our case studies (Appendix 4.4).

Q9: Clarification on Scenarios L1, L2, and L4

A9: Indeed, L1, L2, and L4 are offline scenarios where agents do not actively explore the environment. In L1, EXE is unique in that it engages the learner, potentially aiding the actor with structured suggestions about exploration and exploitation. Supporting this, we observed in CliffWalking that EXE visited the goal in all five trials, compared to Reflexion, which did not reach the goal in any trial within the L1 scenario. For L2 and L4, the improvements in our updated version are modest and can be attributed to the structured suggestions provided by our model. In earlier versions, the notable performance gains in L2 and L4 were likely influenced by stochastic variation.

Q10: Peculiar Performance of PPO in MountainCarContinuous

For MountainCarContinuous, we conducted a comprehensive grid search covering various hyperparameters:

• Learning rate: {1e-3, 1e-4, 1e-5}
• Discount factor: {0.99, 0.95, 0.9}
• Weight for entropy loss: {0.01, 0.05, 0.1}
• Number of repetitions for policy learning: {10, 20}

Despite these efforts, the performance issues persisted. A significant challenge in this environment is the substantial penalty received by the agent when failing to reach the target, which often leads to a strategy of inactivity as a quick convergence solution. We also tested the implementation of PPO using stable baselines3.

It is worth mentioning that our goal was not to fine-tune PPO to achieve superior performance, as extensive parameter searching can equate to expert guidance akin to the L5 scenario. Achieving excellent performance through extensive effort (akin to prompt designing) is possible but usually not the preferred approach.

Q11: SAC Baseline and PPO Implementations on MountainCarContinuous-v0

We incorporated SAC results after searching for optimal alpha, learning rate, and gamma using the stablebaselines3 repository. While SAC was successful in MountainCarContinuous, it did not perform well in CliffWalking.

For each game, we performed a grid search for:

• Learning rate: {1e-3, 1e-4, 1e-5}
• Discount factor: {0.99, 0.95, 0.9}
• Entropy regularization coefficient: {0.1, 0.2, 0.5}
• Auto-entropy regularization coefficient setting.

Both PPO and SAC encountered challenges in certain tasks.

Table: SAC Performance Across Different Games

We genuinely appreciate your thoughtful feedback and guidance regarding the integration of our paper with existing literature and formalisms. Your provided insights will undoubtedly help us improve our overall presentation, better position our contributions, and enrich the related work section.

We have taken your suggestions to heart, and in our revised version, we have included a more thorough discussion of TextGame within the main text as well as in Appendix B. By incorporating these references, we aim to provide a more comprehensive understanding of the existing literature at the intersection of reinforcement learning, large language models, and text-based virtual environments.

We are grateful for your assistance in pinpointing the relevant literature in the field of text games, simulators, and actor-critic models. It has guided us to create a more robust understanding of the existing literature and has allowed us to better locate our work within that context.

Once again, we express our sincere gratitude for your valuable input, which has greatly benefitted our paper. We look forward to any further remarks or suggestions you may have.

-- Respect and Best Wishes from the authors :)

Q2: Experiment motivations and fairness in comparisons.

In considering alternatives to PPO, we posit the question: "Should language agents be regarded as alternatives to PPO for sequential decision tasks when some foundational knowledge is present?" Our aim is not to assert the superiority of language agents but rather to explore their potential applicability in this field. We proceed with the assumption that a researcher has ready access to a simulator and its accompanying documentation, and that the model size and the pre-training data of the LLM are not primary concerns when selecting an approach. Although "Gato"-like meta-RL methods may appear akin to language agents, their lack of open-source availability and their relative novelty in addressing new sequential decision tasks make them less prevalent in current research practices.

The prospect of a fair comparison between language agents and PPO is indeed compelling. Yet, the marked disparities between the two present challenges. For instance, it is impractical to align PPO and language agents in terms of network architecture or model size due to their distinct learning preferences. Consequently, our comparisons are made from the standpoint of practical users, who might weigh different factors when choosing between these methods.

Q3: Rigor and reproducible numerical results.

Our experiment now incorporates numerical results from $5$ different seeds to improve reliability, along with the interquartile range. We have open-sourced the code in this anonymous repository: https://anonymous.4open.science/r/Text-Gym-Agents-5220. Additionally, we have provided detailed descriptions of our implementations in the appendices, ensuring the reproducibility of our work.

Task selection from OpenAI-Gym: We do not involved specific tasks for our experiments for two main reasons:

1. Cost of Evaluation: The evaluation process is resource-intensive, both financially and temporally. For example, we excluded the BipedalWalker environment from our assessment due to its requirement of 2,000 steps per episode, which incurs a cost ten times greater than other tasks.
2. Complexity of Language Grounding: Certain tasks present substantial language grounding challenges. In environments where the difficulty is pronounced, such as FrozenLake, we limit our reporting to partial results without repeated experiments to maintain feasibility. Additionally, tasks with image-based states, like many Atari games or CarRacing in Box2D, require a distinct approach to language grounding. We recognize these as complex issues and have earmarked them for future investigation.

As noted in the Response to all reviewers: Q4, these constraints of cost and complexity have shaped our experimental design, focusing our resources on the most informative and manageable tasks.

Q4: Time and economic costs of the experiments.

The significant time and economic investments our experiments required are noteworthy, emphasizing the value of the preliminary results we have achieved and the rationale for the scope of our study. We anticipate that these initial findings will spur further exploration in the field. Constraints due to OpenAI API's rate limits resulted in a total inference time of approximately 64 hours for our experiments with GPT-3.5. The financial costs are also considerable; as indicated in the table provided, the expenses for conducting these experiments on GPT-3.5 amounted to roughly 2,611 dollars. This data underscores the extensive resources necessary for such research and highlights the importance of our foundational work. Following table show the toal costs on time and economic for each environment and total time spent on each language agent, respectively.