STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models

1. Even though we are using the LLMs to create configuration files, each file contains the objects, actions, and skills that are necessary to certain games. This is like the approach from ScienceWorld which uses configuration files to specify game details.
2. Though we could use LLM’s here, they require lots of compute power than a simpler model. This is especially tune if we want to fine tune the models to a specific domain. It is much less compute-intensive and cheaper than using an LLM to solve the text-based tasks directly.
3. Can the reviewer provide alternative pre-training methods. The pre-training method that we used models how the agent would act in a test scenario. If we want the agent to learn specific skills, the agent plays our LLM-based games to learn these skills which can then be applied later in future environments.

1. We have created a rule based engine that converts the JSON files to inform7 code. The objects in the JSON generated by the LLM are configured to reflect all important aspects required to generate the inform 7 code for a game. The rule-based engine uses these objects to generate the required lines of code. Yes, Glulx is an interpreter that we use to run the generated inform 7 code after it’s been compiled by the inform 7 engine. We thank the reviewer for pointing out this mistake. We will rectify this mistake in the updated draft of the paper.

2. The pre-training games included various categories of skills to be picked up by the agent in order to complete the games. These tasks included skills like cooking, mixing two substances, planting a tree, finding hidden objects, boiling a liquid, etc. A similar set of skillsets (but, in a different setting) were required in the chosen games like finding a living/non-living thing, growing a fruit, etc. Figures 4 and 5 along with tables 2 and 3 compare the performance of a pre-trained STARLING agent with that of a Vanilla TBRL. The comparison shows that the pre-trained agent outperforms the Vanilla one. Thus, showcasing that the pre-trained agent is able to pick-up on the required skills for completing the games.

3. The game ideas are picked up based on certain skills that we would like the agent to pick up. The idea here is to create an environment that could be used to generate a set of games to teach the agent a certain category of skill. For e.g we used the game generation module to create games that require the agent to perform a sequence of complex tasks to achieve the goal like following a recipe for cooking, finding required tools for planting a tree, etc. The objective here is to showcase that the environment can be extended or used to leverage the generative capability of LLMs to teach any skill to the RL agent.

4. The variance in the figures are low because of the difficulty or complexity of the games. Even after the pre-training step, although the performance of these agents improve, but they still aren’t able to perform well on all the tasks except for task (4-2).

5. We thank the reviewer for pointing this out. The example on the right was meant to just showcase the kind of steps/actions taken by the agent. It is from a similar experiment but, does not belong to the exact experiment show-cased on the left.

6. Yes, the reason for this is due to the over-fitting of the vanilla baseline. It has poor generalisation capabilities. The TWC games are relatively simpler and easier than the ScienceWorld games. This is the reason for the stark difference in performances of the two agents on ScienceWorld games when compared to TWC games.

7. The main objective here is to showcase the generalisation capabilities of the STARLING baseline. Also, the ability of the game generation module to generate games for fostering development of agents that can acquire skills which can be leverage across environments. The same pre-trained STARLING agent was able to perform better across Zork, Science-World and TWC. We agree with the reviewer that perhaps a stronger baseline than TBRL to increase the difference even further.

8. We thank the reviewer for this point. Yes, as discussed in the point above, perhaps a stronger baseline can be used to increase the difference between the pre-trained and Vanilla agents. The can help make a stronger case for the main objective too, which is to showcase the generalisation capabilities of the STARLING baseline/

We thanks the reviewer for pointing out the typos and minor things. We will address all of them in the updated draft.

1. The skills learned using LLM refers to the skills learned from playing in the environments created by the LLM. This is an error in the wording of the paper which we will rectify.
2. We picked these 4 tasks because they represented complex and varied skills that would be necessary to be successful. These 4 skillsets can be extended to others and require agents to be generalizable. This is shown by the generalizability of the agents when they use pre-training. The pre-training games included various categories of skills to be picked up by the agent in order to complete the games. These tasks included skills like cooking, mixing two substances, planting a tree, finding hidden objects, boiling a liquid, etc. A similar set of skillsets (but, in a different setting) were required in the chosen games like finding a living/non-living thing, growing a fruit, etc. Figures 4 and 5 along with tables 2 and 3 compare the performance of a pre-trained STARLING agent with that of a Vanilla TBRL. The comparison shows that the pre-trained agent outperforms the Vanilla one. Thus, showcasing that the pre-trained agent is able to pick-up on the required skills for completing the games.
3. The 25 games were the test set. The 100 games were generated from the same distribution and were randomly split 75-25. 75 was the training set and 25 was the test set. What is meant exactly by the development set?
4. Humans were given five randomly selected environments from the set of 100 generated environments. They then played through those environments based on the actions that made sense. The environments were presented as Text-based games such as Zork. The human participants chose actions to take in the environment from a list of admissible actions until they reached a failure state or successfully completed the objective of the environment. The number of actions or steps and score that each human received was recorded.
5. The nature of ScienceWorld games includes dead-states/failure-states which is a wrong action that the agent can take that is irreversible. If the agent takes these steps they cannot complete the objective of the environment. An example of this is burning a necessary food ingredient to a certain recipe. The agent cannot reverse this action and leads to a failure state. The performance of these games varies dramatically due to how different and complex the distribution of games are. Even after the pre-training the agents is given the option of many options to take. of steps to complete further indicating how complex the games are. The performance drops at a later stage Since STARLING has been pre-trained, it has better generalization abilities as it is able to better avoid failure states rather than completely avoid them.
6. Yes, pre-training helps with initialization not just for Zork, but for TWC and other environments as well. Even though the vanilla performs similarly to STARLING in the training set, STARLING performs better in the test set indicating its superior generalizability.

1. The environments are designed in such a manor where the agent must pick from a pre-determined list of admissible actions rather than like a QandA style response. Therefore, data leaking is not a concern here.
2. LLM’s have been trained on a wide set of internet data already and likely already have generalization capabilities. Therefore, an LLM agent likely would not show the generalizability. By using a simpler and less complex model we show how agents can learn generalizability while being less complex. LLM’s are costly and large models, we are demonstrating generalizability among smaller, cheaper, and less complex models.
3. We picked only 100 episodes because the TBRL agents started to overfit after 100 episodes. The games are complex but not complex enough to warrant training past 100 episodes without the risk of overfitting.
4. We ensure there is no overlapping in the games that are generated. We want the agent to learn skills from the environments. The games that we create have different skill sets that are required. Since each game requires a different skill set, we know there is no overlap between the games.