LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models

We sincerely thank the reviewer for the positive evaluation and insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.

Q1. Multiple historical episodes input

Thank you for this point. Your understanding is correct: our current method updates the exploration strategy based on $M$ actions sampled uniformly from the single latest episode.

Intuitively, providing data from more historical episodes could give the LLM a richer context for its analysis. To see what will actually happen, we test 2, 3, and 5 episodes, where we average 3 repeated trainings with different random seeds:

We find that performance does improve when using more historical data, but there is a bottleneck. The best performance is achieved when using data from approximately 3 episodes. When providing too many episodes, it may result in an input sequence that is overly long and complex, which can hinder the LLM's ability to effectively process the information. More inputs also lead to a proportional increase in computational cost.

Q2. Output format of exploration probability distribution

Thank you for this detailed question. Yes, your understanding is correct. The LLM outputs only the probability distribution for the exploration noise, not a complete state-conditioned action distribution.

• For discrete actions, you are correct that our method integrates with an $\epsilon$-greedy framework. The agent chooses the highest-value action with probability $(1-\epsilon)$, and only with probability $\epsilon$ does it sample an action from the LLM-generated distribution.
• For continuous actions, we generate a bias vector corresponding to the dimension of the action space, which is then added to the original symmetric exploration noise to create a biased, directional exploration distribution.

We adopt this design for two main reasons. First, generating a full state-conditioned action distribution is highly complex, as states are often high-dimensional vectors or images that are difficult to obtain clean mathematical expressions. Second, by focusing only on the exploration distribution, our method produces a concise exploration signal that follows a form consistent with traditional methods. This ensures that LLM-Explorer functions as a simple plug-in module, making it easily compatible with a wide range of existing RL algorithms.

Q3. Convergence and computational cost

Thanks for this question.

First, our primary objective with LLM-Explorer is to improve the efficiency of the training process, enabling the agent to achieve better performance within a limited number of environment steps. This focus on sample efficiency is a standard paradigm in existing works [1-3], such as the Atari 100k challenge, and is the main goal of our work.

Second, we acknowledge the reviewer's point that full convergence can reveal different aspects of an algorithm's performance. As established by prior work [4], reaching full convergence on these tasks can require tens of millions of steps. Due to time constraints, we conduct a new experiment where we train agents for 1 million steps, a point at which the learning curves show a clear trend toward convergence. To address the computational cost concern and ensure a fair comparison, we allow the baseline DQN algorithm to train for more steps, matching its total wall-clock time to the extra LLM consumption in LLM-Explorer. For each task, we average 3 repeated trainings with different random seeds:

The results indicate that the final performance of LLM-Explorer is similar to, and in some environments slightly better than, the baseline. This aligns with our design goal. It demonstrates that our method primarily enhances exploration efficiency during the critical early and middle phases of training, while also ensuring that the agent reaches a high-quality final solution that does not sacrifice convergence performance.

[1] Model-based reinforcement learning for Atari. ICLR 2020.
[2] Data-Efficient Reinforcement Learning with Self-Predictive Representations. ICLR 2021.
[3] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. ICLR 2022.
[4] Bigger, better, faster: Human-level atari with human-level efficiency. ICML 2023.

Q4-5. State awareness and LLM’s prior knowledge about tasks

Thanks for this critical point. The initial state for each episode in the benchmark environments is randomly initialized as the default settings according to the gym package. Actually, our state-agnostic design is a deliberate choice. Here, we provide more discussions to better explain and improve our approach.

First, omitting state information is a trade-off between performance and efficiency. States in RL are often high-dimensional vectors or images, which are computationally expensive to process. As our experiments demonstrate, analyzing the previous action-reward trajectory, in conjunction with the task definition, is sufficient to guide exploration effectively.

Second, you are right that the reason why the method can operate without the state information may be the dependence on the LLM's prior knowledge of the tasks. To explore this, we conduct a simple test:

1. We first state the required format of the task description and manually craft a task description for the 'Alien' game from the Atari.
2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.

We find that the LLM can generate accurate descriptions from the game names alone, indicating prior knowledge of the well-known tasks. To test the framework's ability independent of this prior knowledge, we select three less common Atari games (outside the 26 core games [1]). For these games, we confirm that the LLM is unable to generate correct task descriptions from the name alone, suggesting a lack of specific prior knowledge.

We then manually provide detailed task descriptions for these games and train agents. For each task, we average 3 repeated trainings with different random seeds:

The results show performance improvement, demonstrating that our method is not solely dependent on the LLM's prior knowledge. Instead, it can analyze the provided task description and the action-reward data to produce an effective exploration strategy.

Third, we acknowledge that it is difficult to completely rule out the influence of all implicit knowledge within the LLM. In scenarios that are entirely novel, the performance of our state-agnostic method may be limited. For these cases, incorporating state information via a Vision-Language Model (VLM) is a possible solution. Specifically, we design an extended experiment:

• We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode.
• For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model follows our original state-agnostic design.

For each task, we average 3 repeated trainings with different random seeds:

Comparing the outcomes, the VLM-based approach shows an improvement. However, this comes at the cost of higher computational overhead due to the VLM's complexity.

In general, this highlights a trade-off. Our original state-agnostic design is intentionally lightweight, which is a practical and efficient solution for general use. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial. We will also add a more detailed discussion around the potential risk of this design to the limitations section of our paper.

[1] Model-based reinforcement learning for Atari. ICLR 2020.

Thanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please let us know if you have any further concerns.

We sincerely thank the reviewer for the positive evaluation and insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.

W1. Computational overhead

We agree that computational time and cost is crucial. Here, we report the time and API cost for 500k-step training. For each task, we average 3 repeated trainings with different random seeds:

On average, each run consumes $1.3 USD in API calls and takes about 10 hours. We consider this to be an acceptable and reasonable overhead.

W2. State awareness and LLM's prior knowledge about tasks

Thanks for this critical point. We provide more discussions to better explain and improve our approach.

First, omitting state information is a trade-off between performance and efficiency. States in RL are often high-dimensional vectors or images, which are computationally expensive to process. As our experiments demonstrate, analyzing the previous action-reward trajectory, in conjunction with the task definition, is sufficient to guide exploration effectively.

Second, you are right that the method may depend on the LLM's prior knowledge of the tasks. To explore this, we conduct a simple test:

1. We first state the required format of the task description and manually craft a task description for the 'Alien' game from the Atari.
2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.

We find that the LLM can generate accurate descriptions from the game names alone, indicating prior knowledge of the well-known tasks. To test the framework's ability independent of this prior knowledge, we select three less common Atari games (outside the 26 core games [1]). For these games, we confirm that the LLM is unable to generate correct task descriptions from the name alone, suggesting a lack of specific prior knowledge.

We then manually provide detailed task descriptions for these games and train agents. For each task, we average 3 repeated trainings with different random seeds:

The results show performance improvement, demonstrating that our method is not solely dependent on the LLM's prior knowledge. Instead, it can analyze the provided task description and the action-reward data to produce an effective exploration strategy.

Third, we acknowledge that it is difficult to completely rule out the influence of all implicit knowledge within the LLM. In scenarios that are entirely novel, the performance of our state-agnostic method may be limited. For these cases, incorporating state information via a Vision-Language Model (VLM) is a possible solution. Specifically, we design an extended experiment:

• We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode.
• For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model follows our original state-agnostic design.

For each task, we average 3 repeated trainings with different random seeds:

Comparing the outcomes, the VLM-based approach shows an improvement. However, this comes at the cost of higher computational overhead due to the VLM's complexity.

In general, this highlights a trade-off. Our original state-agnostic design is intentionally lightweight, which is a practical and efficient solution for general use. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial.

[1] Model-based reinforcement learning for Atari. ICLR 2020.

W3. More random seeds

Here, we increase the number of random seeds from 3 to 5 for all of our Atari experiments:

The results from the 5-seed runs are consistent with our original findings. Due to the time limit, we will add more random seeds for other experiments and update the results into the final version of our paper.

Q1. Temperature and stability

Thank you for noticing this detail. Actually, we have investigated the temperature parameter in our research process. With various temperatures, we average 3 repeated trainings with different random seeds for each task:

While a lower temperature reduces output variance, we find that it also leads to less diverse exploration strategies, resulting in limited performance gains. Therefore, we turn to the current setting of 1.0. But further increasing the temperature leads to a sharp performance decline. This suggests that excessively high temperatures introduce meaningless randomness rather than beneficial, flexible exploration. This also aligns with general recommendations from providers like OpenAI.

Q2. Adaptive update interval

This is a very insightful suggestion. Our existing results have shown that simply reducing the update frequency leads to a decline in performance (Table 3). Following your suggestion, we design a simple heuristic mechanism to trigger LLM updates more adaptively. After each episode, we calculate the average score improvement over the last 5 episodes ($g_1$) and the last 10 episodes ($g_2$). If $g_1$ is lower than $g_2$ by more than a given threshold $G$, we infer that the current exploration strategy may be losing effectiveness and trigger an LLM update. For each task, we average 3 repeated trainings with different random seeds:

The results show reduced average update frequency. Also, the resulting performance degradation is less severe than a fixed update interval change. We believe that more sophisticated adaptive triggers are valuable for future research to improve the performance-cost balance.

Q3. Smaller LLMs

It is interesting to explore the minimal capable LLM size. We test multiple Qwen2.5 models with varying sizes. For each task, we average 3 repeated trainings with different random seeds:

The performance degrades with smaller model sizes, and $\leq$7B parameters generally fail to work. Also, as you mentioned, fine-tuning a specialized LLM, for example, distilling the exploration strategy generation from a large LLM into a smaller one, would further reduce the minimum LLM size and save more computational cost.

Thanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please let us know if you have any further concerns.

We sincerely thank the reviewer for the insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.

Q1. Automatic task description prompting

We agree that this is an important point.

To improve our framework, we design to employ an additional LLM to automatically generate the task descriptions:

1. We first state the required format of the task description (line 120) and manually craft a task description for the 'Alien' game (Appendix G).
2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.

Prompt: You are writing a task description for {TaskName} to support the subsequent task solving. The required format is {TaskDescriptionFormat}. Here is an example for the Alien game from the Atari benchmark: {TaskDescriptionExample}. Please strictly follow the format and cover all details of actions, rewards, end conditions, and goals.

3. We replace the original, hand-written task descriptions with the automatically generated ones and re-run the experiments. For each task, we average 3 repeated trainings with different random seeds:

As the results show, LLM-Explorer with the auto-generated task descriptions performs similarly to the original one. This means that generating task descriptions can be automated without a significant drop in performance, which strengthens the scalability of our method to a broader range of tasks.

Q2. Failure mode and safeguards

It is true that LLM-Explorer is not universally effective and, as you pointed out, its performance is limited in certain tasks like Qbert and Seaquest.

According to our experiences and suggestions from Reviewer NcaS, we consider that one possible reason is that the LLM outputs extremely skewed distributions in some cases, which harm the training. To diagnose, we calculate the KL divergence of each distribution generated during training to a uniform distribution and examine the deviation between the maximum and average values.

This reveals that the two tasks with the worst performance indeed exhibit a much greater deviation, confirming the more skewed distributions. These extreme distributions might push the exploration policy too aggressively and hinder the learning.

Consider the characteristics of these tasks: they are both highly dynamic, featuring numerous moving objects and complex interactions. Such complexity makes it challenging for LLMs to obtain clear analyses. Also, the intrinsic hallucination of LLMs can cause the model to generate irrational exploration distribution in some cases.

As you suggested, we design a simple safeguard mechanism to mitigate this. Before applying the generated exploration distributions, we calculate its KL divergence to uniform. If this value exceeds the running average of past KL divergences by more than 10 standard deviations, the agent discards the skewed distribution for that cycle and falls back to the default uniform exploration strategy. For each task, we average 3 repeated trainings with different random seeds:

This demonstrates that such instability can be partially mitigated. Also, we believe the safeguards design you suggested is valuable for future research.

Q3. Extended baseline comparisons

We apologize for the insufficient baseline comparison. Here, we add comparisons against multiple recent adaptive exploration methods:

• RIDE [1]. A type of intrinsic reward that encourages the actions that changes its learned state representation.
• NovelD [2]. A method that weights all novel areas equally to balance the unexplored spaces.
• E3B [3]. A SOTA exploration method that encourages the agent to explore states that are diverse under a learned embedding.
• SOFE [4]. A design that expands the state space but simplifies the optimization of the agent’s objective. It can be combined with and augment frameworks like E3B.

For each task, we average 3 repeated trainings with different random seeds:

These modern techniques augment the extrinsic rewards with intrinsic rewards. In contrast, our method leverages the analyzing and reasoning capabilities of LLMs to provide context-aware guidance for the exploration. The results show that LLM-Explorer leads to the best performance.

[1] RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments. ICLR 2020.
[2] NovelD: A Simple yet Effective Exploration Criterion. NeurIPS 2021.
[3] Exploration via Elliptical Episodic Bonuses. Neurips 2022.
[4] Improving Intrinsic Exploration by Creating Stationary Objectives. ICLR 2024.

Q4. Performance of small and large LLMs

This is indeed an interesting observation. We thank the reviewer for prompting this deeper analysis, which improves the understanding of our framework.

First, to test our hypothesis that larger models "greedily fit specific actions", we measure the entropy of the exploration distributions generated during training.

The results show that the average entropy from GPT-4o is consistently lower. This indicates that GPT-4o indeed has a tendency to concentrate on fewer actions.

Second, we explore ways to better leverage larger models. As suggested, we design prompt engineering to encourage more creative exploration. We add the following instruction to the end of our original prompt:

Prompt: Please think creatively and consider all kinds of feasible actions. DO NOT limit the agent to a few kinds of fixed actions.

With the extended prompt, we average 3 repeated trainings with different random seeds for each task:

While this shows a slight positive effect, the performance improvement is limited, suggesting that simple prompting alone cannot fully overcome the model's inherent tendency.

Third, we investigate adjusting the temperature parameter. With various temperatures, we average 3 repeated trainings with different random seeds for each task:

We find that there is a delicate balance. Increasing the temperature from 0.5 to 1.0, which we use now, can improve the performance. But further increasing the temperature leads to a sharp performance decline. This suggests that excessively high temperatures introduce meaningless randomness rather than beneficial, flexible exploration. This also aligns with general recommendations from providers like OpenAI.

Finally, to observe if this counter-intuitive performance is universal. We test multiple Qwen2.5 models with varying sizes. For each task, we average 3 repeated trainings with different random seeds:

In the results, the performance degrades with smaller model sizes, and models with $\leq$7B parameters generally fail to work. This suggests that the "smaller is better" phenomenon is only likely to occur in certain model types and sizes comparisons (like GPT-4o vs. GPT-4o mini). Overall, larger and more capable models still tend to perform better.

Thanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please reconsider our paper and let us know if you have any further concerns.

Dear Reviewer 65Q1,

The author-reviewer discussion period will end soon on Aug. 8. Please read the authors' rebuttal and engage actively in discussion with the authors.

AC

Q3. Extended discussion on limitations.

Thank you for this suggestion. Based on our discussion and incorporating feedback from other reviewers, we provide the deeper analyses to the method's limitations and potential improvements.

1 Failure modes and safeguards

Failure mode 1: The LLM outputs extremely skewed distributions in some cases, which harm the training.

As we discussed above, the tasks with the worst performance exhibit more skewed output distributions. These extreme distributions might push the exploration policy too aggressively and hinder the learning. This situation is likely to happen to tasks with highly dynamics and complex interactions. Also, the intrinsic hallucination of LLMs can cause the model to generate irrational exploration distribution in some cases.

Safeguard 1-1: As we discussed above, we have considered a basic safeguard mechanism to mitigate this. If the output distributions are too skewed in the measurement of KL divergence, the agent discards the skewed distribution for that cycle and falls back to the default uniform exploration strategy.

To improve this, we further test an alternative version of the above mechanism. We recognize over-skewed distributions with the same measurement of KL divergence, but when the distributions are too skewed, the agent discards the skewed distribution for that cycle and continue to use the distribution from the last cycle. For each task, we average 3 repeated trainings with different random seeds:

The results indicate that this new mechanism, which reuses the last valid strategy, achieves better performance. Our current method for identifying over-skewed distributions, which is based on KL divergence (as advised by Reviewer NcaS), is a preliminary and simple approach. Designing more sophisticated mechanisms to more reliably identify exploration distributions that are detrimental to training remains a valuable and important area for future research.

Safeguard 1-2: Furthermore, considering that the intrinsic hallucination of LLMs can also cause the model to output an irrational exploration distribution, we design a simple mitigation strategy inspired by self-consistency work in LLM reasoning [1]. During each update cycle, we use the same prompt to have the LLM generate an exploration distribution twice. Due to the randomness introduced by the temperature setting, this process yields two different distributions. We then calculate the KL divergence to uniform for both outputs and select the distribution with the smaller KL divergence. For each task, we average the results of three repeated training runs with different random seeds.

This approach also demonstrates a slightly positive effect on performance. To better counter the negative impacts of intrinsic LLM hallucination within the LLM-Explorer framework, adapting other sophisticated methods from the field of LLM reasoning, such as self-reflection [2], presents another potential direction for future research.

[1] Self-consistency improves chain of thought reasoning in language models. ICLR 2023.
[2] Self-Refine: Iterative refinement with self-feedback. NeurIPS 2023.

Thanks again for your insightful comments and suggestions.

Hopefully, our discussions will solve your concerns. If so, please kindly consider raising the score as positive feedback for our efforts. Also, we would be glad to further improve our paper if you have any specific suggestions for further exploration and discussion.

Sincerely,
The authors

We sincerely thank the reviewer for the insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.

Q1. Automatic task description

The concern regarding the manual creation of task descriptions is an important point.

Following your suggestion, we design to employ an additional LLM to automatically generate the task descriptions:

1. We first state the required format of the task description (line 120) and manually craft a task description for the 'Alien' game (Appendix G).
2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.

Prompt: You are writing a task description for {TaskName} to support the subsequent task solving. The required format is {TaskDescriptionFormat}. Here is an example for the Alien game from the Atari benchmark: {TaskDescriptionExample}. Please strictly follow the format and cover all details of actions, rewards, end conditions, and goals.

3. We replace the original, hand-written task descriptions with the automatically generated ones and re-run the experiments. For each task, we average 3 repeated trainings with different random seeds:

As the results indicate, the performance of LLM-Explorer using the auto-generated task descriptions is similar to the performance achieved with the original manual descriptions. This demonstrates that the process of generating task descriptions can be effectively automated without a significant drop in performance, which strengthens the scalability of our method to a broader range of tasks.

Q2. State-aware variant

We agree that incorporating state information for more context-aware exploration is highly pertinent.

In line with this suggestion, we add new experiments to investigate the potential benefits of a state-aware approach. Given that the state in many RL tasks consists of image frames, we believe that a Vision-Language Model (VLM) is a more suitable choice to process complex visual information. Our experimental setup is:

• We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode. The VLM's task is to generate the exploration probability distribution for the subsequent episodes.
• For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model, following our original state-agnostic design, processes only the action and reward sequences to generate the exploration strategy.

For each task, we average 3 repeated trainings with different random seeds:

Comparing the outcomes, the VLM-based approach shows a modest improvement in performance, confirming that additional state context can indeed refine the exploration strategy. However, this comes at the cost of a significant increase in computational overhead due to the VLM's complexity.

This highlights a key trade-off. Our original state-agnostic design is intentionally lightweight. As our main experiments demonstrate, analyzing the previous action trajectory and reward feedback, in conjunction with the task definition, is sufficient to guide exploration effectively and achieve performance gains over baseline algorithms. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial. Taken together, our proposed text-only LLM-Explorer remains a practical and efficient solution for general use.

Q3. Compute budget

We agree that analyzing computational time and cost is crucial for assessing the practical application potential of our method. As the reviewer suggests, we report the wall-clock time and API cost for a 500k-step run across all Atari and MuJoCo tasks. For each task, we average 3 repeated trainings with different random seeds:

On average, each training run consumes approximately $1.3 USD in API calls and takes about 10 hours to complete. We consider this to be an acceptable and reasonable overhead.

To provide a more direct comparison with baselines under a fixed time budget, we conduct an additional experiment against NoisyNet. We count the time overhead used for the LLM API call in each update cycle of LLM-Explorer to allow the NoisyNet agent to perform additional training rollouts and updates. This ensures both methods operate within the same total wall-clock time. For each task, we average 3 repeated trainings with different random seeds:

In comparison, even when the NoisyNet baseline is allowed to train for more iterations, LLM-Explorer maintains a clear performance advantage. This shows that our design, calling LLM once every $K$ episodes, can well balance efficiency and performance.

Q4. Stability checks

It is true that LLM-Explorer is not universally effective and, as our results show, its performance is limited in certain tasks like Qbert and Seaquest.

The reviewer's suggestion to use KL divergence as a measure of distribution skewness is quite suitable for diagnosing these failures. Following this suggestion, we analyze the exploration distributions generated during training for each task. We calculate the KL divergence of each distribution relative to a uniform distribution and examine the deviation between the maximum and average KL divergence values.

This analysis reveals that the two tasks with the worst performance, Qbert and Seaquest, indeed exhibit a much greater deviation, confirming they experience more skewed distributions. These extreme distributions might push the exploration policy too aggressively in specific directions, hindering the agent's ability to learn.

To mitigate this, we design and test a simple safeguard mechanism. Before applying the LLM-generated exploration distribution, we calculate its KL divergence to uniform. If this value exceeds the running average of past KL divergences by more than 10 standard deviations, the output is considered an outlier. In such cases, the agent discards the skewed distribution for that cycle and falls back to the default uniform exploration strategy of the original RL algorithm. For each task, we average 3 repeated trainings with different random seeds:

This demonstrates that such instability can be partially mitigated. Also, developing more stable and LLM-guided exploration is valuable for future research.

Thanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please reconsider our paper and let us know if you have any further concerns.

Dear Reviewer NcaS:

The author-reviewer discussion period will end soon on Aug. 8. Please read the authors' rebuttal and engage actively in discussion with the authors.

AC