Can large language models explore in-context?

Additionally, the authors have not adequately cited related works [1-8].

Thanks for the references! We actually became aware of some of these between the submission and now and have already incorporated appropriate discussions into the manuscript. But we’ll be sure to discuss them in a revision.

Let us comment on [1-8], as per your list.

[2,3,5,6] are concurrent, unpublished works (from 2024), and should not be considered as novelty issues. (Note that our paper has appeared on arxiv since March 2024.) We’ll provide a detailed comparison in a revision*.

Of the remaining papers, [1,4] consider bandit tasks at a very small scale (T=10, 2 arms), which is really insufficient to distinguish between good and bad algorithms, and [7,8] have no bandit tasks at all.

All bandit tasks but [4] focus on LLM vs human comparisons, whereas we focus on LLM vs algorithms. We believe this is an important distinction.

*Two brief notes, though: [2] only experiments with Claude v1.3, which is considered roughly on the level of GPT-3.5 and hence probably not indicative of GPT-4 performance; [5] only considers bandit tasks at a very small scale.

[Q1] What factors within the model (size, architecture, RLHF, internal representation), its training dataset, and training protocol (at a specific point during training) could be preventing LLMs to fail at exploration? It may be beneficial for the authors to consider investigating some of these factors further, which could potentially enhance the quality of the paper. It would be beneficial to investigate the emergence of exploration behavior in LLMs by examining smaller models, such as GPT-2/LLaMa-2.

Thanks for the suggestion. We did experiment with LLaMA-2 in the paper (results are summarized in Fig 3 and reported with more details in the appendix) and found that it is much worse than GPT-3.5-Turbo, which itself is much worse than GPT-4. In a sense, this already demonstrates that exploration behavior is emergent, and that it perhaps can be improved further by scaling.

[Q2] It has been demonstrated that LLM models can be fine-tuned for reinforcement learning tasks, enabling them to function as decision-makers capable of generalizing to novel tasks with high efficiency [1-2]. These findings suggest that, despite their limited capacity for exploration when tested off-the-shelf, LLMs can be effectively fine-tuned to enhance their ability to explore. It would be valuable for the authors to investigate whether fine-tuning LLMs (such as the GPT-2 /LLaMa-2 model) on simple RL tasks could facilitate their exploration capabilities.

Thanks for these references. In our related work section (Appendix A), we discussed some works that train language models from scratch on trajectory data from another agent, resulting in suitable decision-making (and exploratory) behavior. Based on this, it’s not too surprising that LLMs can be fine-tuned to improve their decision-making capabilities; we’ll definitely include the citations. However, it’s worth emphasizing that fine-tuning requires expertise & money and/or infrastructure & access. Many practitioners are using (or will use) off-the-shelf LLMs for decision-making tasks without fine-tuning, and so it is critical to understand what is possible with standard training.

The failure mode of an agent following an epsilon greedy policy with a high epsilon value is analogous to that of Suffix Failure. Conversely, KMinFrac is analogous to the failure mode of an agent following a random policy or an epsilon greedy policy with a low epsilon value. Therefore, I believe the metrics themselves are quite trivial and do not constitute a novel contribution. Rather than relying on a qualitative comparison between LLM's choices and those of alternative bandit algorithms, the authors may find it beneficial to fit the models of exploration to LLM's choices and conduct a more quantitative evaluation

Both approaches seem perfectly reasonable to us but we respectfully disagree that one is “qualitative” while the other is “quantitative.” Our approach is completely quantitative: we suggest quantitative statistics to measure, demonstrate that they correlate strongly with performance (measured via reward over a much longer time scale), and quantitatively compare to state-of-the-art methods. Compared with the suggested modeling approach, our statistics allow for direct measurement of exploratory behavior without imposing any modeling assumptions (which would likely be wrong/erroneous given the erratic behavior of LLMs that we witnessed in our experiments).

Thank the authors for their detailed responses to our queries.

Re. Q1: Can the authors include a mixed-effects regression analysis, where they perform a regression from the abovementioned factors onto the surrogate measures to see how much each factor determines the extent of exploration and lack thereof? This allows us to understand the contributions of the factors at a more fine-grained level.

Re. Q2: We agree with the authors that expertise & money and/or infrastructure prevent fine-tuning to be a plausible option for everyone. But we differ from them in that there is a sufficient audience who would be interested if there is a simple fine-tuning protocol that makes LLMs explore optimally. We think smaller models can be a good test bed for such approaches and are quite confident that the protocol can even be standardized if shown to provide sufficient gains.

Re. external validity concerns as raised by two other reviewers: We agree with the authors that the bandit task provides a well-studied, controlled setting to study exploration in LLMs. However, given that these models are typically made to solve tasks vastly different from the bandit setting, We still have concerns about whether the findings will hold for those settings. If authors find the same effects in coding or some problem-solving tasks with the help of their prompting strategy, that would greatly increase the potential impact of the findings.

Keeping these points in mind, I am happy to increase my score from 3 to 5.

While the focus on multi-armed bandits provides a clean experimental setup, it may not fully represent the challenges of more complex reinforcement learning problems. The generalizability of the findings to broader settings is unclear.

Failures on a fundamental special case such as MAB plausibly carry over to the broader scenarios. But we completely agree for the positive findings. However, this would be the case for any experiment setup: the findings do not necessarily generalize beyond the precise experiment setting. Among all possible experiment setups, we argue that simple/clean/fundamental special cases are preferable. MAB is precisely such a setup.

The paper is predominantly empirical. A theoretical framework for understanding why LLMs struggle with exploration could provide deeper insights and guide future research.

Indeed! But we view it as a next step in this line of research. Our contribution is the prerequisite to this: we identify that there is a need for such a framework. Nevertheless, we hope that the suffix failure and MinFrac statistics provide some hint as to how one might develop a deeper understanding.

While the authors attempt to explore reasons for LLM failures in Section 3.3, this analysis feels underdeveloped. A more systematic investigation of failure modes could yield valuable insights.

Agreed! We included section 3.3 primarily to point out the difficulty of such investigation, particularly of using single-round experiments to isolate long-horizon issues. We can emphasize this in a revision.

While the authors compare summarized vs. raw history, more systematic ablations of different prompt components could help isolate the factors contributing to successful exploration.

Again we agree. For this we were primarily limited by cost for GPT-4. Note that we have reported results for all 24 prompt configurations for GPT-3.5-Turbo in the appendix, so “systematic ablations” on a weaker LLM are included in the manuscript.

In line 161, it is mentioned that no parameter tuning is performed for the baselines with tunable parameters. In this case, we are not sure that the baselines are performing to their highest capacity and therefore, the comparison might not be fair.” (Also, Q1)

This is rather standard in bandit experiments. It puts the baselines at a disadvantage, and even then they (for the most part) outperform the LLMs. So even without parameter tuning there is strong evidence that LLMs do not effectively explore.

Longer horizons T and replicates N might be required to obtain statistically significant results.

This is a fair concern. As we discuss in Section 2 ("Scale of the experiments"), our scale (T x N x #configs x #instances) is near-prohibitive given the high cost of invoking modern LLMs, yet insufficient if one relies on standard statistics such as accumulated rewards. This necessitates a more subtle study focusing on surrogate statistics focusing suffix and uniform-like failures.

We note that most/all follow-up or concurrent studies of LLMs in bandit problems were smaller-scale (e.g. [68]; [1-6] mentioned by Reviewer Jht5; Wu et al, "A benchmark for LLMs as intelligent agents".ICLR'24; Park et al., "Do LLM agents have regret?A case study in online learning and games. arXiv:2403.16843, 2024).

[Q2] In line 97, it is mentioned that “all the arms, except for the best arm, have the same mean reward”. What is the reason for this choice?

This choice (a) is the standard lower bound construction for MABs, and (b) has the fewest degrees of freedom, leading to a simpler and more defensible experiment setup. Of course other designs are also reasonable.

[Q3] If the summary is not available for more complex situations, can few-shot examples help instead?

We suspect so, and mention few-shot prompting as a natural direction for future work in Section 4.

As an experimental article, the findings are naive and obvious: the LLMs are not designed for solving decision-making tasks. It is consistent with intuition that LLMs can explore in-context when CoT with summarized history.

Whether LLMs are “designed” for decision-making tasks is not very relevant to our study. LLMs are already being deployed as decision-making agents (e.g. [40, 54, 63, 48, 5, 71]), and so it is crucial to understand their capabilities as such. Besides, LLMs are not explicitly “designed” for many tasks that they are good at.

We do not understand how LLMs not being designed for (or failing at) decision-making is consistent with the success of CoT and history summarization. Indeed, it was unclear if CoT would help much and LLMs are quite capable of performing the simple arithmetic required for history summarization.)

Re. “naive and obvious”: we emphasize that the scale of our experiments (as discussed in Section 2) was already near-prohibitive given the LLM costs, but still insufficient with the standard statistics (cumulative rewards), and hence necessitated a considerable subtlety (e.g., tracking suffix and uniform-like failures). The “prompt space” also required a somewhat careful design.

I feel that experiment results on more challenging tasks are necessary, such as some textual interactive games. In practical, the scenarios human encounters and explores are more complex than a bandit.

Real-world decision-making problems are indeed much more complicated than ours. However we emphasize that our MAB setting distills the essence of exploration (as justified by decades of theoretical research) and embeds as a special case into essentially all RL/decision-making formulations. If an agent cannot solve a simple task, what hope does it have for more complex ones?

Also: simpler settings require the experimenter to make fewer arbitrary choices, and can lead to more generalizable findings.

More discussions about related work are appreciated.

You might have missed our detailed related work section, deferred to Appendix A due to page limit. Nevertheless, we’d be happy to add any specific references you’d like.

There lacks a more systematic analysis of which specific prompt design elements are most critical for successful exploration? This could help in understanding the sensitivity of LLMs to prompt variations.

We already consider a non-trivial space of prompt designs, see Fig2. (And we note that it is at least comparable to that considered in concurrent work on bandit tasks for LLMs.)

We totally agree a more systematic analysis of prompt designs could be very useful. However, it is also extremely challenging because the prompt space (even just for simple bandit problems) is already massive. It might be more tractable to do this in simpler (non-sequential) settings, but LLM sensitivity suggests it is difficult to generalize the findings.

[Q1] Only one configuration (GPT-4 with chain-of-thought reasoning and an externally summarized interaction history) resulted in satisfactory exploratory behavior. Why this specific configuration was successful and others were not? Are there any insights into the underlying mechanisms?

An advanced LLM (GPT-4) and summarization are obvious benefits. The latter avoids occasional arithmetic errors that LLMs are known to make (and we found in our logs). CoT is known to help in many other tasks, but the reason is unobvious. One hypothesis (from other work) is that CoT gives the model access to more compute, which improves reasoning capabilities. In our experiment logs, CoTs often did describe correct algorithm behavior, like the correct posterior or upper confidence bound computations. We can describe this in more detail in the final version.

[Q2] The paper concludes that external summarization is essential for desirable LLM behavior in simple environments. How do the authors envision scaling this approach to more complex settings where external summarization might be infeasible? Are there any preliminary ideas or experiments that could address this limitation?

This is a fundamental open question for future work. One perhaps naive idea is via “orchestration”: we invoke the LLM once and ask it to summarize the history, then we use this “self-produced” summarization in a second invocation where we request a decision. While not provably correct, such summaries seem simple enough to experiment with.

The results may indicate the intelligence embedded in LLMs is still not aligned with human.

True. However, we focus on comparing LLMs to algorithms rather than humans.