Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models

We thank the reviewer for recognizing the contributions of our work, noting that "the integration of foundation models with the Go-Explore framework is a noteworthy contribution that can alleviate several of the inflexible inconveniences present in the original framework, such as hand-crafted heuristics and random exploration." We appreciate that you find our empirical evaluation solid and value our ablation studies.

Issue 1 and Question 1: Addressing Prompt Engineering Concerns

Thank you for raising this concern, we conducted experiments to show that IGE remains effective even with no hints on how to solve the task and just relying on purely factual environment descriptions. For example, as you suggested, in the BabyAI environment, we removed statements like "do not repeat actions if they have no effect". The results, shown in Table 1 below, indicate that even with minimal prompts, IGE strongly outperforms baselines (statistically significant, p < 0.05) with a domain-general prompt (save for a description of that environment). The evaluation setting is identical to that of Table 2 of our paper. Even with these minimal prompts, IGE outperforms baselines, indicating that extensive prompt engineering is not necessary. This shows that IGE is independent to any prompt engineering and instead is strictly additive on top of any foundation model agent architecture. We have updated the revised paper to reflect this finding in Appendix C.2.

Table 1: IGE Performance with Domain-General Prompts

Game of 24

BabyAI "Put Next To"

TextWorld "Cooking Game"

Issues 2, 4, and 5: Applicability to More Complex Environments

While the reviewer is right that the environments in the paper considered are “relatively controlled and intuitive”, they still pose a huge challenge for current RL algorithms. For example, we have a direct comparison to SOTA RL algorithms like the language model-based GLAM [Carta et al., ICML 2023] on the BabyAI environment and find that it gets next to 0 performance on the harder “open door” and “put next to” environments. We agree that testing IGE on Atari would be extremely interesting future work, but we believe it is out of scope for this paper which introduces a novel powerful algorithm and already shows strong results on environments that have eluded RL papers that have been accepted to prior conferences. Additionally, we believe that it is often best for algorithms to be introduced with problems that are simple and intuitive, as they help reveal the principles that make the new algorithm work well, and this is a common approach in machine learning papers.

Furthermore, we respectfully disagree that FMs are known to struggle in more complex environments. Indeed, we have already observed multimodal foundation models becoming an essential tool in extremely promising RL research, including being capable of playing classic video games like Doom [de Wynter, 2024] zero-shot without training, generalizing to a large number of (and even held out) 3D video games [Baker et al., 2021, Wang et al., 2023, SIMA Team, 2024], and succeeding on extremely difficult real-world robotics tasks, including “any-to-any” models such as RT-2 [Brohan et al., 2023] and RFM-1 [Covariant AI, 2024] that can reason over “text, images, videos, robot actions, and numerical sensor readings”. Overall, combining FMs + RL is one of the most active, promising directions in RL, and is in our view at least worthy of research to study its potential. We have added a discussion on this in the conclusion of our paper.

Since our algorithm is strictly additive on top of the capabilities of any FM agent, there is little reason to doubt that IGE’s capabilities will improve in tandem with the ever-growing capabilities of modern foundation models, and enhance them by adding the ability to reliably explore new environments.

Issue 3: Scalability and Computational Expense

While the reviewer is right that querying an LLM could be computationally expensive, our algorithm is remarkably fast compared to traditional RL since (1) there are no gradient updates and (2) traditionally RL is notoriously sample-inefficient. Indeed, we exceed the performance of the classic RL GLAM algorithm in Section 4.2 on the order of minutes, whereas the experiments in their paper took >10K GPU hours. Thus, we are actually significantly cheaper in computational cost. We have highlighted this point in Section 5 of the revised paper.

Looking to the future, we have already seen vast improvements to both the capabilities of LLMs and inference speed over the recent years (and corresponding dramatic reductions in cost), and we expect this trend to continue over time. Our new experiments (in Table 2 of the general response) show that Llama3-400B works well with IGE and is more than 10x cheaper. These developments directly improve both the performance and speed of Intelligent Go-Explore, and led to a corresponding approximately 10x decrease in the cost of IGE. Moreover, as foundation models (and compute in general) continue to exponentially decrease in cost, IGE will be able to be used in a broader range of important domains.

Issue 5: Importance of Action Selection

The reviewer is right that action selection is often the most significant factor in IGE's performance; however, we find that state selection in BabyAI where environment horizons are long also plays a critical role. Equally, without intelligent archive filtering, we would not expect to be able to tackle much larger and more ambitious environments as we would expect the state archive to be overwhelmed in these settings. In contrast, Table 3 shows that we can cut down the state archive size up to 5x in these harder environments, hinting at IGE's scalability to much larger state spaces. We have added this discussion to Appendix C of the revised paper.

Issue 6: Providing a More Grounded Discussion of Limitations

We appreciate the feedback and have revised the manuscript to tone down our discussion of these exciting future directions, making it clear that they represent potential avenues for future work that require thorough exploration. We believe many potential extensions of IGE are readily realizable; however, to provide just one example, as suggested by Reviewer xqm5, it could be quite powerful to do imitation learning by fine-tuning models like GPT-4 to distill IGE’s trajectories back into the base foundation model to improve performance. That would be relatively cheap on OpenAI's fine-tuning API. We have revised Section 3 and the conclusion to provide a more grounded discussion of limitations and future work.

Question 2: Alternate Action Selection Strategies

We completely agree that random action selection and FM action selection are two extremes. Our framework is completely flexible and we could even consider substituting the action selection policy in IGE with any RL policy. This would enable us to keep the foundation model IGE scaffolding that prevents detachment and derailment and allow us to use much cheaper options to roll out trajectories. We believe this would represent exciting future work and we have augmented the paper with this discussion in Appendix F.

---

We appreciate your detailed and constructive feedback. Please let us know if we have adequately addressed your concerns? If so, we would appreciate you considering increasing your score. If not, please let us know if there is anything else we can answer and/or address in the remaining discussion period!

Thank you for the rebuttal. However, my concerns remain insufficiently addressed:

• The new "domain-general" prompt is not provided, making it difficult to verify the claim that IGE is independent of prompt engineering. Moreover, crafting factual environment descriptions can still be as, or in certain cases more, effortful as defining heuristics in the original Go-Explore.

• The justification for avoiding Atari benchmarks, where Go-Explore excels, does not address whether IGE genuinely improves upon Go-Explore in true hard-exploration tasks. This remains critical for substantiating the paper's main claim.

• The foundation model-guided action selection is the main driver of IGE's performance, yet it is primarily compared to random actions. Additionally, examples of foundation models excelling in tasks like Doom or robotics, while promising, are external to the paper and do not directly demonstrate whether IGE, as implemented, can be applied to hard-exploration tasks. It feels too speculative to rely on improvements in foundation models or results from other works. Simply observing that GPT-4 outperforms GPT-3 does not imply a steady improvement trend, especially as recent advancements in LLMs have not yielded similarly substantial gains.

While the core ideas of the paper are potentially interesting and suitable for a venue like ICLR, the presentation and claims would benefit from a more modest tone and better evaluations to substantiate its strong claims. For these reasons I willl not raise my score.

Dear Reviewer ZTB4, Thank you for your continued engagement and valuable feedback. We have also uploaded a revised version of the PDF where we have adopted all of your previous suggestions. Below, we address each of your new points.

1. The new "domain-general" prompt is not provided, making it difficult to verify the claim that IGE is independent of prompt engineering. Moreover, crafting factual environment descriptions can still be as, or in certain cases more, effortful as defining heuristics in the original Go-Explore.

We apologize for not including the exact "domain-general" prompts earlier. We have now added these prompts to Appendix C.2 in the revised paper. These prompts are minimal, containing only basic environment descriptions without any task-specific hints or strategies.

Crafting factual environment descriptions requires significantly less effort and domain expertise than designing effective heuristics for Go-Explore. Often, these descriptions can be copied from existing sources like Wikipedia or code repositories. In our case, these are short and took on the order of minutes to copy. For example, the Game of 24 environment description was largely copied from the Yao et al., 2023 (as we explain in the appendix) which took seconds. On the other hand, finding effective heuristics for classic Go-Explore takes many rounds of experiments and iterations as you may not find optimal choices for them straight away. This takes much more time than simply copying and pasting a domain description. As we discussed, we believe the success of IGE is independent of what description we choose for the environment. In the experiments in Appendix C.2, we use the same reduced description for all baselines, and IGE strictly improved upon that baseline performance.

2. The justification for avoiding Atari benchmarks, where Go-Explore excels, does not address whether IGE genuinely improves upon Go-Explore in true hard-exploration tasks. This remains critical for substantiating the paper's main claim.

We believe the environments we evaluated are indeed true hard-exploration tasks. For example, in BabyAI and TextWorld, state-of-the-art RL (GLAM, ICML 2023) algorithms that have been published in major ML conference fail to find optimal solutions even after millions of environment steps, and sometimes only on a single seed of the environment. In contrast, IGE solves these tasks efficiently with less than a thousand environment steps, showcasing significant performance gains. These results demonstrate that IGE genuinely improves upon the prior SOTA in challenging exploration settings. Do you disagree that these environments are hard for RL algorithms?

While extending IGE to Atari games is valuable, it would involve significant computational resources (primarily because the environment horizon could be orders of magnitude longer than in the examples that we choose) and is beyond the scope of this paper. It also prices out small academic labs like ours to require these experiments which could take many thousands of dollars. However, we agree with the reviewer that this is an important direction for future research and the acceptance of this paper could allow us to write grants to try to acquire the funds to conduct this extremely important and valuable research.

3. The foundation model-guided action selection is the main driver of IGE's performance, yet it is primarily compared to random actions….**

Thank you for this question. We certainly agree that foundation model action selection is at the core of IGE’s performance but we do not believe this is not a downside. Indeed, the literature has shown that designing effective foundation model agents for complex environments is highly non-trivial and we strongly beat SOTA FM agents like ReAct (Yao et al., ICLR, 2023) which uses thought-action prompting and self-reflection agents (Shinn et al., NeurIPS, 2023). Intelligent Go-Explore demonstrates large improvements over these by considering additional mechanisms such as intelligent state selection and archiving in a Go-Explore framework—this helps unlock the remainder of the performance required to consistently get near optimal performance in many environments.

Is there a clear alternative that we should test instead that would work for the environments we choose? In this work, we are trying to as much as possible avoid training any neural network and solve challenging exploration tasks in-context with foundation models in a matter of minutes rather than in hours+ with RL. So is there a control with foundation models you would find more convincing? This is the best and fairest control we could think of. If there is not a better control you can suggest, is it fair to reject our paper on this front?

Dear Reviewer ZTB4,

Thank you so much for your feedback which has helped us improve the paper and for raising your score!

Thank you for your additional comments! Regarding the independence of IGE, we will clarify this further to specifically mean that reductions in prompts effect FM baselines across the board (including Naive LLM, ReAct, Reflexion) but IGE still consistently (statistically significantly) beats them, independent of initial performance.

We are happy that you agree that the current presentation takes a much more measured tone and frames the contributions more accurately. We will also take the suggestions to evaluate on Atari (e.g. Freeway or MinAtar) on board and endeavor to follow up this work with these exciting experiments!

Thanks again!

Best,

The authors

We thank you for your positive feedback and thoughtful questions. We are pleased that you find "exploiting the pre-trained biases of LLM for RL is an interesting idea and so far largely unstudied for exploration in reinforcement learning," and that "the paper is very clear and well written," providing "thorough ablations" of our method.

Issue 1: Addressing Hand-Crafted Features in Prompts

We acknowledge that providing environment descriptions in prompts involves some specification. However, we conducted experiments to assess the robustness of IGE to prompt variations. Specifically, we removed any hints on how to solve the tasks and kept the prompts purely factually descriptive about the environment. For example, as suggested by Reviewer ZTB4, in the BabyAI environment, we removed statements like "do not repeat actions if they have no effect". The results, shown in Table 1 below, indicate that even with minimal prompts, IGE strongly outperforms baselines (statistically significant, p < 0.05) with a domain-general prompt (save for a description of that environment). The evaluation setting is identical to that of Table 2 of our paper. This demonstrates that IGE's effectiveness does not rely heavily on prompt engineering, and the need for hand-crafted prompts is minimal compared to designing heuristics for traditional exploration methods. We have updated the final paper to reflect this finding in Appendix C.2.

Table 1: IGE Performance with Domain-General Prompts

Game of 24

BabyAI "Put Next To"

TextWorld "Cooking Game"

Issue 2: Examples of Serendipitous Trajectories

Thank you for this interesting question! One piece of evidence for seeing the benefit of serendipity relative to standard RL algorithms is the sheer difference in sample efficiency and performance. For example, our algorithm finds solutions to challenging RL environments orders of magnitude quicker than prior RL algorithms (e.g. on BabyAI, GLAM (Carta et al., 2023) fails to find optimal solutions at all (1-4% success rate on the hardest environments) even after millions of environment timesteps). Similarly, in the TextWorld "Coin Collector" domain, previous work (Xu et al., 2020) shows that millions of timesteps are required for agents to find coins in a single seed. In contrast, IGE succeeds in less than a thousand environment steps. This remarkable sample efficiency suggests that our algorithm effectively recognizes key states as they appear and captures serendipity as it happens.

Aside from performance indicating that it must be happening, one could also consider comparing the states selected by hand-designed heuristics (classic Go-Explore) to those discovered by Intelligent Go-Explore and then manually evaluate many to find a few that are surprising. This would be an extremely interesting study on its own but is non-trivial. For example, it requires some level of subjective judgment in choosing which manual heuristics for classic Go-Explore to try. It also requires one to manually review the states found by IGE vs. Go-Explore (and possibly vice versa) and then determine to what degree they count as surprising, to describe why, etc. The flavor of this would be similar to human observers recognizing “Move 37” in AlphaGo was completely unexpected even to Go experts. For all these reasons, we believe that such a study is beyond the scope of this paper and should not be a prerequisite for publication. However, we will mention in our paper that these interesting future research directions are motivated by our work and would be great future research directions.

Question 1: Transferring IGE to Continuous State-Action Spaces

Transferring IGE to continuous state-action spaces would be an extremely promising direction! Indeed, we have already observed multimodal foundation models becoming an essential tool in extremely difficult real-world robotics tasks, including “any-to-any” models such as RT-2 [Brohan et al., 2023] and RFM-1 [Covariant AI, 2024] that can reason over “text, images, videos, robot actions, and numerical sensor readings”.

Since the papers cited have already tokenized the state and action spaces of general environments into an FM and shown the ability of the model to generate actions for any state, it should be simple to then ask the FM to judge the interestingness of any given state compared to prior states observed, which is the only additional requirement of IGE. We have added this discussion to the conclusion of the revised paper.

Question 2: Managing the Size of the State Archive

IGE intelligently manages the size of the state archive in the TextWorld domain by using the foundation model to filter and retain only the most interesting and promising states, preventing the archive from becoming unmanageable (Section 3.3). Our experiments show that intelligent archive filtering significantly reduces the number of states while maintaining performance (see Table 3 in the paper and Appendix C). We did not observe significant issues related to archive size in our experiments. We have made the existing discussion in Section 5 more clear and added details in Appendix C.

Question 3: Handling Hallucinations During Trajectory Creation

The reviewer is right that foundation models have been known to hallucinate. However, in IGE, the foundation model does not create any trajectory (but is rather queried for state/action choices from real options) and exactly saves found states into an archive like the original Go-Explore. Therefore, IGE provides robust scaffolding for foundation model agents that explicitly prevent the hallucination of impossible states.

There is a concern that the LLM could pick impossible actions, but we observe that malformed inputs only consist of less than 0.1% of total actions. We discuss this in Appendix C, in these rare cases, we can simply take the simple choice of randomly sampling an action from the environment.

---

We appreciate your constructive feedback, which helped us improve the paper by clarifying how IGE handles prompt variations and potential hallucinations. These contributions enhance the understanding of IGE's advantages and robustness. Please let us know if we have adequately addressed your concerns? If so, we would appreciate you considering increasing your score. If not, please let us know if there is anything else we can answer and/or address in the remaining discussion period!

We thank the reviewer for their insightful feedback and for recognizing the originality and strong performance of our method. Below we address each of your comments. We hope you do not mind if we mention that we were surprised by how low your score is given your review, as it seemed that the issues you raised, while important, were not in line with that low of a score. The other reviewers gave the paper much higher scores (and offered to further increase them if we address their issues, which we have). All told, in light of all of this we hope you will consider increasing your score, as right now it makes the paper unlikely to be published and shared with the ICLR community. We (and the other reviewers) do think it makes a valuable contribution that should be published, and we hope you will also agree after all of the improvements we have made.

Issue 1: Clarification of "Serendipitous Discoveries"

We apologize for any confusion caused by the term "serendipitous discoveries". In the context of IGE, by "serendipitous discoveries" we refer to unexpected but valuable states that are encountered during exploration without prior anticipation by the human user. The original Go-Explore relied on hand-crafted heuristics to define ahead of time what types of states would count as interestingly new, which thus had no ability to recognize a state that is surprising, interesting, and potentially useful in an unanticipated way (e.g., a hand-coded heuristic might have declared that new x,y locations in the game and differing numbers of keys are interesting, but thus could not recognize that drinking a potion and then tripling in size constitutes a new, interesting state). IGE leverages the foundation model's internalized human notions of interestingness to recognize and add even entirely unanticipated novel states to the archive. This improves exploration by identifying valuable stepping stones that might otherwise be missed. As we quoted in the introduction, in the words of Isaac Asimov—“The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That’s funny'." We have clarified this definition in the revised paper, specifically in Section 1 (Introduction).

Issue 2: Adjusting Strong Claims on "Previously Impossible"

We agree that the phrasing "previously impossible" was not clear, and too strong without the proper scope. What we meant is that this was previously impossible within the Go-Explore framework. In comparison to Klissarov et al. (2023), our algorithm requires no training and finds solutions to hard RL environments within minutes. We have adjusted the language in the revised paper to more accurately reflect this, in Section 1 (Introduction).

Issue 3: Claims about Performance Scaling with Foundation Models

Thank you for this suggestion to include experiments with non-GPT foundation models. We evaluated IGE using other models (Claude Sonnet 3.5 and Llama-3 400B) on the Game of 24. The results, presented in Table 2, show that IGE consistently outperforms baselines (statistically significant, for all baselines with Llama-3 400B and for all except Reflexion on Sonnet 3.5, p < 0.05) across different models, demonstrating that IGE is a generic improvement that can be used in conjunction with any foundation model, likely including as new models are released that are even more powerful. We have included these findings in Section 5 of the revised paper.

Table 2: IGE Performance with Different Foundation Models on Game of 24 (Same Eval Setting as for Table 2 of the original submission)

Issue 4: Clarification on IGE's Complementarity to Other Frameworks

We agree that "strict improvement on top of any FM agent framework" is unclear. We have changed this in the revised paper to reflect that IGE is complementary to various FM agent reasoning strategies, such as zero-shot, few-shot, or chain-of-thought prompting. IGE integrates these agentic strategies with a structured exploration mechanism grounded in the Go-Explore framework, enabling better performance. We have clarified this in Section 3 of the updated manuscript.

Issue 5: Clarifying Empirical Results

Thank you for raising this point. While the performance gains may appear modest in some easier environments where baseline methods already perform well, IGE shows significant improvements in harder environments where exploration is more challenging.

For instance, our algorithm finds solutions to challenging RL environments orders of magnitude quicker than prior RL algorithms (e.g., on BabyAI, GLAM (Carta et al., 2023) fails to find optimal solutions at all (1-4% success rate on the hardest environments) even after millions of environment timesteps). In contrast, we find solutions on the order of minutes with less than a thousand environment steps. We have revised our presentation in Section 5 to highlight the significance of these results.

Similarly, in the TextWorld "Coin Collector" domain, previous work (Xu et al., 2020) shows that millions of timesteps are required for agents to find coins in a single seed. In contrast, IGE also succeeds in less than a thousand environment steps. This sample efficiency shows a remarkable improvement over prior SOTA in RL.

Question 1: Details on Baselines and Environment Descriptions

Yes, all LM baselines receive the same environment descriptions and observations for each game. This ensures a fair comparison, as any differences in performance are due to the methods themselves rather than discrepancies in the information provided. We have clarified this in Section 4 of the revised paper.

Question 2: Sensitivity to Prompt Variations

Thank you for this important question. We performed experiments to assess the sensitivity of IGE to prompt variations. Specifically, we removed any hints on how to solve the tasks and kept the prompts purely factually descriptive about the environment. For example, as suggested by Reviewer ZTB4, in the BabyAI environment, we removed statements like "do not repeat actions if they have no effect". The results, shown in Table 1 below, indicate that even with minimal prompts, IGE strongly outperforms baselines (statistically significant, p < 0.05) with a domain-general prompt (save for a description of that environment). The evaluation setting is identical to that of Table 2 of our paper. This demonstrates that IGE's effectiveness is robust to prompt variations and does not rely on extensive prompt engineering. We have included these findings and full details in Appendix C.2 of the revised paper.

Table 1: IGE Performance with Domain-General Prompts

Game of 24

BabyAI "Put Next To"

TextWorld "Cooking Game"

Question 3: Implementation of State-Conditional Action History

The state-conditional action history in IGE stores the set of actions previously attempted from each archived state. For each state in the archive, we maintain a list of actions that have been tried from that state, which helps prevent the foundation model from repeating the same actions and encourages exploration of new actions. This history is provided in the context when prompting the foundation model for the next action. This is relatively lightweight since our state archive is kept at a manageable size, as can be seen in Table 3 of our submitted paper. We have added a detailed explanation in Appendix D.3 of the revised paper.

Question 4: Exploration Challenges in Evaluation Tasks

Yes, the exploration challenges that Go-Explore addresses, such as detachment and derailment, are present in our evaluation tasks. For example, in the TextWorld "Coin Collector" environment, the agent navigates a maze-like structure where it can easily prematurely abandon promising initial paths (detachment) or struggle to return to previously visited rooms (derailment). Just as in the original GE, IGE mitigates these issues by archiving interesting states and reliably returning to them for further exploration, guided by the foundation model's intelligence.

Question 5: State Reset Mechanism

In our experiments, we assume the ability to reset the environment to any previously visited state, as in the original Go-Explore. For the explicit RL "gym"-style environments like BabyAI and TextWorld, this is straightforward and already implemented. In the text-based Game of 24, we build an environment that can recreate the state based on the archived information. Just as in the original Go-Explore, future extensions of IGE could consider prompting an LLM with past trajectories to return to desired states without the need for explicit reset mechanisms. We have clarified this in Section 2 of the revised paper.

Question 6: Difference Between "Classic Go-Explore" and "X All 3 above"

"Classic Go-Explore" refers to the original algorithm using hand-designed heuristics such as selecting states based on visitation counts, taking random actions, and adding unique states to the archive. In contrast, the "X All 3 above" variant in our ablation study replaces the foundation model guidance in IGE with unintelligent choices like random state selection (and the same unintelligent choices for actions and archiving). This helps demonstrate the importance of each component of IGE by comparing it to both the original Go-Explore and a fully ablated version. We have clarified this in Section 5 of the revised paper.

Question 7: Comparison with Exploration Methods in Games Like Minecraft

Thank you for highlighting these works. The focus and methodology of these works are quite different. Both Wang et al. (2023) and Zhu et al. (2023) employ agents that act in the environment via collections of high-level code or algorithmic policies tailored to Minecraft. In contrast, IGE provides a generic method to operate in any state and action space, given a minimal description of the environment, by leveraging the intelligence of foundation models at test time. The code policies used in these Minecraft agents are indeed interesting and could potentially be integrated into an IGE-like framework, representing a promising direction for future research into more efficient exploration agents. We have added a discussion on this in Appendix F of the revised paper.

---

We appreciate your detailed and thoughtful feedback, which helped us improve the paper by clarifying key concepts and addressing your concerns about scaling and methodology. Please let us know if we have adequately addressed your concerns? If so, we would appreciate you considering increasing your score. If not, please let us know if there is anything else we can answer and/or address in the remaining discussion period!

Dear Reviewer iSQ8,

Thank you for your detailed review and feedback. As we approach the end of the discussion period, we notice that you have not yet responded to our detailed answers to your concerns. Your score differs substantially from other reviewers who rated the paper considerably more favorably (with ratings of 6, 8, and 8). Since ICLR typically requires consensus among reviewers for acceptance, your current score could prevent the paper from being published despite the positive feedback from other reviewers. We have uploaded a revised PDF and worked diligently to address each of your concerns by:

• Clarifying the definition of "serendipitous discoveries": Added a precise and rigorous explanation in Section 1 to remove any ambiguity.
• Adjusting strong claims throughout the paper: We revised language to avoid overstating our contributions, ensuring all claims are accurate and appropriately scoped.
• Adding experiments with different foundation models: We conducted new experiments using Claude Sonnet 3.5 and Llama-3 400B, demonstrating that IGE consistently outperforms baselines across various models (Section 5).
• Conducting experiments on prompt robustness: We performed experiments with domain-general prompts, showing that IGE's effectiveness does not rely on extensive prompt engineering (Appendix C.2).
• Clarifying how IGE is complementary to other frameworks: We explained in Section 3 that IGE integrates well with various agent strategies and provides general improvements over existing methods.
• Enhancing the presentation of empirical results: We highlighted the significance of our findings in Section 5, especially in challenging environments where baseline methods perform poorly.
• Providing detailed responses to your specific questions: We addressed all your other queries, including baseline implementations, sensitivity to prompts, state-conditional action history, exploration challenges, state reset mechanisms, ablation studies, and comparisons with other methods.

Given these improvements and our detailed responses, would you be willing to review our rebuttal and reconsider your rating? If there are remaining concerns that would need to be addressed to merit a higher score, we would greatly appreciate your feedback before the rebuttal period ends.

Thank you, the authors.

We thank the reviewer for their extensive feedback and insightful questions. We appreciate that you recognize our paper "considers a fundamental problem of managing the exploration-exploitation trade-off" and "fits into the important line of work concerned with applications of foundation models in RL".

Question 1: Extension to Stochastic Environments

We appreciate this important question. We believe that IGE can be extended to stochastic environments, similar to how the original Go-Explore (Ecoffet et al., 2019) was extended (Ecoffet et al., 2021). In the 2021 Go-Explore work, a robust goal-conditioned policy is trained via imitation learning on the trajectories found during exploration, enabling generalization to stochasticity. In the context of IGE, we can similarly collect successful trajectories and provide them in-context to the foundation model, leveraging its ability to generalize and handle stochastic outcomes. We have added a discussion on this potential extension in Appendix F of the revised paper.

Question 2: IGE Resembling Hierarchical Search and Best-First Search

Indeed, the Go-Explore framework, and by extension IGE, shares similarities with hierarchical search and Best-First Search, as it prioritizes exploration from the most promising states. However, IGE builds on this approach by leveraging the foundation model's intelligence to dynamically assess the interestingness and potential of states, rather than relying on fixed heuristics. This allows IGE to adaptively explore the search space in a more informed manner. We have added a comparison and discussion of this relationship in Appendix F of the revised paper.

Question 3: Comparison to MCTS

Thank you for the fascinating question! For these extremely hard combinatorial problems like chess and Go, you would likely need an extremely good interestingness function. This relates to your later question on whether we can extract good interestingness functions from foundation models. If we could train one or otherwise, combined together with a solid value function just like in MCTS, IGE could work extremely well and discover novel and interestingly new strategies. We have added this discussion to Appendix F of the revised paper.

Question 4: Relation to GFlowNets

GFlowNets aim to sample compositional structures proportionally to a specified reward function, which differs from IGE's goal of exploring and discovering interesting states without predefined rewards. While both involve generating states through sequential decisions, IGE focuses on leveraging the foundation model's notions of interestingness rather than sampling according to a reward distribution. Moreover, GFlowNets are not directly intended for hard-exploration/sparse-reward problems. IGE should therefore be much better in hard exploration tests where there is little to no reward signal to guide search. We have added a comparison to GFlowNets in Appendix F of the revised paper.

Question 5: Real-life Applications of IGE

IGE's framework is very general, and we envision that it could be extended to many real-life problems involving exploration in complex spaces, such as discovering novel proteins in synthetic biology, or exploring mathematical conjectures. Recent work has already begun to explore applications of LLM agents across science, for example, Laurent et al. (2024) for biology research and AlphaGeometry (2024) for olympiad-level math problems.

On the practical side, LLMs have been adapted for various web-browsing and computer-based tasks (Liu et al., 2023) as useful personal assistants, many of which require exploration across long horizons (e.g., building an app from scratch).

We have expanded the existing discussion on this in the conclusion and Appendix F to add these exciting points!

Question 6: Accelerating Open-Endedness Research

We agree! The reviewer is right that open-endedness is often formulated as an unsupervised exploration algorithm, so we should aim to develop the best and most general open-ended exploration algorithms possible. Our algorithm provides a general mechanism to explore and discover a diverse set of interesting states or solutions in arbitrary environments without predefined objectives, much like human scientific exploration. We have added a discussion on this potential in Appendix F of the revised paper.

Question 7: Robustness and Potential Biases

This is a great question. Although representation in pre-training is hard to measure even for open-weight models, we have strong evidence that IGE’s success is relatively independent of the base LLM performance. For instance, in our new results in Table 2 of the general response, with Claude Sonnet 3.5 on Game of 24, the naive action selection performance is low (0.24), but IGE raises it significantly to 0.86.

The general question of bias is a very interesting one; for example, if a foundation model was biased to think that nothing “purple” is interesting, then we wouldn’t explore states that are “purple”. We do not observe such issues in our current evaluation (indeed, we would expect them to manifest less in games and RL/control problems rather than human-centric problems), but this raises an interesting point that is tied to the wider literature of reducing bias in foundation models in general. We have expanded on this in Appendix F of the revised paper.

Question 8: Ease of Setting Up IGE and Incorporating Additional Tools

Setting up IGE for a new task involves providing a brief environment description (e.g., from Wikipedia or a codebase which should require minimal knowledge of the domain). Importantly, our experiments with domain-general prompts (see Table 1 in the general response) demonstrate that IGE's success is independent of any domain-specific prompt engineering. We removed any hints about how to solve the tasks and kept the prompts purely factually descriptive. Even with minimal prompts, IGE substantially outperforms baselines, indicating that per-domain prompt engineering is not necessary.

Furthermore, since we use generic foundation model APIs to select novel states, actions, or when to archive, we can readily integrate further tools into the LLM calls, which could allow us to use things like web search for more hints or tools to do complex processing with ease. As we discuss in the paper, IGE is an extremely generic framework for exploration that is strictly additive on top of any foundation model strategy. We have added this discussion to Section 3 of the revised paper.

Question 9: Performance with Open-Source Models

We evaluated IGE using other foundation models, including Claude Sonnet 3.5 and the open-weight Llama-3 400B, to address concerns about IGE's dependence on specific models. The results, presented in Table 2 of the general response, show that IGE consistently outperforms baselines (statistically significant, for all baselines with Llama-3 400B and for all except Reflexion on Sonnet 3.5, p < 0.05) across different models, demonstrating that IGE is a generic improvement that can be used in conjunction with any foundation model, likely including as new models are released that are even more powerful. The evaluation setting is identical to that for Table 2 of our paper. We have added all of these results to Section 5 of the revised paper.

Table 2: IGE Performance with Different Foundation Models on Game of 24

Questions 10 and 11: Potential of Fine-Tuning or RL Training

Fine-tuning or additional RL training could potentially enhance foundation models' performance in specific tasks. In the context of IGE, one could envision fine-tuning the foundation model on the trajectories discovered during exploration, further improving its decision-making and ability to generalize. However, our current approach shows that significant gains can already be achieved without additional training, which is advantageous for efficiency. We discuss this in Section 3: "These solutions could easily then subsequently be used for downstream reinforcement learning or even improve the foundation model in the next task by in-context learning—thus allowing an agent to bootstrap its own learning indefinitely."

Furthermore, there could be equivalent benefits for even non-LLM-based RL algorithms. The trajectories that are discovered by IGE could be fed into a traditional imitation learning algorithm just as in the original Go-Explore. Or we could do offline RL on diverse data collected all throughout exploration and hope to get even better policies. We have added this discussion to Appendix F of the revised paper.

Question 12: Investigating LLMs' Notions of Interestingness

Understanding and formalizing the foundation model's notions of interestingness is indeed an intriguing direction for future research. In a similar way that reward models are collected on human preferences and used to fine-tune LLMs, we could also consider extracting interestingness preferences and fine-tuning an LLM to select more interesting states or actions. This could then lead to far more efficient versions of IGE, or even improvements to FMs themselves. We have added this discussion to the future work section in Appendix F of the revised paper.

Minor Comment on Notation

Thank you for suggesting to use the assignment operator in Algorithm 1. We have corrected the notation to accurately represent the addition of elements to a set in Algorithm 1 of the revised paper.

---

We appreciate your comprehensive and thoughtful questions, which have helped us expand and clarify important aspects of our work. Please let us know if we have adequately addressed your concerns? If so, we would appreciate you considering increasing your score. If not, please let us know if there is anything else we can answer and/or address in the remaining discussion period!

2. Performance with Different Foundation Models: We evaluated IGE using other foundation models, including Claude Sonnet 3.5 and Llama-3 400B, to address concerns about IGE's dependence on specific models. The results, presented in Table 2, show that IGE consistently outperforms baselines (statistically significant, for all baselines with Llama3-400B and for all except Reflexion on Sonnet 3.5, p < 0.05) across different models, demonstrating that IGE is a generic improvement that can be used in conjunction with any foundation model, likely including as new models are released that are even more powerful. The evaluation setting is identical to that of Table 2 of our paper. We have included these findings in Section 5 of the revised paper.

   Table 2: IGE Performance with Different Foundation Models on Game of 24

3. Clarification of "Serendipitous Discoveries": We have provided a precise definition of what we mean by "serendipitous discoveries" in the context of IGE and explained how IGE enables discovering and leveraging unexpected but useful states during exploration, without prior knowledge from the human to craft predefined heuristics. We have clarified this definition in the revised paper, specifically in Section 1 (Introduction), where we define serendipitous discoveries as 'states encountered during exploration that are valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user.'

4. Additional Clarifications and Discussions: In response to specific concerns, we have added explanations and discussions to improve the clarity of our paper. This includes addressing questions about IGE's extension to stochastic environments, comparisons with other methods like MCTS and GFlowNets, the applicability to continuous state-action spaces, and the potential for future research directions. These discussions have been added to Section 6 (Conclusion and Limitations) and Appendix F of the revised paper.

These changes address all the major questions raised by the reviewers. We believe the paper is now significantly stronger, and we deeply thank all the reviewers for their valuable feedback. We hope that the improvements we have made address your concerns fully. If there are any further questions or comments, we would be happy to discuss them. We kindly ask you to consider increasing your scores in light of these improvements.