We thank the reviewer for their insightful review and for recognizing the novel perspective.

We appreciate the reviewer's concern about lacking comparisons between our top-down approach and bottom-up approaches that also leverage LLMs. This is a valid point, and we have now expanded our evaluation to include two additional strong RL baselines (DreamerV3 [1] and EDE [2]) as well as two LLM-based bottom-up approaches (LLM-as-policy and Eureka [3]) to provide a more comprehensive assessment of our method's effectiveness.

Extension to other RL baselines

We've added DreamerV3 (50M parameters) [1] and Exploration via Distributional Ensemble (EDE) [2], both highly ranked on the original Crafter leaderboard. DreamerV3 achieves slightly better performance than PPO (0.15 ± 1.39, N=4) but remains below our Strategist + PPO framework. Preliminary results indicate that EDE reaches high reward (~6 avg after 2M training steps) by initially discovering a hunting strategy and subsequently learning survival skills to lengthen episodes. Critically, our Strategist framework can be combined with EDE, enabling it to be steered toward distinct strategies such as hunting and breeding. While the hunting strategy performs comparably to early EDE stages, the breeding strategy—which was overlooked by vanilla EDE—enables a 2× reward improvement over EDE's final performance. This demonstrates that our approach is broadly applicable across different RL algorithms and can enhance the exploration capabilities of existing methods.

New LLM-based baseline experiments

We implemented an LLM-as-policy approach (shown to be effective in prior works such as Voyager and Reflexion), using GPT-4o-mini for direct action selection at each timestep. Notably, this is the same model used to annotate states in the Strategist’s reward shaping procedure to ensure a consistent LLM backbone across both baselines and our main method. This approach achieved 0.43 ± 1.01 reward (N=15), substantially underperforming our Strategist framework combined with PPO or EDE agents while being computationally expensive due to per-step inference requirements. Additionally, we evaluated Eureka [3] with GPT-4o, a state-of-the-art bottom-up LLM-based iterative reward discovery method. In our preliminary experiments with an evolutionary search (population=3, depth=3), Eureka's designed rewards didn't meaningfully outperform the environment reward, yielding results equivalent to baseline PPO while requiring a higher exploration budget. These comparisons directly demonstrate the advantages of our top-down strategic approach over bottom-up LLM-augmented alternatives in terms of both effectiveness and computational efficiency.

Distinction between bottom-up and top-down methods

We appreciate the clarification requested regarding the distinction between top-down and bottom-up methods. Indeed, we agree that bottom-up approaches augmented with LLMs can significantly improve sample complexity, and we emphasize that our top-down method is not inherently mutually exclusive but rather orthogonal to these bottom-up methods. Consequently, direct empirical comparisons were initially omitted as our goal was not to position these approaches in competition, but to present a complementary framework. For instance, methods using an LLM as a policy or hierarchical RL can be readily integrated alongside our Strategist framework to potentially further improve sample efficiency, as discussed in our future work section.

Additional comments and revisions

We acknowledge the importance of the detailed questions raised about Section 4.3, and will address these clearly in a newly added appendix section, briefly summarized here due to character constraints:

• Regarding feasibility, this probability represents how much of the probability mass of the current strategy aligns with policies achievable by the agent’s learning hypothesis class.
• Initially, attributes like specificity, feasibility, and value are estimated by the LLM based on its prior knowledge. As demonstrated in Section 5.3, these estimates can be dynamically updated with actual environment interactions, opening possibilities for future multi-armed bandit-inspired strategy adjustments during learning, which we encourage in our future work discussion.

Thank you again for your valuable insights and constructive suggestions.

References

[1] Hafner, Danijar, et al. "Mastering diverse domains through world models.".

[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. "Uncertainty-driven exploration for generalization in reinforcement learning.".

[3] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models.".

We sincerely thank you for your detailed and insightful feedback.

Additional RL baselines

Following your suggestion, we have added two competitive RL baselines: DreamerV3 [1] and Exploration via Distributional Ensemble (EDE) [2], both high ranking on the original Crafter leaderboard. DreamerV3 achieves slightly better performance than PPO (0.15 ± 1.39 (std, N=4)) but remains below our Strategist + PPO framework.

Preliminary results indicate that EDE reaches high reward on Crafter-Medium (~6 avg) by initially discovering a hunting strategy and subsequently learning survival skills to lengthen episodes further. However, our Strategist approach is orthogonal to the underlying RL agent: combined with EDE, the Breeding strategy—which vanilla EDE does not exploit—results in a 2× increase in rewards compared to EDE’s final performance. This highlights the value of our top-down approach in uncovering effective, alternative strategies that traditional bottom-up methods may not discover.

LLM baselines

We agree it is important to compare our method to other LLM-based baselines. We have added two additional LLM baselines: LLM-as-Policy and Eureka [3].

First, we implemented an LLM-as-Policy baseline with GPT-4o-mini, which selects actions at each step directly. This baseline achieved only 0.43 ± 1.01 reward (N=15), significantly underperforming compared to our Strategist framework. Furthermore, its computational cost is high due to requiring inference at each timestep.

Second, we compared our method to Eureka, an LLM-based approach designed for iterative reward discovery using GPT-4o. Eureka’s primary objective is finding high-quality reward functions, without aiming for sample efficiency. In our experiments, Eureka's evolutionary search (population size 3, depth 3) did not yield rewards meaningfully superior to the default environment reward, performing equivalently to baseline PPO despite utilizing greater exploration resources. For completeness, we will provide results with a deeper Eureka run in the revised manuscript.

SayCan is designed for robotic tasks using predefined action primitives and a pretrained value function for feasibility evaluation. This setup does not transfer well to environments where the space of meaningful behaviors is broader and the choice of primitives is not natural, plus the existence of a pre-trained low-level actor is a strong requirement, rarely available. Instead, our method leverages LLMs for high-level strategy generation and uses lang. cond. RL to ground those strategies through interaction. In practice, approaches like SayCan and action-proposing LLMs are actually orthogonal to ours—they focus on low-level decision-making given a set of primitives, which could be combined with our high-level strategic planning and exploration.

Additional comments

• Generalizability: we clarify that one of the strategist's strengths is precisely its flexibility. Our method is environment-agnostic conditioned on the LLM being able to generate meaningful strategies given domain descriptions, which is reasonable for a broad range of tasks. Unlike other methods, the Strategist does not require pre-training, expert demonstrations or predefined sub-goals.
• Hierarchical RL: Thank you for pointing out key prior work. We expanded Appendix D.3 to better situate our method within the hierarchical RL literature. In addition to citing Feudal RL (Dayan & Hinton, 1992), we now include discussions of MAXQ (Dietterich, 2000) and Feudal Networks (Vezhnevets et al., 2017), highlighting both differences and potential complementarities with our framework.
• Confidence intervals in Fig. 4: We acknowledge the nonstandard use of the 80% interval. We now include 68% intervals in the appendix, noting that this change does not affect the qualitative conclusions.
• Crafter environment modifications: We modified Crafter to better expose strategic trade-offs (e.g., hunting vs. breeding), making the evaluation of strategy discovery more explicit. Our method remains applicable to the original environment, but we found it less effective for highlighting the core contributions given its achievement-based reward.
• Minecraft-meat collection example: To address accessibility, we added a real-world motivating example (hospital management) in the appendix, illustrating how our framework supports discovering diverse high-level strategies in complex decision-making settings.

Thank you once again for your constructive critique and valuable recommendations.

References

[1] Hafner, Danijar, et al. "Mastering diverse domains through world models.".

[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. "Uncertainty-driven exploration for generalization in reinforcement learning.".

[3] Ma, Yecheng Jason, et al. "Eureka: Human-level reward design via coding large language models."

We appreciate the reviewer's feedback regarding our evaluation methodology. To address these concerns, we have substantially expanded our experimental comparisons to include two additional RL baselines specifically designed for sparse reward settings and long episodes (DreamerV3 [1], EDE [2]), as well as two LLM-based approaches (LLM-as-policy, Eureka [3]).

Model-based and exploration-focused RL baselines

We now include DreamerV3 (50M parameters) [1], a leading model-based RL approach, which slightly outperforms PPO but still falls short of our Strategist+PPO combination. Additionally, we evaluate EDE (Exploration via Distributional Ensemble) [2], another high-performing method on the original Crafter leaderboard. Preliminary results indicate that it achieves strong performance on Crafter-Medium (~6 avg reward after 2M training steps) by following a hunting strategy while learning to maximise survival to extend episodes. Notably, our Strategist framework is complementary to the underlying RL algorithm. When paired with EDE, it introduces a breeding strategy that vanilla EDE fails to uncover. This results in a 2× improvement in reward, highlighting how high-level strategic guidance enhances and expands the exploration capabilities of existing methods. This also demonstrates that our method is broadly applicable across different RL approaches and can draw from the strengths of each method depending on the environment of interest.

Comparison with LLM-based approaches

We have expanded our evaluation to include two prominent LLM-based baselines to clarify our method's advantages. First, we implemented an LLM-as-Policy baseline using GPT-4o-mini, which selects actions at each step directly. This baseline achieved only 0.43 ± 1.01 reward (N=15), significantly underperforming compared to our Strategist framework. Furthermore, its computational cost is high due to requiring inference at each timestep, which underscores its impracticality for complex environments.

Second, we compared our method to Eureka [3], an LLM-based approach designed for iterative reward discovery using GPT-4o. Eureka’s primary objective is finding high-quality reward functions, which often comes at the expense of sample efficiency. In our experiments, Eureka's evolutionary search (population size 3, depth 3) did not yield rewards meaningfully superior to the default environment reward, performing equivalently to baseline PPO despite utilizing greater exploration resources. For completeness, we will also provide results with a deeper Eureka run in the revised manuscript. These results clearly illustrate that our Strategist framework—translating high-level LLM-generated strategies into actionable rewards—is significantly more effective and sample-efficient than both direct LLM action-selection and standard iterative reward optimization methods, especially in complex, strategic domains.

Modified Crafter environment

Regarding the modification of the Crafter environment, our intention was to explicitly showcase the benefits of strategic exploration facilitated by our approach. By adjusting the environment to feature clear alternative strategies, such as hunting versus breeding, we aimed to rigorously test the Strategist Agent's ability to identify and exploit distinct high-level strategies effectively.

Comparison with SayCan

We thank the reviewer for highlighting important differences between our method and SayCan. Indeed, SayCan addresses bottom-up planning in robotic environments, which typically admit defined primitive actions. In such contexts, feasibility assessment through pretrained value functions is straightforward but requires costly pre-training of low-level actors, limiting sample efficiency.

Conversely, our top-down Strategist Agent is tailored for complex strategy-driven environments that inherently have a wide range of possible high-level actions. Unlike SayCan, our method does not rely on a fixed set of predefined primitives, making it suitable for broader strategic exploration. Our approach fundamentally capitalizes on the domain-agnostic strategic planning capabilities of LLMs combined with language-conditioned RL, enabling iterative refinement and robust exploration. Thus, our framework is uniquely positioned to effectively address settings where bottom-up methods like SayCan would be less feasible or impractical.

Thank you again for your valuable feedback and suggestions.

References

[1] Hafner, Danijar, et al. "Mastering diverse domains through world models.".

[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. "Uncertainty-driven exploration for generalization in reinforcement learning.".

[3] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models.".

We thank you for recognizing the strengths of our paper, including the clarity of writing and insightfulness of the formalism.

Clarifying the "human-inspired" framing

We acknowledge your concern regarding our use of "human-like" to describe the approach. Upon reflection, we agree that the term could imply unintended commitments to psychological realism, which was not our goal. We have revised the paper to replace "human-like" with "human-inspired," which better captures our intent to indicate that our method leverages high-level strategic decomposition reminiscent of human planning processes, rather than modeling human cognition per se.

While our primary objective indeed is not to ground our approach explicitly in human behavioral data or cognitive models, we recognize that some readers may find connections to this literature insightful. Therefore, we have included a targeted discussion in the appendix, briefly summarized here:

Hierarchical decomposition of goals has been central to human planning since the foundational work of Miller, Galanter, and Pribram (1960), and Lashley (1951). Humans naturally generate and evaluate multiple strategies, selecting among them by optimizing a cost–benefit tradeoff, as described by Lieder and Griffiths (2017). Moreover, humans utilize structural inference to abstract and simplify complex tasks (Gershman & Niv, 2010), supported by cognitive maps and mental models for flexible, model-based planning (Wang & Hayden, 2020). Lastly, the principle of bounded rationality highlights that humans regularly prune decision trees and favor simpler, cognitively less demanding plans (Huys et al., 2012; Lai & Gershman, 2024). These insights substantiate our strategist agent as balancing specificity (simplification and structure) against value (optimality), mirroring established principles of human decision-making.

We hope this clarification addresses your concern effectively.

Additional comments and revisions

We appreciate the point regarding our characterization as a "New Paradigm in RL." Upon reconsideration, we agree this phrase could be overstated. We have thus updated the description to "framework," which we believe is well supported by the formalization presented in Section 2.

We also thank the reviewer for highlighting specific presentation issues, such as the undefined terms "PPO" and "PAC." We have addressed these explicitly in the revised version to enhance clarity.

Finally, we have expanded our experimental evaluation to include strong RL baselines (DreamerV3, EDE) and LLM-based approaches (LLM-as-policy, Eureka). For detailed results showing our method's advantages over these baselines, please see our responses to the other Reviewers.

Thank you again for your valuable feedback.