Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning

Q4: How such design sampling / regularization contributes to the better sample efficiency?

A4: Our ablation study (Figure 7 and Figure 8) demonstrates how the sampling and regularization components contribute to the improved sample efficiency of the proposed method.

• First, the annealed sampling strategy plays a critical role by balancing guidance from the LLM policy and the agent’s own policy during the imitation learning phase. By gradually reducing reliance on the LLM policy as training progresses, the agent is guided effectively in the early stages, avoiding poor exploratory behaviors while progressively learning to depend on its own policy. This strategy ensures a smooth transition from imitation learning to reinforcement learning, as evidenced by the superior convergence speed of our model compared to the baselines. In the ablation study, the Annealed Sampling model (III: NP+NS) achieves higher success rates and faster training compared to both the Base Model and Optimized Prompt, highlighting the importance of annealing in leveraging LLM guidance during exploration and contributing to better sample efficiency.

• Second, The KL-regularization term further enhances sample efficiency by aligning the policy network and value network with the LLM policy, enabling the agent to utilize structured guidance while ensuring stable updates. This regularization is particularly important during the imitation learning phase (IV: NP+NS+IL), where incorporating KL-regularized value updates improves the training of the value network, which in turn enhances exploitation in the reinforcement learning phase. Without these value updates, as shown in the ablation study (IV), the agent requires more training steps and achieves slightly worse performance.

• Last, when both the annealed sampling strategy and KL-regularization (policy and value updates) are integrated into our proposed model (V: NP+NS+IL+TV), the agent achieves the best performance among all tested variants. The smooth transition from the imitation learning phase to the reinforcement learning phase allows for effective initialization of the policy and value networks, ensuring high sample efficiency in both exploration and exploitation.

We hope our clarifications have addressed your concern!

Thank you for your detailed comments.

Q1: Novelty. To ground LLM as a policy planner in real environment, modeling the decision problem as hierarchical RL is well-discussed in literature. Though this work also uses LLM to help learning a RL policy beyond simple imitation, but the key regularized based method has been discussed in previous work.

A1: We acknowledge that using hierarchical RL to model decision problems for grounding LLMs as policy planners has been discussed in prior literature, such as the work referenced. In response, we have updated the Related Works section of our paper to incorporate a more comprehensive discussion of these studies like [1]. While previous studies like [2] provide valuable insights about our adapted regularization methods, our approach focuses on a lightweight, option-based framework that utilizes predefined high-level options rather than dynamically generating high-level actions. Our method also differs by integrating imitation learning and reinforcement learning in a two-phase process to address computational efficiency and token usage, which we believe is a novel contribution to this area. [1] Li, B., Wu, P., Abbeel, P., & Malik, J. (2023). Interactive task planning with language models. arXiv preprint arXiv:2310.10645. [2] Zhang, S., Zheng, S., Ke, S., Liu, Z., Jin, W., Yuan, J., ... & Wang, Z. (2024). How Can LLM Guide RL? A Value-Based Approach. arXiv preprint arXiv:2402.16181.

Q2: it seems that the main contribution is to leverage LLM in the exploration stage. I would suggest authors to better discuss and highlight the contributions

A2: To clarify, the main contribution of our work extends beyond simply leveraging light-weight LLMs in the exploration stage. In detail, IHAC incorporates several novel design elements. First, it uses separate learning phases to effectively leverage LLM guidance during the initial stage and refine the policy through reinforcement learning in the later stage. Second, it employs an adaptive sampling strategy that dynamically combines inputs from both the LLM and the RL agent during imitation learning. Third, it introduces KL-regularized terms applied to both the policy and value functions, which enhance stability and efficiency during training. We hope this clarification addresses your concerns and highlights the broader scope of our contributions.

Q3: Can authors provide a more detailed discussion on the importance and effectiveness of using LLM policy in exploration and exploitation respectively? How such design sampling / regularization contributes to the better sample efficiency?

A3: We would like to clarify the distinct roles of the LLM policy in exploration and exploitation within our framework. For exploration, the LLM policy plays a critical role in accelerating exploration, especially during the imitation learning phase. By leveraging the LLM’s general knowledge and reasoning abilities, the agent is guided toward high-level actions that are more likely to yield rewards or progress in the task. This significantly reduces the time spent exploring irrelevant or suboptimal trajectories, which is particularly beneficial in sparse-reward or complex environments where standard exploration techniques often struggle.

For exploitation, we assume that you are asking how does IHAC leverage the information collected so far. In fact, the agent relies entirely on its own learned policy to select actions. This transition from LLM-guided exploration to independent exploitation ensures that the policy becomes fully autonomous and is capable of optimizing performance without continued reliance on the LLM. This design not only minimizes computational and token costs but also ensures that the final policy adapts effectively to the task environment.

Thank you for your comments.

Q1: This work has nothing to do with hierarchical RL, however, this concept seems to be the key point and contribution of the paper. Hierarchical RL usually learns both high-level planning and low-level control. However, in this work, high-level actions are already pre-defined and provided and the agent does not learn the low-level control. The setting degenerates into the most common single-layer RL, just like the common robotics setting where high-level skills are provided. Lines 172-180 also do not show the mapping from high-level action to low-level control.

A1: Thank you for your comment. We acknowledge that our work does not involve learning low-level actions, as this is not a necessary component of our approach. However, we believe that the option-based framework in hierarchical reinforcement learning (HRL) does not inherently require learning the option space. As established in prior works, such as Sutton et al. (1999) [1], the distinction between hierarchical RL and standard RL primarily lies in the use of temporal abstractions, such as options, instead of primitive actions. In our work, we explicitly utilize time-dependent options for decision-making, which aligns with this key characteristic of HRL.

It is also important to clarify that hierarchical RL is not the central contribution of our work. Instead, our key contributions are:

• First, we introduce a novel framework that leverages large language models (LLMs) to guide high-level decision-making, particularly during the early stages of training. Our framework, IHAC, uses an external LLM to determine high-level options. This approach harnesses the LLM’s ability to provide actionable guidance, which is especially valuable when the agent lacks sufficient experience with the environment. As training progresses, our framework transitions to a standard RL algorithm to refine the policy, achieving a balance between LLM guidance and computational efficiency.
• Second, we propose an Adaptive Sampling Strategy that combines inputs from both the RL agent and the LLM during the imitation learning phase, resulting in more effective action derivation.
• Additionally, we design a high-level policy action distribution and a corresponding high-level value function to effectively guide learning. These innovations accelerate RL training by seamlessly combining imitation learning and reinforcement learning phases.

We believe these contributions demonstrate the value of our approach, even though it does not involve learning low-level control. If you have additional questions or require further clarification, we would be happy to address them.

[1] Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211.

Q2: The proposed algorithm is trivial and theoretically incorrect. In phase I, the learned value can only be applied to offline policy, since the agents also use LLM to sample actions. However, in Line 201, the authors claimed to run a standard RL algorithm like PPO.

A2: We believe there may be a misunderstanding regarding the design of our algorithm. Specifically, in Phase I, IHAC learns a value function while actions are sampled through the LLM. In Phase II, IHAC transitions to running PPO using the advantage function, which is based on the value function learned during Phase I. This ensures that all steps in our algorithm are theoretically sound and consistent with reinforcement learning principles.

We also respectfully disagree with the characterization of our algorithm as "trivial." IHAC incorporates several novel design elements. First, it uses separate learning phases to effectively leverage LLM guidance during the initial stage and refine the policy through reinforcement learning in the later stage. Second, it employs an adaptive sampling strategy that dynamically combines inputs from both the LLM and the RL agent during imitation learning. Third, it introduces KL-regularized terms applied to both the policy and value functions, which enhance stability and efficiency during training.

These contributions represent significant advancements over existing methods and have not been explored in prior work. We hope this clarification addresses your concerns regarding both the theoretical soundness and the innovative aspects of our approach. Please feel free to reach out if further clarification is needed.

Thanks for the clarifications, which solve some concerns. However, my major concerns still remain:

1. I am pretty sure that this work is not HRL. I am glad to see that the authors mentioned the sMDP paper. This submission actually works in an sMDP setting with temporally-extended actions, which indeed has some overlays with HRL but not exactly the same. None-HRL methods, like PPO can be directly applied to this setting without any modifications. And it is also a common setting in robotics where the low-level actions are pre-defined skills and the high-level policy is learned by RL. Readers will be misled by the term, "Hierarchical", and assume the low-level actions will also be updated. The references I provided are also in this setting. None of them claims that they are HRL.

2. It seems that the authors are not familiar with the LLM+RL research, which also partially caused the previous issue. In the related work section, works like ReAct and Reflexion have nothing to do with this setting and should not be mentioned here. The works I provideded studied exactly the same setting, however, it is a pity that the authors refused to even mention them since they use RL to train LLM instead of a small network, even though the subsection is LLM agent.

3. I agree that using a pre-trained value function can somehow empirically helps PPO learn faster, however, it definitely has some theoretical issues. PPO's actor is initialized from scratch. The sampled trajectory has a mismatch with the pre-trained critic. I recommand authors to provide more analysis why your algorithm can bridge this gap.

We would like to thank you for your thoughtful feedback and valuable comments on our work.

Q1: Novelty is limited as it primarily introduces a KL-regularized pre-training phase to the standard RL training process.

A1: Thank you for your feedback. We would like to clarify that our primary contribution extends beyond the introduction of a KL-regularized imitation learning phase. Specifically, we believe our approach is novel in the following ways:

First, our algorithm is designed to fully utilize the LLM during the early stages of training, providing effective guidance when the RL agent has limited experience with the environment. This significantly accelerates the initial learning process. Importantly, as training progresses, our method transitions to relying on the agent’s own policy and value network, enabling it to operate independently of the LLM in the later stages. This lightweight design makes our approach more practical and computationally efficient compared to other LLM-based RL methods, which often depend on the LLM throughout the entire training process. By reducing reliance on the LLM during the later training phases, our approach minimizes token usage, making it cost-effective and scalable—particularly in environments requiring prolonged interactions or extensive exploration.

Second, we propose a novel mechanism that balances guidance from the high-level policy distribution and the agent’s policy using an adaptive imitation ratio. This ensures a smooth transition from exploration to exploitation (as detailed in the "Annealing Strategy in Sampling" section of the paper). This mechanism enables a gradual and controlled shift from reliance on external guidance to the agent’s autonomous decision-making, enhancing overall training efficiency.

Finally, unlike existing methods that focus solely on policy updates, we introduce KL-regularization terms for both the policy and value networks. Regularizing both components ensures that the value network effectively contributes during the later training phases, resulting in improved efficiency and performance (see Equation (1) in the paper). Our ablation studies (Section 4.5, Figure 6) demonstrate the significance of this design choice. Training the policy alone without updating the value network leads to significantly lower efficiency. While performance is similar during the imitation learning phase, the policy-only method improves much more slowly during the reinforcement learning phase, requiring additional iterations for the value network to converge through PPO. This highlights the critical importance of training a high-quality value network in our method.

Q2: Extra hyperparameters are introduced, such as \alpha, \lambda_t, and the number of iterations of Phase 1.

A2: Thank you for pointing out the introduction of additional hyperparameters such as $\alpha, \lambda_t$, and $p$ (the number of iterations in Phase 1). To address this concern, we have conducted a detailed sensitivity analysis, which is included in the appendix of the revised paper. The sensitivity analysis demonstrates that while these hyperparameters are indeed introduced, their impact on the overall performance is minimal, as long as they are chosen within reasonable ranges. Specifically,

• For $\alpha$, our analysis shows that the model remains robust across different values of $\alpha$, with no significant degradation in performance as long as the balance between policy and value updates is maintained.

• For $\lambda_t$, our results reveal that $\lambda_t$​ primarily serves to gradually transition the agent from LLM guidance to autonomous policy learning. As shown in our analysis, the performance is largely unaffected by variations in $\lambda_t$​, provided that the decay schedule ensures a smooth reduction in LLM influence.

• The number of iterations in Phase 1 determines the duration of LLM-guided imitation learning. Our results indicate that the performance is relatively insensitive to changes in this parameter, as long as Phase 1 provides sufficient guidance for the agent to initialize its policy effectively.

We hope this clarification, along with the added sensitivity analysis, addresses your concern. Thank you for helping us improve the clarity and completeness of our work.

We would like to thank you for your thoughtful feedback and valuable comments on our work. We answer your questions as follows.

Q1: The reduction in token use may be due to early stopping: Additionally, the paper’s claim of a 90-95% reduction in token use likely stems from the fact that LLMs only generate guidance in the first phase.

A1: Since our algorithm consists of two phases, and LLM only plays in the first imitation learning phase, it is correct that the reduction in tokens comes from the reduction of the use of LLM. In fact, we believe that this reflects the true strength of our proposed algorithms: our proposed new frameworks can achieve the same or better performance compared with existing baselines, with much less tokens.

Q2: If the algorithm doesn't improve in the second phase, the improvement in token efficiency is less compelling. I recommend plotting the training curve and marking the transition to the second phase on the curve to highlight the effectiveness of the two-stage metho

A2: We appreciate your suggestion regarding the importance of reinforcement learning in the second phase. In response, we have updated Figure 8 in the revised manuscript to make the two phases more distinct for the experiments conducted on MiniGrid. Dashed lines now clearly mark the transition between the imitation learning phase and the reinforcement learning phase. As shown in Table 3, imitation learning accounts for 10% of the total training iterations, corresponding to 1.5k steps out of the 15k total steps. The updated figures provide a clearer illustration of the role and effectiveness of the second phase.

Furthermore, you observed that success rates for 3 out of 4 tasks appear to saturate after 10% of training, particularly in simpler environments like KeyInBox and TwoDoorKey, where imitation learning alone suffices for success. While we agree with this observation for these simpler tasks, it does not apply to more complex environments such as SimpleDoorKey and RandomBoxKey, where reinforcement learning is essential.

For the most challenging environment, RandomBoxKey, imitation learning alone is insufficient to produce satisfactory results, making the subsequent reinforcement learning phase indispensable. Our algorithm is designed not to settle for problems that can be solved solely through imitation learning but to extend its capabilities by leveraging reinforcement learning for tasks where imitation learning falls short. Our approach is further validated in more complex environments like NetHack, where the difficulty far exceeds that of MiniGrid, and imitation learning alone yields limited success. The combined two-stage framework effectively addresses both simple and complex tasks, as demonstrated by the training curves and results across diverse environments.

Q3: The reviewer is concerned that the use of ActionNet to translate high-level options into low-level actions via a pre-defined mapping might give your method an unfair advantage and points out that it is unclear whether the baseline algorithms also utilize ActionNet.

A3: Thank you for raising this important question. We would like to clarify that ActionNet, the translator used to map high-level options to low-level actions, is employed consistently across our proposed algorithm and the two baseline algorithms. This has been explicitly stated in the main paper and further emphasized in the appendix. Specifically, for any given fixed state, all three methods rely on the same ActionNet translator to determine the corresponding high-level action distribution. The primary distinction between our method and the baselines lies in how this high-level action distribution is utilized in subsequent operations. As such, we believe the comparison is both fair and valid.

Please let us know if you have further concerns or require additional clarification!