Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning

We deeply appreciate your constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.

---

Q1. How is the temporal abstraction parameter c determined, given that subgoals are dynamically generated?

A1. The low-level policy dynamically determines when to end a subtask by analyzing environment observations, rather than strictly waiting for c steps, shown in L.214~215. This adaptive mechanism ensures efficient execution while maintaining the hierarchical control structure.

---

Q2. How does the framework evaluate subtask completion and handle failures when subgoals are abstract or when the low-level policy cannot execute the high-level plan effectively?

A2. The low-level policy executes subtasks, while the high-level policy learns finer-grained subtasks through feedback. In ScienceWorld, during early epochs, the high-level policy generates unachievable subtasks like "heat substance" but finds broken stove, later adapting to executable ones like "Navigate to foundry". Similarly in Minecraft, it evolves from "build a house" to specific tasks like "gather wood" and "create foundations".

---

Q3. How does ETO collect trajectories in the online regime, and what dataset does it use?

A3. For ETO baseline, we first perform SFT on the base model, then let it interact with the environment online to collect both failure trajectories and expert demonstrations for DPO training. We will provide more detailed baseline implementation details in the appendix.

---

Q4. What data did AWAC use in Section 4.4? Was the dataset emptied or did it retain the demonstrations from Section 4.2?

A4. Following AWAC, we kept the offline demonstration data instead of emptying the dataset, as it helps maintain base distribution performance. Unlike Section 4.2, Section 4.4 focused on evaluating adaptation to novel scenarios by selecting specific out-of-domain tasks and collecting additional data through online interaction

---

Q5. What exactly is the AC baseline in Section 4.4? A clearer explanation of these details would improve the study's transparency.

A5. The AC (Actor-Critic) baseline differs from AWAC(Advantage Weighted Actor-Critic) by using standard policy gradient instead of advantage weighting. We will provide more detailed implementation specifications in the appendix.

---

Q6. Section 2's literature review could be enriched with additional works (GLAM, POAD, and ELLM) on online RL approaches for LLM agents.

A6. Thanks for your helpful advice. We will adjust the related work section based on these suggestions.

---

Q7. Where is the empirical evidence showing how the generated subgoals differ between demonstrations and new tasks, and how do they support the claimed generalization ability?

A7. Our approach demonstrates cross-task generalization through qualitative and quantitative evidence. Appendix Figure 9 shows subgoal reuse between tasks, with both directly transferable components and analogous patterns. The effectiveness is validated on three out-of-domain tasks unseen during training, where GLIDER significantly outperforms baselines.

---

Q8. Why wasn't the low-level policy finetuned in offline-to-online experiments, and what would be the performance impact if it was?

A8. L.252~258 explain this design choice. Low-level skills use intrinsic reward functions instead of task-specific ones, allowing cross-task generalization and robustness to distribution shifts between offline training and online deployment. While joint finetuning could improve performance, adjusting only the high-level policy is sufficient given the task-agnostic nature of low-level skills.

---

Q9. Figure 2's method diagram is overcomplicated and hard to understand, with unclear connections and training flows.

A9. We will simplify the Figure 2. During value head finetuning, the LLM backbone remains fixed, while policy training updates the LLM to learn behaviors.

---

Q10. What is the rationale behind including a length regularization term in Equation 3?

A10. The length regularization term constrains action sequences to match the inherently short nature of valid environment interactions. By dividing the SFT loss by sentence length n as $-E[\log\frac{\pi_{\theta}(a|s)}{n}]=-E[\log\pi_{\theta}(a|s)] + \log n$, it effectively prevents unnecessarily long action generation.

---

Q11. The Appendix B mentions "Cross-task Generalization sampling". Were these data necessary?

A11. Distribution shift in offline RL leads to policy failures on out-of-distribution states. Cross-task sampling simulates potential shifts by exposing the model to varied scenarios, helping handle unseen tasks.

---

Best regards,

Authors

Methods

Q1. The optimal policy guarantee and the correctness of Eq. (7), given that actions are at the token level while rewards are formulated at the transition level.

A1. First, our policy derivation of inherits Advantage-Weighted Regression (AWR) [1-2], a popular algorithm using convergent maximum likelihood loss to convert RL into supervised learning subroutines. Our method follows their theoretical properties, and we will establish the theoretical counterpart in the appendix.

Second, GLIDER correctly attributes the full transition reward to each token through Eq. (7). Assigning the transition-level advantage $A$ to each token $w_i$ is equivalent to assigning the correct advantage to the whole action as $E[\exp(A(s,u))\cdot\sum_{i=1}^n\log\pi_{\theta}(w_i|s,w_{1:i-1})]=E[\exp(A(s,w_{1:n}))\cdot\log\pi_{\theta}(w_{i:n}|s)]$, using the relationship between total probability and conditional probabilities in autoregressive models.

[1] Advantage-weighted regression: Simple and scalable off-policy RL, arXiv:1910.00177, 2019.

[2] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, arXiv:2006.09359, 2020.

---

Q2. Superior parameter efficiency to what?

A2. The superior parameter efficiency is mainly reflected in the design of the RL model and the hierarchical structure.

First, we share the same frozen LLM backbone for actor and critic to greatly reduce model parameters, in contrast to many RL algorithms using independent actor and critic. Second, we let the high and low levels share the same model and differentiate them with a lightweight hierarchy prompt, as opposed to typical hierarchical methods with independent models at each level. Also, we adopt PEFT to further improve parameter efficiency following the common practice in LLMs.

---

Q3. Clarification about generalization, fine-tuning, and non-stationarity.

A3. In the offline-to-online stage, we focus on GLIDER’s fast adaptation ability to unseen tasks using a small number of fine-tuning steps, i.e., few-shot generalization. We will make more precise clarification.

---

Q4. Hierarchical RL needs domain knowledge.

A4. We design a generally applicable hierarchy to minimize reliance on domain knowledge. The low-level policy is instructed by an intrinsic reward indicating the sub-task completion, which is easily accessible fro environment observations, alleviating the necessity for any manual or task-specific design.

---

Experiments

Q5. A weakness is the lack of code.

A5. We have uploaded the code as supplementary materials for reproducibility when submitting the main paper. Further, we will release detailed tutorials and datasets to ensure full reproducibility.

---

Q6. Why training for only 5 epochs in SFT? Did you attempt to optimize its hyperparameters?

A6. We trained SFT for 5 epochs to avoid overfitting. Also, we optimized the hyperparameters and added dropout in LoRA layers, which can be found in our submitted code.

---

Q7. The dataset for SFT, the role of SFT, the comparison between SFT and GLIDER on expert datasets.

A7. We use expert datasets for SFT, matching the nature of supervised learning. The role of SFT is to construct a base agent to improve learning stability and sample efficiency for subsequent stages. GLIDER adopts RL to steer the base agent towards use-specified tasks, excelling both imitating demonstrations and adapting behaviors through trial-and-error. Experiments show a 11.6%~19.6% performance improvement over SFT.

---

Q8. Did the authors experiment with multiple seeds?

A8. Yes. Our method obtains close results across different seeds. The following table shows the example result of GLIDER with LIama-3-8B, score on unseen tasks in ScienceWorld.

---

Q9. Regularizer in Eq. (3), and using the end-of-text token in SFT.

A9. $n$ is the length of the sentence generated by policy $\pi_{\theta}$. We divide the original SFT loss by the sentence length as $-E[\log\frac{\pi_{\theta}(a|s)}{n}]=-E[\log\pi_{\theta}(a|s)] + \log n$. A smaller length $n$ will induce a larger gradient of loss, thus regularizing the model to output relatively shorter sentences. Also, we did use the end-of-text token in SFT, and our length regularizer is complementary to it.

---

Summary Response

We appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and clarifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

---

Q1. The authors could introduce GPT-based prompting baselines (Reflextion or ReAct) to compare whether small-scale LLMs with hierarchical training can match or even outperform larger models.

A1. Following your advice, we compare GLIDER using small-scale LLMs (LIama-3-8B) to ReAct using large-scale LLMs (GPT4). Results in the following table demonstrating that our small-scale LLMs with hierarchical training can match prompting baselines with much larger LLMs, with a 5%~98% performance improvement.

---

Q2. The novelty of the hierarchical approach in LLM to generate high-level plans.

A2. We innovatively combine the strengths of LLMs with the structural advantages of hierarchical RL to address the significant challenge of tackling long-horizon interactive tasks. While building upon existing research in LLMs and hierarchical RL, we propose a novel integration of these areas, design a parameter-efficient and generally applicable hierarchy, and enable fast offline-to-online adaptation.

---

Q3. The paper relies on environmental observations to indicate sub-task completion. Is this design applicable when extended to other, possibly more complex, environments?

A3. This design alleviates the necessity for any manual or task-specific design, and is naturally applicable in more complex environments. For example, in Minecraft, a broad task like "build a house" could be decomposed into simpler sub-tasks like "gather wood materials". The completion of that sub-task can be easily monitored by an attribute “inventory changes” of the environment observation. Our design remains robust at any complexity level, as long as the environment provides clear completion signals.

---

Q4. Did you observe any effects of the quality of high-level sub-tasks generated on the execution performance of the low-level actions?

A4. The quality of generated sub-tasks can affect the execution performance of the low-level policy. Intuitively, if the generated sub-task is too hard (e.g., close to the original task), the low-level policy can fail to accomplish it. That is why we need the hierarchical decomposition to break down the original task into simpler sub-tasks. By trial-and-error, the high-level policy adapts its behavior using environment-provided rewards, and is reinforced to generate increasingly fine-grained sub-tasks.

This “reinforcement” mechanism can be illustrated by a task evolution example: when the high-level policy generates a sub-task “heat substance” in the kitchen, the low-level policy attempts to use the stove but receives feedback that it's broken. After this failed attempt, the high-level policy adapts by generating a more specific sub-task “navigate to foundry to find blast furnace”.

---

Q5. Testing on more complex environments, such as WebShop.

A5. We will add a new benchmark, WebShop, to further validate GLIDER’s performance. Due to very limited time, we have only completed the offline dataset collection, and we are rushing to update them before the rebuttal deadline.

---

Q6. More extensive literature like RL-GPT and Thought Cloning.

A6. Thanks for your advice, and we will cite more relevant references.

---

Summary Response

We appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and justifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

Best regards,

Authors

---

Q1. About online fine-tuning, generalization, empirical evidence on fast online adaptation.

A1. Online fine-tuning refers to adapt a pretrained policy to unseen tasks using several fine-tuning steps, i.e., few-shot generalization to new tasks. We train a policy using offline datasets on all tasks except test-conductivity, find-animal, and boil. Then, we fine-tune the trained policy by interacting with these tasks online. The evidence is in Sec. 4.4. Generalization Analysis via Online Fine-tuning (L394-L429), also shown in the following table. GLIDER obtains a 65%-226% performance improvement over baselines.

---

Q2. Performance comparison between sharing weights and using different weights.

A2. We compare the full GLIDER to two more ablations: using different weights for the actor and critic, and using different for the high-level and low-level policies. The following table shows the performance on unseen tasks in ScienceWorld, with the LIama-3-8B backbone. We save 50% of model parameters with a <3% performance loss.

---

Q3. The novelty of combining existing ideas and demonstrates its use case in solving long context tasks for LLMs.

A3. We innovatively combine the strengths of LLMs with the structural advantages of hierarchical RL to address the significant challenge of tackling long-horizon interactive tasks. We propose a novel integration of these areas, design a parameter-efficient and generally applicable hierarchy, and enable fast offline-to-online adaptation.

---

Q4. It's not clear to me if they really do need a value function here or if it can be surpassed.

A4. Classical RL algorithms learn a value function to backpropagate expected optimal returns using dynamic programming updates. GLIDER learns a value function to estimate the action advantage, and regress the advantage-weighted policy. The existence of a value function is a crucial property of RL, which enables the agent to adapt behaviors through trial-and-error rather than only imitating demonstrations. The ablation study (L354-L369) shows a 13.9%~33.5% performance improvement of pure ORL over SFT.

---

Q5. L241: What is explicit modeling of the behavior policy?

A5. GLIDER maximizes an advantage-weighted likelihood function as $\max E[\exp(A(s,a))\cdot\log\pi(a|s)]$. We learn a value function to estimate the action advantage, and regress the advantage-weighted policy. The maximization of the likelihood $E[\log\pi(a|s)]$ corresponds to an explicit modeling of the behavior policy.

---

Q6. L213: How $n_h/n_l$ affects the gradient of loss.

A6. $n$ is the length of the sentence generated by policy $\pi_{\theta}$. We divide the original SFT loss by the sentence length as $-E[\log\frac{\pi_{\theta}(a|s)}{n}]=-E[\log\pi_{\theta}(a|s)] + \log n$. A smaller length $n$ will induce a larger gradient of loss, thus regularizing the model to output relatively shorter sentences.

---

Q7. L247-259: You first train low level and then fix it and later train only high level?

A7. At SFT and ORL, we simultaneously train the high- and low-level policies. Then, at the offline-to-online stage, we fix the low-level skills and only fine-tune the high-level policy.

---

Q8. L270: what is temporal abstraction knowledge?

A8. Temporal abstraction refers to abstracting a sequence of temporal actions into a skill, gaining the potential to greatly speed planning and learning on large problems.

---

Q9. L89: hierarchical token level not clear.

A9. “The hierarchical token-level actors and sentence-level critics” refers to that we have a token-level actor and a sentence-level critic at both levels.

---

Q10. It would be good to apply it on multiple datasets.

A10. We will add a new benchmark WebShop. Due to very limited time, we have only completed the offline dataset collection, and we are rushing to update them before the rebuttal deadline.

---

Summary Response

We appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and justifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

Best regards,

Authors

We deeply appreciate your positive and constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.

---

Q1. The training pipeline is a bit complicated (three stages), which might be tough for others to reproduce.

A1. We have provided code with the submission to ensure reproducibility of our main paper. Detailed tutorials and datasets will be released later.

While GLIDER involves a three-stage pipeline, each stage serves a distinct purpose. As shown in the following Table, where we compare the performance of baseline, ORL-only, and complete GLIDER implementations on unseen ScienceWorld tasks, the offline RL phase alone can still outperform baseline approaches. Adding SFT improves the results further by efficiently learning valid interactions, though at additional training cost. The optional online phase, enabled by AWAC's smooth offline-to-online transition, makes the overall process more robust and adaptable.

---

Q2. The paper doesn’t get deeply into real-world deployments or more physically grounded tasks, so scope could be potentially limited.

A2. This work demonstrates significant algorithmic value through a hierarchical approach that decomposes complex tasks into high-level planning and low-level execution skills. The effectiveness has been validated in sophisticated benchmarks featuring dynamic, sparse reward environments and diverse task scenarios. Meanwhile, these applications in environments with varied tasks and changing states demonstrate the robustness of this design, showing potential for embodied domains and enabling more sophisticated skill composition in robotic manipulation, game strategy learning, and autonomous navigation where multi-level control and temporal abstraction are essential for handling real-world complexity.

---

Best regards,

Authors

We deeply appreciate your constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.

---

Q1. What are the advantages of GLIDER compared to direct LLM planning approaches?

A1. We have conducted thorough ablation studies on hierarchical structure. As shown the following Table, hierarchical models consistently outperform their non-hierarchical counterparts across all LLM backbones, demonstrating the necessity of our approach.

---

Q2. What is the parameter efficiency of GLIDER and how does it compare with baseline methods in terms of parameter count?

A2. Based on our additional experimental evaluation on ScienceWorld using LLaMA3-8B as the base model, GLIDER demonstrates parameter efficiency in two key aspects: (1) With LoRA, it requires only 0.18GB trainable parameters while achieving superior performance over baselines ETO and NAT; (2) GLIDER with LoRA (0.58% parameters) achieves comparable performance to full fine-tuning (99.40% parameters), showcasing effective parameter utilization through prompting-based hierarchical control.

---

Q3. Why the benchmarks are appropirate?

A3. ScienceWorld and ALFWorld are ideal benchmarks for evaluating GLIDER, featuring dense and sparse reward structures respectively. The complex, non-deterministic nature of both environments requires the agent to continuously adjust its strategy rather than pre-computing a complete solution path, validating GLIDER's real-time planning abilities.

---

Q4. How does GLIDER handle sparse-reward scenarios?

A4. GLIDER addresses sparse-reward challenges through its hierarchical structure. High-level planning breaks down complex goals into manageable subgoals, while low-level execution ensures efficient exploration and clearer credit assignment for each subtask. This is validated by our strong performance compare to baseline in AlfWorld, a sparse-reward benchmark.

---

Q5. Grammar and Clarification of Terminology

A5. We appreciate the reviewer's suggestions and will thoroughly proofread the manuscript. Regarding the "long-horizon" task, it refers to tasks that cannot be fully planned at the beginning but require continuous planning throughout a long sequence of steps. For example, when "boiling water", if the sink breaks, the agent must replan to find alternative water sources.

---

Best regards,

Authors