Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals

We sincerely thank the reviewer for the insightful comments.

1. Theoretical Analysis of Diffusion-Based Subgoal Generation and GP Regularization

We appreciate the reviewer’s insightful comments on the theoretical justifications. In the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf, we have provided a formal, detailed proof addressing these concerns. Specifically, for the diffusion model, we rigorously demonstrate that, under a bounded noise prediction error, the reverse diffusion process converges to a valid state-conditioned subgoal distribution. Regarding the GP regularization, our analysis shows that the GP not only quantifies uncertainty but actively guides the diffusion process - pulling generated subgoals toward the GP predictive mean, thereby enforcing that the generated subgoals are both reachable and well-structured. This formal proof clarifies that the GP component does not merely filter outputs but plays a critical role in stabilizing subgoal generation in HRL. We invite the reviewer to refer to the additional theoretical analysis for details.

2. Subgoal Reachability Metric

We thank the reviewer for this valuable suggestion. In our submission, we already quantify reachability by reporting the distance between the generated subgoals and the reached subgoals (i.e., the final state after a k-step low-level rollout) in Table 1. This metric, which is consistent with the state-of-the-art subgoal generation method SAGA, provides a clear quantitative measure of subgoal reachability. Additionally, Figure 5 offers a qualitative visualization of generated versus reached subgoals in the Ant Maze (W-shape, sparse) environment, using the same starting location for direct comparison across methods. We invite the reviewer to refer to Table 1 and Figure 5 in the manuscript for detailed quantitative and qualitative comparisons.

3. Ablation Study of GP Alone Impact

Following the reviewer’s suggestion, we conducted an ablation study in the challenging Ant Fall environment specifically evaluate the impact of GP alone (Baseline-GP) by replacing the diffusional policy with the original baseline policy. From this comparison, we observe that GP-alone approach can improve the baseline by ~16%. This result highlights the significant performance benefit provided by the GP-driven regularization and uncertainty-aware subgoal selection, even independent of the diffusion policy.

4. Failure Cases

In our extensive experiments on the MuJoCo benchmarks, we have not encountered significant failure cases attributable to GP misguidance. In all tested environments, the GP consistently provided reliable uncertainty estimates that, when integrated with the diffusion model, improved subgoal reachability and overall performance. That said, we acknowledge that further investigation in even more challenging or real-world scenarios may reveal nuanced failure modes. We plan to explore these avenues in future work to better understand potential limitations. For now, our results indicate that within the tested settings, the GP component is robust and does not introduce harmful bias into subgoal generation or selection.

5. Quantification of Computational Overhead

We quantified the overhead from the diffusion model and GP: HIDI's inference time is ~37% longer than baseline, using ~15% more GPU memory and ~12% more RAM.

We sincerely thank the reviewer for the insightful comments.

1. Non-Stationarity Analysis: We clarify that HIDI does not claim to eliminate non-stationarity, an inherent challenge in HRL due to the evolving low-level policy. Instead, HIDI mainly addresses mitigating its adverse effects, in line with previous HRL methods such as HAC and HIRO.

• Theoretical Analysis: We have provided additional theoretical proofs of the HIDI framework (accessible via anonymous link - https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf) which detail: (1) the validity of using a diffusion model for state-conditioned subgoal generation, showing it can accurately approximate the target distribution, and (2) the role of GP regularization in guiding the diffusion process by aligning generated subgoals with the GP predictive mean, promoting achievable and structurally consistent subgoals. This analysis reinforces our empirical findings by providing formal justification for HIDI's effectiveness.
• Adaptive Subgoal Generation: Our approach employs a conditional diffusion model to generate a rich, state-conditioned subgoal distribution. Although this distribution is inherently non-stationary as the environment and low-level policy evolve, the diffusion model’s flexibility allows it to adapt quickly to new data. By continually training on relabeled subgoals, the diffusion model adjusts to reflect the current capabilities of the low-level policy. This results in subgoals that are more aligned with what the low-level can reliably achieve.
• GP Regularization: The incorporation of a GP prior further stabilizes the high-level policy by quantifying uncertainty. The GP provides structured guidance on regions of the state space that have been well explored, thereby encouraging the high-level policy to generate subgoals that are not only diverse but also practically achievable. In this way, while the underlying distribution remains non-stationary, the GP acts as a corrective mechanism, anchoring the subgoal proposals to areas supported by historical data.
• Empirical Robustness: Fig. 1 (smoother curves) and Table 1 (smaller generated vs. reached subgoal distance) show HIDI's improved alignment and robustness compared to baselines, indicating effective mitigation of non-stationarity issues.

2. Superior Sample Efficiency: We respectfully maintain that Fig. 1 supports our claim of superior sample efficiency, defined as achieving a performance level with fewer environment interactions (x-axis). HIDI consistently reaches high success rates faster than baselines.

• Steeper Curves: On challenging tasks (Reacher, Pusher, Ant Fall, Ant FourRooms), HIDI's learning curve rises more steeply, indicating faster convergence to high performance with fewer samples.
• Performance at 3M Steps: The table below shows HIDI achieves the highest IQM success rate at 3 million steps across nearly all tasks, demonstrating superior efficiency.

Table: Interquartile mean (IQM) of success rate with 95% stratified bootstrap confidence intervals, evaluated at 3 million steps:

We sincerely thank the reviewer for the insightful comments.

1. Comparison to Diffusion-based & Uncertainty-aware Baselines:

We respectfully clarify that our evaluation already includes strong baselines covering an uncertainty-aware approach, i.e., HLPS (Wang et al., 2024), and HIDI significantly outperforms HLPS on all tasks. Regarding diffusion-based RL, we emphasize that our experimental comparisons focused on state-of-the-art HRL methods, since our contribution lies in improving hierarchical RL decision making. Directly comparing to methods like Diffuser or Diffusion-QL would be inapplicable: those methods address offline trajectory optimization or action-generation, whereas HIDI addresses online subgoal policy learning. Nonetheless, we ensured our work is informed by those advances – for example, we cite Diffusion-QL’s success in modeling complex action distributions as a motivation. Rather than incorporate their entire pipelines (which would change the problem setting), we extracted the core insight (diffusion models can represent multi-modal distributions effectively) and applied it in a new context (subgoal generation). By deliberately using a basic diffusion model, we demonstrated improvements on top of existing HRL algorithms without relying on specialized offline-training techniques. We believe this choice actually strengthens our contribution: it shows that any future improvements in diffusion-based policy modeling can be combined with our HRL framework to yield even better results.

2. Benchmark Diversity and Alignment with Existing HRL Works:

We acknowledge the reviewer’s point regarding the use of simulated MuJoCo environments. However, our choice is fully aligned with the standard practice in HRL research. State-of-the-art HRL methods such as HIRO, HRAC, HIGL, SAGA, and HLPS have predominantly evaluated their methods on MuJoCo benchmarks, which provide a well-understood and controlled setting to measure progress in long-horizon, continuous control tasks. Our selection of diverse tasks within the MuJoCo suite (including Reacher, Pusher, multiple Ant Maze variants, and a vision-based Ant FourRooms variant) was deliberate to cover a wide range of challenges (e.g., sparse vs. dense rewards, stochasticity, and different state spaces).

Nonetheless, we agree that additional experiments on more diverse or real-world tasks (such as robotic manipulation on physical platforms or navigation in real environments) would further substantiate the generalizability of our approach. We are actively planning follow-up studies to evaluate HIDI in such settings.

3. Additional Theoretical Analysis: We have provided additional theoretical proofs of the HIDI framework (accessible via the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf), which addresses two critical aspects: (1) the validity of the diffusion model for state-conditioned subgoal generation, establishing conditions under which the reverse diffusion process accurately approximates the desired subgoal distribution and (2) the role of GP regularization in guiding the diffusion process by pulling generated subgoals toward the GP predictive mean, ensuring that subgoals are both achievable and structurally consistent. This comprehensive analysis reinforces our empirical results by providing formal guarantees for the effectiveness of our approach, and we invite the reviewer to refer to the additional theoretical analysis for full details.

We sincerely thank the reviewer for the insightful comments.

1. Low-Level Policy Training & High-Level Adaptation:

As correctly noted by the reviewer, our approach builds on the adapted HIRO framework, where the low-level policy trains on intrinsic rewards to achieve subgoals reliably, without direct extrinsic objective access. We focus on adapting the high-level policy because it faces non-stationarity – the set of feasible subgoals changes as the low-level improves. Adapting the high-level ensures generated subgoals align with current low-level capabilities while pursuing the task objective. Forcing low-level adaptation risks convergence to conservative local optima. Our approach (diffusion + GP regularization + selection) keeps the high-level agile, supported by ablation results (Fig. 2) showing improved sample efficiency and performance, and theoretical analysis (Appendix A.1, updated proofs linked below).

2. Ablation Study Clarity (HIDI-A, HIDI-B):

The reviewer's interpretation of the baselines is correct: HIDI-A denotes our framework without the subgoal selection strategy, i.e., the high-level always executes the diffusion model’s subgoal (with GP prior) without optionally selecting the GP’s predictive mean. HIDI-B is a variant with no GP regularization and no subgoal selection - the high-level uses only the diffusion policy for subgoals.

3. Clarifying the Role of the Gaussian Process (GP) Prior:

The GP prior is a critical component of our framework, and its contribution is twofold: (1) The GP models the state-subgoal distribution from the replay buffer. The $L_{gp}$ loss term (Eq. 11) regularizes the diffusion model towards generating subgoals consistent with previously achieved transitions. This focuses learning on feasible regions, reducing wasted samples. The ~15-16% performance gain from HIDI-B (diffusion-only) to HIDI-A (diffusion+GP) in Fig. 2a-b quantifies this benefit. Table 1 further shows HIDI (with GP) generates more achievable subgoals (smaller generated vs. reached state distance) compared to variants without GP. We have strengthened our theoretical proof (accessible via the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf) by analyzing the role of GP regularization in guiding the diffusion process by pulling generated subgoals toward the GP predictive mean, ensuring that subgoals are both achievable and structurally consistent. (2) The GP predictive mean $\mu_*$, reflecting low-uncertainty regions, guides exploration via our subgoal selection strategy (Eq. 18). With probability $\epsilon$, selecting $\mu_*$ steers the agent towards areas supported by experience, balancing exploration and exploitation. Removing the GP (HIDI-B) leads to less stable learning (Fig. 2).

To further substantiate our claims, we have conducted an additional ablation study in the challenging Ant Fall environment to specifically evaluate the impact of GP alone (Baseline-GP) by replacing the diffusional policy with the original baseline policy. From this comparison, we observe that GP-alone approach can improve the baseline by ~16%. This result highlights the significant performance benefit provided by the GP-driven regularization and uncertainty-aware subgoal selection, even independent of the diffusion policy.

4. Comparison to an Alternative High-Level Exploration Strategy:

Following the reviewer's suggestion, we have evaluated epsilon-exploration as an alternative strategy in the challenging Ant Fall environment:

From this comparison, we observe that epsilon-greedy exploration considerably degrades overall performance when subgoals are randomly generated with a probability of 0.1 (aligning with our subgoal selection scheme). This clearly demonstrates the benefit of our proposed subgoal selection strategy, which aligns subgoals with feasible trajectories, rather than relying on random subgoal sampling that is likely to yield infeasible subgoals.

5. Sparse Reward Setting Characterization:

We utilize standard implementations of the MuJoCo environments commonly used in HRL research. The "-1" sparse reward setting aligns with the existing reward formulations employed in prior state-of-the-art HRL papers evaluating on these environments (e.g., HIGL, DHRL, LESSON, HESS). Changing this widely adopted setup would require re-running all baseline methods, which is infeasible given the current time constraints.