Uncertainty-Regularized Diffusional Subgoals for Hierarchical Reinforcement Learning

1. HIDI's exploration is buit upon the idea of uncertainty. However, state space uncertainty struggles on high-deminsional tasks, such as MDPs with image observations. I would like to see more discussion on this issue.

• The paper does not well explain how diffusion models help to address the non-stationarity issue in HRL.
• The impact of exploration and the GP-based uncertainty modeling on performance is not adequately analyzed. Figure 3 shows HIDI without explroation has similar performance as HIDI.
• The proposed approach relies on Gaussian Processes, which typically face scalability issues as the number of training points increases. Although inducing points are introduced, there is no detailed discussion or analysis of how well the approach scales to larger.
• The environments used in the experiments are limited. This makes it difficult to evaluate the improvement of the proposed method compared to previous algorithms comprehensively.
• In environments with high randomness, such as the stochastic Ant Maze, the advantages of the proposed approach do not seem to be very apparent.

• It is not clear how the proposed algorithm helps with non-stationarity issues, and how this issue is disentangled from other factors in HRL.
• It is stated in the paper that HIDI is more sample efficient, however, sample efficiency is not directly measured in the experiments.
• More analysis of the components of HIDI including the hyperparameters would be beneficial
• Not enough reproducibility details for other baselines

Minor fixes include:

• Line 527 should be Figure 5, not 4.
• Figure 3, should mention the type of shaded regions and the number of seeds
• Experiments and related work should be separated

1. The paper poses a problem of off-policy learning in the abstract and introduction, which was the same problem HIRO was designed to solve. However, the contributions of this paper upon HIRO does not appear to address the non-stationarity issue. Instead they propose a diffusion model as a subgoal generator, without motivating how such a model can address the off-policy problem.
2. Additionally, their second contribution, the GP prior seems to model the uncertainty over the subgoals. This was not directly related to the original off-policy problem, rather to help with uncertainty estimation on these diffusional subgoals.
3. The paper claims the GP model “potentially reduces the amount of training data needed for the diffusional model to converge”, but this was never tested. An experiment that could verify whether this is the case would be very useful for researchers seeking to incorporate diffusion as a generative model in reinforcement learning.
4. Line 250 is needlessly speculative: GPs were claimed “potentially leading to more robust and sample-efficient learning”, but Figure 3 shows that ablating the GP does not result in any significant performance drop.
5. It was stated that “diffusion models typically require a substantial amount of training data”, it would have been nice if the paper cited instances of this problem happening.
6. The paper claimed HIDI was more sample efficient than a count-based alternative (HESS), but they only show the performance of the two algorithms, no metric measuring exploration. Could HIDI be more sample efficient because it is exploring less? Since exploration is not directly related to the main problem the paper is addressing, this is not detrimental, but the reader may find the result distracting.
7. The paper’s last contribution of using inducing states is a clever trick to reduce the computational complexity of the GP, but they claim this gives better exploration. This was not empirically tested.
8. The exploration aspect of the sparse GP seems orthogonal to the paper’s initial objective of addressing the non-stationarity of a changing low level policy. Wouldn’t adding exploration make the higher policy training even more off-policy, as the low level policy has an epsilon chance of exploring?
9. The Environments and Empirical Analyses are written as subsections of the Related Works section. This should be a separate section.
10. It should be clarified that all the claims of “How do various design choices within HIDI impact its empirical performance and effectiveness?” are for the AntMaze W env

Minor Fixes

1. Is the sparse reward setting really sparse? The -1 per step signal is still informative for the lower level policy which reaches subgoals. A sparser reward could be 0 everywhere and +1 at the main goal.
2. Line 176: make the N calligraphic and the \mu and \Sigma boldface to match the equation.
3. Line 179: The learnt noise \boldsymbol{\epsilon}_{\theta_h} was introduced, but never defined anywhere. This may be standard notation with diffusion models, but having it explained (even in the appendix) can help readers with a reinforcement learning background understand.
4. Line 227: Introduces \mathbf{X}, but never defines or uses this matrix anywhere.
5. Line 254: Typo “Assuming the hyperparameters θgp are learned and fixed” -> “Assuming the hyperparameters θgp are learned and then fixed”
6. Line 259: This sentence needs to be re-structured, as its meaning is ambiguous. “We denote the “imaginary” subgoals g ̄ and they are not real observations without including a noise variance for them.”
7. Line 423: This paragraph gives details of the dimensionality of the goal spaces across envs, it may also help the reader to know what the dimension of the original state/observation space is for each environment.

1. The authors state (line 204) that the diffusion objective aids in aligning the high-level policy’s behavior with the "optimal" relabeled subgoals, thus addressing non-stationarity. However, they provide no motivation, examples, or proof to clarify how or why the diffusion objective achieves this alignment.
2. The authors claim (line 249) that Gaussian Process (GP) priors provide uncertainty measures, regularize the diffusion process, and enhance the robustness and sample efficiency in learning. While the uncertainty aspect is clear, the other claims about regularization, robustness, and sample efficiency remain unconvincing.
3. Just as there is little clear motivation for using GP priors, the rationale for the proposed exploration strategy is also unclear—particularly why $\mu^*$ should be considered a good estimate of promising subgoals. Based on the results in Figure 3, HIDI and HIDI-A perform similarly, which suggests that including this exploration strategy may be redundant.
4. I don’t think that providing an external reward of -1 per step qualifies as sparse. It might appear sparse for the high-level policy since it achieves 0 only when it is near the goal, but it still provides useful feedback for subgoal learning via intrinsic reward. If the reward were instead set to 0 for each time step, that would make the problem genuinely sparse and challenging to solve.
5. The assertion that "increasing is better," as demonstrated in Figure 3 (middle) on lines 504-508, seems unconvincing since there are instances where $N=3$ slightly outperforms $N=5$.
6. The ablation studies were only conducted on Ant Maze environment, and I can’t find the other even in the appendix.

Minor nitpicks:

1. In Figure 3 (right) and Figure 4, all settings appear identical to me, so the statements in lines 509-520 seem redundant.
2. Consider using distinct letters for the different diffusion models. Having two instances of $\epsilon$ with different meanings is confusing.
3. The $N$ is not the right format on line 176.
4. The underscore for $\epsilon$ is small on line 185.