Bridging Sub-Tasks to Long-Horizon Task in Hierarchical Goal-Based Reinforcement Learning

1. The difference between the performance of the algorithm using FLR and the vanilla algorithm is not significant. Especially in AntMaze4Leg, the performance of FLR+DHRL is almost the same as that of the original DHRL.
2. The experiment was only conducted in two scenarios. I think this is a more empirical paper and should be experimented in more different domains. The current experimental results are not convincing enough.
3. I think FLR is more about improving the original algorithm capabilities, rather than really giving it the ability to solve short-sighted problems. For example, in AntMaze3Leg, as training progresses, PIG cannot effectively suppress the number of episodes where the agent falls, and neither can FLR+PIG.
4. The code is not available, which is not conducive to the reproduction of experimental results.

• The authors claim that their proposed method can work as a plug-in for existing HGRL methods. However, the experiments are conducted only on top of DHRL. Since RL algorithms often perform high-performance variance in different environments/tasks, I think it is necessary to make comparisons with more baselines. And also, I think it would be better if the authors make comparisons on different environments, not only the Maze.
• Key designs lack of explanation, e.g., how to determine the $sg$ in equation 5? Parameter searching via enumeration, or gradient-based?
• Missing introduction of environment details
• The investigation on the effectiveness of $\alpha$ is not thoroughly, as the authors only test a range of $\alpha$ when $\gamma = 0.99$. More trails should be included.
• The writing is not good enough. For instance, the supplementary should be improved, e.g., there should include some natural language description for the given algorithm tables, and it seems the authors didn't conduct repeated experiments in Figure 6; The paper mentioned FPS algorithm, but what is FPS algorithm? No reference and no introduction; Figure 3 shows the results of "initial random rollouts" settings, but there is no description for this setting.

And I suggest the authors improving the writing further, especially the details of the involved environments. As you should not suppose your readers are all familiar with the related work of goal-conditioned RL.

• Only in the simplest scenarios, the proposed method has obvious performance improvement. More experiments should be conducted in various scenarios(such as FetchPush, FetchPickAndPlace, AntMazeBottleneck, and UR3Obstacle) to highlight the superiority of the proposed method.
• As can be seen from the ablation study, the experiment is sensitive to the hyperparameter $\varepsilon$, even in a simple task, not a robust hyperparameter is provided.
• The proposed solution is only suitable for controlling the local reaching task of the robot, because when achieved goal is subsituted with the location of the object being operated, it has the same hindsight goal for all the goal relabeling using HER, and FLR will not yield results as expected.
• It seems that the subscripts of $sg_{i}$ and $k_{i}$ are not standardized.
• At the beginning, the value of $V_{\pi}$ is definitely biased, and these biased values will continue to exist in the experience buffer when updating, which will continue to degrade theperformance, and may be a reason for the unsatisfactory performance on complex tasks.
• The proposed solution is only suitable for controlling the local reaching task of the robot, because when the operated object is not changed in position，it has the same hindsight goal for all the goal relabeling using HER, and FLR will not yield results as expected.