Goal-conditioned Reinforcement Learning with Subgoals Generated from Relabeling

Thank you for the detailed review and for the suggestions for improving the work. Below, I have carefully addressed your comments and concerns.

1. Section 6.1: It lacks comparison with suitable baselines (see Question 5).

Thank you for raising this issue. We will address it in Responses 5.

2. Section 6.2: This section appears redundant, as GCQS's advantage in sample efficiency is already demonstrated in Figure 5 and Section 6.1.

Thank you for your suggestion. We have moved Section 6.2 to the appendix and revised it to include a comparison with the state-of-the-art algorithm, PIG [1].

3. The definition of Q-BC should be clarified upon its first appearance (line 80).

Thanks for the suggestions here. We have done this.

4. There is a typo in line 312-313 - 'policy objective that reaching achieved goals'.

Thanks for the suggestion – we have fixed this.

5. Section 2 discusses GCWSL methods like GoFar (Ma et al., 2022) and WGCSL (Yang et al., 2022) without mentioning their offline nature. Yang et al. prove the theoretical guarantees in the offline goal-conditioned setting—do these guarantees still hold in the online setting?

Firstly, the WGCSL [2] paper indicates (i.e., in line 14 of the abstract and the results in Figure 15) that WGCSL is applicable to both online and offline settings. Secondly, [3] indicates that WGSL and GoFar belong to the family of Advantage-weighted Regression (AWR). Finally, the work in [4] also highlights the relationship between DWSL and AWR. Given that AWR [5] is effective in online settings, we believe these baselines are also applicable in the online setting. Additionally, ActionModels [6] combines goal-chaining with conservative Q-Learning, which, as noted, is not suitable for the offline setting. Consequently, we have removed this baseline in the revised version. In the revised version, we have included a comparison with the subgoal-based state-of-the-art method, PIG [1]. The results indicate that our GCQS method can achieve performance comparable to the state-of-the-art method PIG [1], even without the additional algorithmic design for planning or subgoal discovery.

6. In Section 5.2, the relabelled (achieved) goal $g'$ is redefined as subgoals $s_g$, but both $g'$ and $s_g$ are used in the text, which causes some confusion. Would it be clearer to consistently use the notation $g'$ throughout the paper?

Thank you for your suggestions regarding the expression in the paper. $g'$ represents the achieved goals, which are relabeled targets, while $s_g$ denotes subgoals sampled from $g'$. Although they may share the same physical meaning in practice, their conceptual representations differ. Specifically, $g'$ refers to relabeled data, whereas $s_g$ can be understood as goals sampled from the relabeled data, serving as essential waypoints toward reaching the desired goal. The distinction made in Section 5.2 emphasizes this difference to provide clarity.

7. The comparison in Section 6.1 appears unfair, as some baselines, like Actionable Models, WGCSL, and GoFar, are designed for offline goal-conditioned RL and may not perform as well in the online setting.

Thank you for raising this issue. We have addressed it in Response 5. Additionally, we have included a comparative experiment with online-related subgoal sota algorithm in Section 6.2.

[1] Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." arXiv preprint arXiv:2303.11166 (2023).

[2] Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR.

[3] Yang, Wenyan, et al. "Swapped goal-conditioned offline reinforcement learning." arXiv preprint arXiv:2302.08865 (2023).

[4] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. "Distance weighted supervised learning for offline interaction data." International Conference on Machine Learning. PMLR, 2023.

[5] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv preprint arXiv:1910.00177 (2019).

[6] Chebotar, Yevgen, et al. "Actionable models: Unsupervised offline reinforcement learning of robotic skills." arXiv preprint arXiv:2104.07749 (2021).

Thank you for your valuable questions and suggestions. In response, we have incorporated several other state-of-the-art subgoal-based baseline algorithms, such as BEAG [1], DHRL [2], and HIGL [3], into the complex long-horizon Antmaze tasks. In Section 6.2 of the revised version [4], results in long-horizon Antmaze indicate that GCQS demonstrates performance that is slightly inferior to or comparable with PIG, while outperforming both HIGL and DHRL. Additionally, GCQS achieves twice the training speed of PIG and three times that of BEAG, with significantly reduced memory costs.

Additionally, we have provided a visual demonstration for the claims made in lines 106-107. In Figure 11 of the revised version, we highlight the advantages of the phasic policy structure in the Gridworld environment. Under noisy Q-values, traditional flat policies do not always yield correct actions and may even generate erroneous ones, particularly in states far from the desired goal. In contrast, our GCQS approach consistently produces correct policies that guide the agent towards the desired goals.

[1]Yoon, Youngsik, et al. "Breadth-First Exploration on Adaptive Grid for Reinforcement Learning." ICML 2024.

[2]Lee, Seungjae, et al. "DHRL: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning." NIPS 2022.

[3]Kim, Junsu, Younggyo Seo, and Jinwoo Shin. "Landmark-guided subgoal generation in hierarchical reinforcement learning." NIPS 2021.

[4] https://pdfhost.io/v/~QVHMgEkR_ICLR2025_GCQS_Rebuttal.

[5]Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." ICLR 2023.

We thank the reviewer for the detailed review. Below, we address the raised concerns and questions.

1.on the high level, the necessity of the proposed method is not clear. I understand the key insight the authors aimed to convey using Figure 2. However, from my perspective, isn't this a property rather than a problem of hindsight relabeling? e.g. when the number of steps required to achieve different goals is uniformly sampled from 1-50, then during the learning process, hindsight relabeling using trajectories having a length of 50 will generate a uniformly distributed length from 1-50, and as the learning proceeds, the policy will learn to achieve closer goals, and averaged trajectory length will be smaller than 50 --- this will lead to a compounding effect in increasingly having more shorter paths.

Existing HER-based methods tend to prioritize shorter segments of trajectories during the update process, meaning that updates often utilize short-length goals from the same trajectory, while longer segments remain unused and this valuable information is completely overlooked. Our perspective emphasizes leveraging longer trajectory information within the same hindsight framework. To achieve this, we have developed a novel algorithm called GCQS, which can be viewed as an extended version of GCAC. Specifically, GCQS leverages the achieved goals trajectory as the subgoal distribution. As illustrated in Figure 6, we analyzed the trajectory information utilized by GCQS during the update process. The results indicate that GCQS effectively incorporates trajectory data of varying lengths, showcasing strong performance across different scenarios.

2. to solve the above challenge, why is the proposed method "necessary"? For instance, why can not one just to re-sample or down-sample on some of the state-goals? In the current write-up, it is not very clear why the multiple designing factors introduced are well-motivated or necessary.

Re-sampling or down-sampling does not necessarily enable the use of longer trajectories for updates. In contrast, our GCQS framework is designed to leverage longer trajectories during the update process effectively.

3. according to the theory, using a smaller $\eta$ will lead to a tighter bound, does this mean that we need the \eta to be as close to 0 as possible?.

In Equation (15), the KL divergence approaches zero only when $\beta$ approaches positive infinity. Otherwise, it is unlikely to converge to zero. In our work, $\beta$ is set to 0.2, which means the KL divergence does not approach zero, and consequently, $\eta$ is also not close to zero. As demonstrated in the ablation study (Figure 12), increasing $\beta$ does not necessarily lead to better performance, indicating that setting $\beta$ to positive infinity would be suboptimal.

4. In Figure 5, the averaging/smoothing method for different figures seems different, and some results lack error bars. The FetchSlide performance is noticeably lower than what has been achieved in the literature. From my perspective, it is not necessary to demonstrate a method is always better than baselines in all experimental cases. What is more important is to clearly show what are the realistic motivations for introducing the techniques/designing components of the paper, and then use experiments to highlight --- the settings where those challenges exist, the method shines and on the other hand, when those challenges (controllably) alleviated, the performances of different methods would converge.

Thank you for your suggestion. In the revised version of the paper, we have included the error bar plots for Figure 5 in Section D.4 of the Appendix.

Our implementation is based on the official code from [1], and similarly to [1], the FetchSlide task exhibits a relatively low mean success rate, which may be due to suboptimal hyperparameter tuning in this environment.

5. On presentation. In Figure 1, it would be great if the authors could give some concrete examples and make sure the notations are self-consistent. I acknowledge the authors' effort in explaining their method using Figure 1 and it could be a great idea if the clarity can be further enhanced. For instance, what is "QBC" objective in this figure, what does it mean by "subgoals come from achieved goals", and why the policy can take either s and g or s and s_g as its inputs?

Thank you for your suggestion. In the revised version, we have updated the phrasing for clarity. Additionally, the Q-BC objective is explained in Section 5.1. The phrase "subgoals come from achieved goals" means that subgoals are sampled from achieved goals (i.e., relabeled goals). Specifically, (s,g) serve as the inputs to the policy $\pi$, while (s,s_g) are the inputs to the prior policy $\pi^{piror}$.

[1] Liu, Bo, et al. "Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.

We appreciate the reviewer’s comments and feedback.

For Q1 and Q2:

As discussed in Section 4, we concluded that baseline algorithms often rely on short trajectories for updates (due to uniform sampling of achieved goals). Therefore, addressing this issue could potentially lead to better performance. Following the approach suggested by reviewer JyB4, we implemented simple weighted sampling, or upsampling (where longer trajectories receive higher weights), which favors longer trajectories. The results are presented in Appendix D.1 of the revised version [1]. Our findings show that while simple weighted sampling (or upsampling) does improve the performance of the baseline algorithms, it still does not outperform GCQS. This highlights the importance and necessity of the subgoal mechanism in GCQS, which enables the learning of more optimal actions.

For Q3:

No, in Equation 15, when the KL term equals 0, it leads to a trivial solution, meaning that the policy no longer updates. Instead, the policy should be updated under a smaller value of \eta, which allows for gradual policy updates.

[1] https://pdfhost.io/v/~QVHMgEkR_ICLR2025_GCQS_Rebuttal

We sincerely thank the reviewer for their thorough review and valuable feedback on our work.

1.The lack of comparison with other benchmark problems, such as complex AntMaze tasks presented in [1, 2], where the original HER showed significantly degraded performance compared to similar methods such as PIG, raises questions about whether omitting high-level planning for subgoal generation is valid for other complex and long-horizon GCRL tasks.

Thank you for your suggestion. In the revised version, we have included a comparison with the subgoal-based state-of-the-art method, PIG [1], to enhance the analysis.

2. This is the question for clarification. What is the key difference between the proposed methods compared to existing methods [1,2] which update subgoals throughout training? Specifically, how does relabelling achieved goals as subgoals differ from landmark or subgoal updates in PIG and HIGL?

Our GCQS method does not involve designing additional strategies for planning or discovering subgoals. Instead, it directly selects subgoals from the relabeled achieved goals, aiming to leverage more information from the same trajectory (particularly in long trajectories).

The key difference between the proposed GCQS method and existing methods such as PIG [1] and HIGL [2] lies in the approach and mechanism used for subgoal selection and integration:

(1). Subgoal Discovery and Integration:

GCQS: Utilizes achieved goals directly from the agent's own trajectories as subgoals, leveraging hindsight relabeling. The method reuses these achieved goals within the same trajectory to enhance policy learning. This approach does not involve any additional discovery or planning mechanisms for subgoal identification. Instead, GCQS systematically redefines naturally occurring achieved goals from replayed trajectories, thus simplifying the integration of subgoals into the training process.

PIG and HIGL: Both incorporate explicit subgoal discovery and planning algorithms.

PIG: Uses a graph-based planning algorithm to identify subgoals and updates these subgoals dynamically throughout training. This method involves complex planning to guide the agent's behavior by imitating subgoal-conditioned policies and includes a strategy for subgoal skipping to optimize paths.

HIGL: Focuses on identifying landmarks that serve as promising exploration targets. It selects subgoals based on criteria like novelty and coverage to promote more efficient exploration and policy updates. The selected subgoals or landmarks are used in a hierarchical structure that requires additional computation for their discovery and validation.

(2). Relabeling vs. Dynamic Updates:

GCQS's Relabeling: In GCQS, the subgoals are achieved goals that are relabeled post-experience. The relabeling mechanism converts observed outcomes into intermediate objectives without modifying or dynamically updating them based on external algorithms. This relabeling is derived directly from the agent's own trajectory, making it straightforward and computationally less intensive.

PIG and HIGL's Dynamic Updates: These methods involve active updates and the strategic placement of subgoals or landmarks during training. PIG dynamically plans and modifies subgoals during policy updates, whereas HIGL uses a novelty-based mechanism to adjust the subgoal selection process. These processes aim to guide exploration and learning more explicitly and involve more complex planning steps.

(3). Complexity and Computational Cost:

GCQS maintains simplicity by leveraging the natural data produced by the agent's experience for subgoal selection, avoiding additional planning or search mechanisms.

PIG and HIGL incur a higher computational cost due to their need for subgoal discovery mechanisms, such as graph-based planners (PIG) or coverage/novelty-based sampling (HIGL).

In summary, GCQS differentiates itself by using achieved goals as subgoals directly through hindsight relabeling, which simplifies the training process and reduces computational overhead. In contrast, PIG and HIGL involve more sophisticated subgoal or landmark discovery and updating mechanisms that require additional planning and computational resources to guide training effectively.

3. This method discards the necessity of high-level planning for subgoal generation but I'm curious whether GCQS works well in long-horizon tasks such as AntMaze compared to methods utilizing high-level planning, such as HIGL and PIG.

Thank you for your suggestion. In the revised version, we have included a comparison with the subgoal-based state-of-the-art method, PIG [1]. The results indicate that our GCQS method can achieve performance comparable to the state-of-the-art method PIG [1], even without the additional algorithmic design for planning or subgoal discovery. Notably, we chose to implement PIG as it demonstrated superior performance over HIGL [2] in the Antmaze environment, as reported in the original PIG study.

4. It is unclear how GCQS addresses the issue of short trajectory updates since HER already conducts a similar relabeling scheme. Which component of GCQS handles this?

Existing HER-based methods tend to prioritize shorter segments of trajectories during the update process, meaning that updates often utilize short-length goals from the same trajectory, while longer segments remain unused and this valuable information is completely overlooked. Our perspective emphasizes leveraging longer trajectory information within the same hindsight framework. To achieve this, we have developed a novel algorithm called GCQS, which can be viewed as an extended version of GCAC. Specifically, GCQS leverages the achieved goals trajectory as the subgoal distribution.. As illustrated in Figure 6, we analyzed the trajectory information utilized by GCQS during the update process. The results indicate that GCQS effectively incorporates trajectory data of varying lengths, showcasing strong performance across different scenarios.

5. How important is the SAC component in Eq.12 for GCQS? This omitted ablation leaves the question of the importance of the BC term.

In the ablation study presented in Section 6.3, we examined the influence of Behavior Cloning (BC) on learning to reach achieved goals. This variant, referred to as "No BC-Regularized Q," is depicted by the green curve in Figure 8. The results demonstrate that removing the BC component negatively impacts the performance of GCQS, indicating that incorporating the BC term enhances overall performance.

6. Presenting more details on the derivation of Eq. 25 from Eq. 24 would be helpful for readers.

Thanks for the suggestion – we have fixed this.

[1] Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." arXiv preprint arXiv:2303.11166 (2023).

[2] Kim, Junsu, Younggyo Seo, and Jinwoo Shin. "Landmark-guided subgoal generation in hierarchical reinforcement learning." Advances in neural information processing systems 34 (2021): 28336-28349.

We thank the reviewer for the positive feedback!

1. If prioritizing the closer achieved goals is the main issue of previous GCRL methods, can we simply use data augmentation to rebalance the relabeling trajectory dataset and improve the performance? I think this would be an interesting result to see and could make the paper more solid.

Thank you very much for presenting this interesting idea. However, I am not entirely clear on how data augmentation would be specifically implemented in this context. Could you please provide a more detailed explanation, or suggest relevant literature for further study? I look forward to your feedback.

2. The connection between GCQS and GCWSL: while the authors do not classify GCQS as one GCWSL method, I think there are some similarities to discuss between the two methods. Actually, in offline settings, both Equation 12 and Equation 15 should have a closed-form solution, which is an exponentially weighted form of $\pi_relabel$ and $\pi^{piror}$, as discussed in [1]. This makes me wonder if GCQS is still a GCWSL method (like exponential advantage weight mentioned in WGCSL [2]) and if the closed-form solution could further improve the method.

In the offline setting, the closed-form solutions for Equation (12) and Equation (15) align with GCWSL. However, in the online setting, we directly optimize the Q-function and the behavior cloning (BC) objective, positioning GCQS as an extension of GCAC.

3. Several baselines in the experiments are offline methods, so I am wondering how authors compare them with other online baselines and GCQS to make sure the comparison is fair.

Firstly, the WGCSL [1] paper indicates (i.e., in line 14 of the abstract and the results in Figure 15) that WGCSL is applicable to both online and offline settings. Secondly, [2] indicates that WGSL and GoFar belong to the family of Advantage-weighted Regression (AWR). Finally, the work in [3] also highlights the relationship between DWSL and AWR. Given that AWR [4] is effective in online settings, we believe these baselines are also applicable in the online setting. Additionally, ActionModels [5] combines goal-chaining with conservative Q-Learning, which, as noted, is not suitable for the offline setting. Consequently, we have removed this baseline in the revised version. In the revised version, we have included a comparison with the subgoal-based state-of-the-art method, PIG [6]. The results indicate that our GCQS method can achieve performance comparable to the state-of-the-art method PIG [6], even without the additional algorithmic design for planning or subgoal discovery.

4. The cumulative function should be defined in Theorem 4.1, as it is slightly different from the commonly used probability cumulative function.

Thanks for the suggestions here. This has been addressed in the revised version of the paper.

5. I believe Section 6.1 has a typo: 16 CPUs -> 16 GPUs.

Thanks for the suggestion – we have fixed this.

[1] Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR.

[2] Yang, Wenyan, et al. "Swapped goal-conditioned offline reinforcement learning." arXiv preprint arXiv:2302.08865 (2023).

[3] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. "Distance weighted supervised learning for offline interaction data." International Conference on Machine Learning. PMLR, 2023.

[4] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv preprint arXiv:1910.00177 (2019).

[5] Chebotar, Yevgen, et al. "Actionable models: Unsupervised offline reinforcement learning of robotic skills." arXiv preprint arXiv:2104.07749 (2021).

[6] Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." arXiv preprint arXiv:2303.11166 (2023).

We appreciate your simple yet insightful suggestion. Following your approach, we implemented weighted sampling, which favors longer trajectories. The results are presented in Appendix D.1 of the revised version [1]. Our findings show that while simple weighted sampling (or upsampling) does improve the performance of the baseline algorithms, it still does not outperform GCQS. This underscores the significance of the subgoal mechanism in GCQS, which enables the learning of more optimal actions.

[1] https://pdfhost.io/v/~QVHMgEkR_ICLR2025_GCQS_Rebuttal