Goal-Conditioned Reinforcement Learning with Virtual Experiences

We thank you for your reviews and address your concerns as follows:

Q1: I do not fully understand how subgoals are selected based on task probability density $e$ for curriculum learning.

A1: The method for training task probability density model described in Appendix A. And It can be viewed in conjunction with Fig. 2 (Left). After training, we randomly sample different states (e.g. 5 in Fig.2) from the replay buffer as virtual goals $g_i$. For the agent's current state $s$, we calculate $e_i(s, g_i)$ and filter appropriate virtual goals through "0.8$\bar e$<$e_i(s,g_i)$<1.2$\bar e$" (line 233), with the expectation that it is located on the boundary.

Q2: I am confused why introducing only a single subgoal (i.e. k=1) in AntMaze yields the best results.

A2: This is a good question. We believe that this conclusion is closely related to our self-supervised learning approach. Since an effective high-level policy improvement relies on sampling appropriate subgoals, when the number of subgoals increases (greater than 1), it becomes exceptionally difficult to obtain subgoals that fall precisely on the optimal path through random sampling.

Q3: Can the entire framework be adapted to integrate different lower-level off-policy algorithms, such as HER?

A3: Our goal relabeling method is indeed an extension of HER, with the key aspect being the introduction of virtual experiences constructed through virtual goals. Observing the results in Fig. 7 (left), when only the HER method is used ("a:v=1.0:0"), VE's performance suffers significantly, performing similarly to "SAC+HER," which indicates that our imitation learning does not fulfill its intended role.

Finally, we want to indicate that the noise introduced by virtual experiences mentioned in the paper is actually policy divergence. Specifically, there is a divergence between the old policy $\pi(a^i_t|s^i_t,g^i)$ stored in the replay buffer and the new policy $\pi(a^i_t|s^i_t,g^j)$ after relabeling. The divergence caused by HER's relabeling method is not significant, thanks to its reliance on real trajectories. However, virtual goals lead to more severe divergence. Therefore, what VE imitates are the subgoal-conditioned policies $\pi^{prior}(s,s_g)$, rather than the reconstructed original actions $a^i_t$ (which include noise in Eq. (5)) in the virtual experiences.

We thank you for your recognition of this paper.

We thank you for your reviews and address your concerns as follows:

Q1: What is j, and why is it that if i=j, Eq. 5 is equivalent to Eq. 4? What is the "task probability density" trying to learn? What does the condition (e.g., the range filter) mean from a high-level perspective? Why would there be a "lack of connection between state transitions", as described in line 238? It doesn't seem like state transitions are being tinkered with in Eq. 5.

A1: We apologize for any confusion caused by our previous statement. We hope this explanation clarifies the issue. In Eq. (5), we employ a more general relabeled goal notation (bold) $g$, to signify that our defined generalized goal re-labeling can incorporate actual, virtual, or custom-made goals (Eqs. (4) and (5) mix $s$ and $g$, as we assume $S=G$). Superscripts $i$ and $j$ are used to distinguish whether the relabeled goal comes from the same trajectory as $s^i_t$. Since HER only considers goals from the same trajectory, $i=j$. However, we also use virtual goals from different trajectories to construct the experience needed for learning ($i \neq j$).

As shown in Fig. 2 (Left), we distinguish which virtual goals are at the boundary by learning a task probability density model. This helps us build a step-by-step curriculum (see Fig. 1(b)). The task probability density model does not participate in the optimization process of the high-level policy; it is a relatively independent module used solely for constructing training data.

The term mentioned on line 238 "lack of connection between state transitions" refers to the fact that in GCRL, the policy $\pi(s,g)$ simultaneously considers the current state $s$ and the goal $g$. However, in Eq. (5), both the goal and the reward have been changed (we replace them with actual and virtual goals), weakening (or disrupting) the connection between the new (bold) $g$ and $(s^i_t, a^i_t, s^i_{t+1})$.

Q2: How was VE tuned for the experiments in §5.2? It seems like VE was tuned per each environment separately, but how was this done without tuning on the "test" problem (the goal conditioned environment that you evaluate on)?

A2: The proposed VE contains several core hyperparameters, which we have evaluated in detail and kept consistent during training and testing. In Appendix Fig. 9, we assess the impact of the $p$-norm and the number of subgoals $k$ on the results, and we use $p=\infty$ and $k=1$ in all experiments. For the screening by task probability models, we have set the range to (0.8, 1.2). The proportion of actual goals to virtual goals is evaluated in Fig. 7 (left), and we consistently use "a:v=0.5:0.5" in all experiments. In summary, the hyperparameters used by VE are generally consistent across all environments, and there is no difference between training and testing. Notably, curriculum learning and subgoal planning are active only during training, whereas testing relies solely on trained goal-conditioned policiy interacting with the environment.

Q3: What is the main difference between this work and other works that employ sub-goals? Is it that other works attain sub-goals without learning (e.g. the midpoint, bisection or graph-based methods)?

A3: "Midpoint":[1] This study focuses on offline RL, which differs from the online RL discussed in this paper. It utilizes Transformer to predict intermediate subgoals. "Bisection":[2] Connects subgoal planning with the dynamic programming equation for the all pairs shortest path (APSP) problem, which naturally advantages navigation and obstacle avoidance tasks but is challenging to extend to high-dimensional goal spaces (VE uses the complete state space as the goal space). "Graph-based":[3,4] is the main comparative method in this paper. We believe that a graph structure is an effective form of preserving historical knowledge, but it suffers from low training efficiency and suboptimality. Therefore, we provide knowledge from different tasks by constructing virtual experiences and learn from them via VE. The biggest difference between VE and the methods mentioned above is its use of self-supervised learning for subgoal planning, and the knowledge from historical experiences can be seamlessly integrated into general reinforcement learning approaches via a straightforward self-imitation process. In our understanding, the above methods all rely on learning to master how to plan subgoals.

Dear Reviewer SKRR

Thanks for your time and comments on our work!

We have tried our best to address the concerns and provided detailed responses to all your comments and questions.

Would you mind checking our response and confirming whether you have any further questions?

Best regards, Authors of #5840

We thank you for your response and constructive comments.

Could the authors also clarify the concept of the range filter?

Sorry for the misunderstanding. We want to clarify that "range" and "filter" should be understood separately instead of being combined. The "filter" means that we apply the "task probability density" $e$ in curriculum planning, acting as a "filter" to screen suitable virtual goals $g$ from random sampling. While the "range" means a simple "range" constraint (0.8$\bar e$<$e(s,g)$<1.2$\bar e$) for the filtering method that is adopted in the proposed method.

State transitions are the change in the MDP state, i.e., a probability distribution over the next state defined by a transition kernel $T(s_{t+1}|s_t,a_t)$. If you are only changing the goals for states $s_t$ and $s_{t+1}$, you are not disrupting state transitions. While these "synthetic" states with virtual goals are not real, they do conform to the MDP dynamics; the goal, at least in most goal-oriented MDPs I know, does not affect the transition kernel. The goal affects the policy. Perhaps I am missing a key detail here?

We agree with your understanding of state transitions in the MDP, even under goal-conditioned. We want to clarify that “lack of connection” discusses the problem between the original state transition and the relabeled goal, and it does not represent that whether the state transition itself has been disrupted. So we modify as: "Firstly, from a policy perspective, there is a lack of connection between the original state transitions $(s^i_t, a^i_t, s^i_{t+1})$ and the relabeled goals $g$. This reduces the accuracy of policy evaluation."

The action $a^i_t$ in the original experience is a decision made by the policy $\pi$ under the conditions of the state $s^i_t$ and the original goal $g^i$, so the state transition caused by the action is strongly correlated with the original goal, while the relabeled goal $g$ in the reconstructed data has lost its relevance to the original state transition $(s^i_t, a^i_t, s^i_{t+1})$ and cannot reflect a correct policy, thus reducing the accuracy of policy evaluation.

When I asked about hyperparameter tuning, I was referring to Table 1 in the Appendix. The learning rates of the networks (subgoal prediction, policy, critic) are different across 3 environments, and are not consistent. How were the hyperparameters in Table 1 tuned? I could not find hyperparameters for baselines. The same question can be posed for your experiments with these baselines. Did you tune the baselines, and if so, how?

Thanks for your comments. Our comparisons are fair. We divide the experimental environment into two classes: state-based "Reacher" and "AntMaze" and pixel-based "Sawyer". In "Reacher" and "AntMaze," we search the learning rate (subgoal prediction, policy, critic) from the candidates {0.0001, 0.001, 0.01} for VE with a $3\times 3 \times 3$ grid. We use the same parameters in these environments. However, in "Sawyer," we discover that appropriately increasing the learning rate is a better choice. For the baselines (RIS, PIG, HIGL), they use the same experimental environment as ours. So we keep the parameters in the original paper (as the authors have tuned). For the others (GCSL, SAC+HER), we adjust them using a parameter search method similar to VE.

Doesn't VE also rely on learning to master how to plan subgoals (via the high-level subgoal policy)?

I do agree that the main point of difference is the use of self-supervised learning/reinforcement learning. Discussing this (and above points) is necessary in the paper, somewhere in either the introduction section or §4.1 where you first talk about subgoals in VE.

I would suggest rewriting line 52/61, and instead of contrasting VE to HER, explain prior subgoal-based HER approaches and contrast VE to them. Then it becomes clearer where your proposition lies compared to prior work.

Thanks for your good comment. The methods introduced in the paper can be classified into into learning-based and search-based methods. Learning-based methods generally require only a single forward pass through the network for subgoal planning, making them more efficient and user-friendly than graph search-based methods. In this paper, both VE and RIS employ a learning-based approach, whereas PIG and HIGL utilize graph search techniques. Other reviewed methods do not incorporate subgoal planning. Thanks for your good suggestions, we have made corresponding revisions in the paper.

I understand. But when I read the paper, the pattern in the density maps in Figure 5 were too subtle to warrant space in the main text. The state density maps do not provide information at all. I would suggest moving this figure to the appendix, and using the saved space to (1) better explain VE in §4 and (2) its relation to subgoal-based approaches.

Thanks for your good suggestion. We have revised Sec. 4 in the reversion.

We thank you for your reviews and address your concerns as follows:

Q1: The importance of the curriculum is not clear to me: for one, it introduces much additional complexity (e.g. the need to train a Flow or RealNVP model, whose details are not well discussed in the paper), it provides seemingly little benefit (in Figure 7, middle), and requires special hyperparameters (e.g. (0.8, 1.2) for state density clipping).

A1: Curriculum learning has been shown in certain works to enhance the stability of the learning process and the ultimate success rate [1-3]. In this paper, we regard curriculum learning as a relatively independent module within our proposed framework, which can achieve more significant improvements through better design. Our main contribution is the verification of the advantage of the task probability density model in constructing curricula compared to the state probability density model. Although existing methods still require setting hyperparameters, more elegant automated curriculum methods can be developed in the future. As shown in Fig. 7, the interaction steps at which the agent reaches performance peak are advanced by approximately 29% (190k -> 135k) with the help of curriculum learning, and the standard deviation fluctuations displayed during the training process are smaller, making the process more robust, which also confirms the benefits commonly mentioned in previous works.

Q2: The paper only studies relatively simple state-based domains with intrinsic 2d structure, and most of the analysis is on Antmaze (which has a clear 2d structure, even if this is not the axis being measured) -- this is particularly worrisome, since the high-level policy learning problem gets more complicated in higher dimensional state spaces, and it is unclear how this method generalizes. The paper would be stronger if it had a more complex state-based environment (e.g. one may consider Franka Kitchen or CALVIN)

A2: Thank you for your valuable suggestions. We select the AntMaze environment for analysis, following established research practices [1,4,5], because it offers clear and comprehensible visualization. It is important to note that, in the navigation task, only the two dimensions (x, y) within the complete 31-dimensional state space of AntMaze are directly relevant to the goal. Our proposed subgoal planning method successfully captures this key information without artificially reducing the dimensionality of the goal space (maintaining all 31 dimensions). When extended to more complex state spaces, we believe that the core states remain the 3-dimensional coordinates (x, y, z) that describe the position of the robotic arm or object. We consider that our proposed method also adapts well to this complexity. Furthermore, we conduct experiments in the Sawyer environment, which also features a high-dimensional state space (pixel-based 84$\times$84 RGB), and it demonstrates the potential for planning subgoals based on pixel input. We acknowledge your concerns and incorporate scenarios utilizing pixel input as one of our future research directions.

Q3: Section 4.2 is difficult to follow: it introduces additional notation for the sub-goal sequence that is not explained clearly, and it is not clear whether the mechanism being used is an A2C-like (A2C updates using policy gradient estimates) or if it is AWR-like (which seems to be the form of Equation 8).

A3: In Sec. 4.2, the subscript ( k ) in Eq. (6) indicates the number of subgoals ({$s_g$}_k means a set), so a complete task (s, g) is divided into ( k+1 ) segments, which we organize into a vector. Eq. (7) calculates its $p$-norm, and Eq. (8) minimizes the optimization objective through a self-supervised learning method, where $\hat{s_g}$ represents the sampled subgoal, rather than the output of the high-level policy $\pi^h$. We appreciate your concerns regarding Eq. (8). As you mention, our construction of the self-supervised method indeed leans more towards AWR-like. The difference is that we compare the advantages of sampled subgoals over those output by high-level policies, rather than comparing the advantage of the current policy over the historical (or mixed) policy as in AWR. We revise this wording in the new version.

Dear Reviewer TXca

Thanks for your time and comments on our work!

We have tried our best to address the concerns and provided detailed responses to all your comments and questions.

Would you mind checking our response and confirming whether you have any further questions?

Best regards, Authors of #5840

We thank you for your reviews and address your concerns as follows:

Q1: Can $1<p<2$ be used in the $p$-norm for subgoals? When might $p=\infty$ be suboptimal?

A1: For the $p$-norm, we primarily focus on the commonly used cases of $p=1,2,\infty$. We did not consider the range $1<p<2$; however, computations for these cases are feasible as long as $f = x^p$ satisfies the condition of being a convex function. In our experience, $p = \infty$ demonstrates broader applicability, but it is not always optimal (in terms of optimizing efficiency). This is because it minimizes the largest term in Eq. (6) at all times. While this approach to solving subgoals is relatively stable, there is likely room for further improvement.

Q2: In Fig. 12, most of the proposed subgoals appear very close to the goals.

A2: We apologize for the misunderstanding in this instance. Fig.12 illustrates the final states achieved by the agent under the guidance of the three methods in the most difficult tasks, visualized through the $(x,y)$ coordinates.

Q3: Is there a theoretical justification for why imitating the subgoal-generating policy in eq. (9) is beneficial?

A3: To the best of our knowledge, the most relevant literature is [1], which discusses the feasibility of persistently imitating advantage-based policies. However, However, we have not encountered theoretical research that is closely related to this paper. We are happy to share some intuitive insights for your consideration.

We regard the additional imitation learning component as a form of regularization. When the agent is capable of achieving the goal, the subgoal-conditioned policy is similar to the goal-conditioned policy. In such cases, the regularization term neither introduces conflict nor imposes excessive constraints. Conversely, when the agent's success rate in achieving subgoals significantly exceeds its success rate for the goal, the subgoals effectively simplify the task. In this cases, the subgoal-conditioned policy is likely to diverge from the goal-conditioned policy, thereby correcting the policy gradient derived solely from the value function $Q(s, \pi(s,g), g)$. We believe this is the primary role played by imitation learning in Eq. (9).

Q4: Likewise, is there any mathematical statement that can be made for why this goal relabeling approach is superior to HER?

A4: The generalized goal relabeling method proposed in Sec. 4.1 is an extension of HER. Our primary idea is to incorporate a wider range of potential goals—whether they are actual goals, virtual goals, or artificially crafted expert goals—as relabeling goals. This approach enriches the learning experience for off-policy RL. We have identified a recent study [2] that explains the high sample efficiency of goal relabeling.

Q5: How exactly is the task density model used for curriculum learning? This seems to be an important component to the method (Fig. 10), but I didn't understand how it fits into, e.g., Eqs. (5) and (7).

A5: As described in Sec. 4.1, we employ a task density model to filter virtual goals (see Fig. 2(Left)). This model does not directly participate in the optimization process (e.g. Eqs. (7) and (9)) but rather selects appropriate goals (e.g. 0.8$\bar e$<$e_i(s,g_i)$<1.2$\bar e$) from randomly sampled virtual goals. This filtering process is an integral part of constructing a progressive curriculum for learning.

Q6: What is the justification for the direction of the KL divergence in Eq. (9)? What is $\pi^{prior}$?

A6: The $\pi^{prior}$ referred to Eq. (9) is a generic prior policy, and in this paper, we adopt the policy target network $\pi_{\theta_{m-1}}$ (the paper uses $k$ to denote the number of subgoals, and here we use $m$ to represent the update step) from the soft-update process. In Eq. (9), $\pi^{prior}$ serves as the imitation target, and the KL divergence encourages the current goal-conditioned policy $\pi_{\theta_m} (s,g)$ to approach the subgoal-conditioned policy $\pi^{prior}(s,s_{g_i})$. The rationale for this process is connected to the intuitive assumptions presented in A3.

Dear Reviewer i5cV

Thanks for your time and comments on our work!

We have tried our best to address the concerns and provided detailed responses to all your comments and questions.

Would you mind checking our response and confirming whether you have any further questions?

Best regards, Authors of #5840

We thank all reviewers for their valuable comments. We have revised the paper, according to their comments. All revised parts are marked as blue. The main revisions are summarized as follows:

1. We have corrected some mathematical issues pointed out by reviewers.
2. We highlight the differences between VE and existing methods in related work.
3. We rewrote Section 4.1 to clarify that task probability density is used to screen suitable virtual goals, which is part of constructing training data and is unrelated to subgoal planning.
4. We provide additional details on tuning hyperparameters in the Appendix H.