Goal-Conditioned On-Policy Reinforcement Learning

We address the concern about GCPO performance in more RL domains (W1) in Global Author Response, and the other concerns below.

Q1: Could you run GCPO with only a 1000 demonstrations?

We evaluate the performance of GCPO with 1000 demonstrations on the Reach, PointMaze, and VVC tasks.

Demonstrations: The source of demonstrations for PointMaze and Reach is elaborated in the Global Author Response. For GCPO, 10% $\mathcal{D}_E$ is employed, as detailed in Section 4.3.1.

Additionally, for comparison purposes, we also present the performance of GCPO on the VVC task using $\mathcal{D}_{[\chi=90]}$, which comprises only 144 demonstrations with a flight path azimuth angle of 90, as described in detail in Appendix A.3, and using $\mathcal{D}_E$. The results are presented in the following table:

Performance: It can be observed that on the relatively simple PointMaze and Reach tasks, GCPO achieved nearly 100% success rate when using 1000 demonstrations. On the more challenging VVC task, the success rate with 1000 demonstrations reached 81.12% of the success rate achieved with 10264 demonstrations, while also being significantly higher than the success rate achieved with 144 demonstrations. These results indicate that across tasks of varying complexity, GCPO can achieve good performance with the use of 1000 demonstrations.

L1: Requiring demonstrations is a relatively strong limitations, as e.g. HER does not. Is there any way around this requirement?

We answer this question from two perspectives: Firstly, while GCPO's training relies on demonstrations, the quantity and quality of these demonstrations do not need to be high. GCPO is capable of learning well-performed policies from non-expert demonstrations. Secondly, for complex tasks, methods like SAC and HER are also unable to learn from scratch; leveraging demonstrations to assist learning is a more mainstream approach. We will elaborate on these points in the following discussion.

On one hand, although GCPO relies on demonstrations, it is capable of learning well-performed policies from non-expert demonstrations. Please refer to the response to Q2 in the Global Author Response for a detailed analysis.

On the other hand, for complex tasks, methods such as SAC, HER, etc., also struggle to learn well-performed policies from scratch. For instance, on the VVC task, the performance of policies obtained using SAC+HER+MEGA is shown in the table below, where it can be observed that this method barely learn any capability to complete tasks. In other complex tasks, such as StarCraft [1], Minecraft [2], ObjectGoal Navigation [3], and others, researchers widely rely on demonstrations to train RL policies. Therefore, for the complex tasks mentioned above, we have not yet found a RL method that can bypass the pre-train phase. Hierarchical RL may be a direction worth exploring [4].

In summary, on complex tasks, we have not yet found an RL method that can bypass the use of demonstrations. Although the training of GCPO depends on demonstrations, its capability to learn policies from imperfect demonstrations somewhat relaxes the conditions for using GCPO.

[1] Vinyals O, Babuschkin I, Czarnecki W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. nature, 2019, 575(7782): 350-354.

[2] Baker B, Akkaya I, Zhokov P, et al. Video pretraining (vpt): Learning to act by watching unlabeled online videos[J]. Advances in Neural Information Processing Systems, 2022, 35: 24639-24654.

[3] Ramrakhya R, Batra D, Wijmans E, et al. Pirlnav: Pretraining with imitation and rl finetuning for objectnav[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17896-17906.

[4] Pope A P, Ide J S, Mićović D, et al. Hierarchical reinforcement learning for air-to-air combat[C]//2021 international conference on unmanned aircraft systems (ICUAS). IEEE, 2021: 275-284.

W2: Relatedly, having results for e.g. SAC + HER when doing the same thing as in appendix A.4 would be helpful.

We conduct experiments related to SAC+HER with reference to the experimental setup for GCPO described in Appendix A.4. The training process curves are shown in Fig.2 in the Global Author Response PDF. It can be observed that expanding the input of the SAC+HER policy also results in a certain degree of performance degradation. We believe that the reasons are consistent with the analysis provided in Appendix A.4.

Response to Q3 and Q4

We consolidate your two questions into three sub-questions.

1. The effectiveness of GCPO seems to primarily come from the pre-training, and the self-curriculum does not appear to significantly cover more difficult goals.

The objective of GCRL is to achieve more goals under the expectation of $p_{dg}$, which is independent of any form of goal difficulty. Both MEGA and our GCPO do not alter the objective of GCRL, so the appropriate evaluation metric should be the success rate over $p_{dg}$. Therefore, considering the change in success rate, both pre-training and self-curriculum play significant roles. Specifically:

• In GCRL, difficulty describes how easy or hard it is for a policy to achieve a goal while learning. This difficulty is closely linked to the policy's abilities, and we prefer to term it learning-process-oriented difficulty. Conversely, task-oriented difficulty pertains to the inherent challenge of a goal, unrelated to the policy's form or learning process. It is dictated by the goal itself. A goal that is task-oriented easy might be learning-process-oriented hard. There is no inherent link between these two difficulty types.

• GCRL aims to maximize expected cumulative rewards under the goal distribution $p_{dg}$. When rewards are binary indicators of goal achievement, GCRL's objective is essentially to achieve more goals under $p_{dg}$. Hence, GCRL's objective is inherently independent of any goal difficulty. MEGA, while identifying goals by learning-process-defined difficulty to optimize exploration and learning, does not alter GCRL's objective. Thus, MEGA is not centered on achieving more task-defined difficult goals.

• In VVC, we define $p_{dg}$ as the uniform distribution over the entire goal space. For clarity, we define task-oriented difficulty in Appendix A.1.4 and use it as the x-axis in our figures. Fig.1(a) in the Global Author Response PDF shows all discrete goals in a 3D space. Fig.1(b) depicts the goal distribution over task-oriented difficulty. It is evident that, under our definition, the distribution of desired goals over task-oriented difficulty is not uniform but rather low at both ends and high in the middle. This is exactly why our experimental results exhibits this particular shape.

In essence, the valid metric for evaluating GCRL is the success rate under $p_{dg}$. Table 2 shows that the success rate of GCPO is 2.69 times that of BC, indicating that online learning with self-curriculum can significantly improve a pre-trained policy. Fig.3(b) and 3(c) are merely analyses on learned policies and learning process from the task-oriented difficulty perspective. We find that self-curriculum does gradually sample task-oriented difficult goals during training, and the resulting policies can achieve some of these goals.

2. Does the requirement for sampling demonstrations in Section 4.3.2 limit the applicability of the method?

In Section 4.3.2, the experiment examines how demonstration goal distribution affects GCPO. Results indicate that demonstrations sampled according to $p_{dg}$ yield effective GCPO policies. We believe that this is a point to focus on for enhancing GCPO's performance, rather than a limitation of its applicability. Comparison between the results in Sections 4.3.2 and 4.2 reveals that GCPO can still train when the demonstration goal distribution differs substantially from $p_{dg}$. For instance, using $\mathcal{D}_2$ in Section 4.3.2, with 100 demonstrations (cover only 0.20% goal space) focused on difficulties within (0.2328, 0.2336), GCPO can still train a policy with a 21.86% success rate.

3. What impact does KL have on training?

Indeed, KL impacts GCPO's training. As discussed in our response to the first sub-question, we believe that this impact is only reflected in the success rate, not in the achievement of the task-oriented difficult goals. We train GCPO with various $\lambda$:

As shown, a higher $\lambda$ leads to a lower KL, suggesting the GCPO policy is more statistically similar to the pre-trained policy, aligning its success rate with the pre-trained policy's 17.08%. As $\lambda$ decreases, KL increases, and the GCPO policy's success rate improves. Yet, at $\lambda < 10^{-3}$, the success rate drops, which we attribute to the weak KL constraint leading to catastrophic forgetting of the pre-trained knowledge, impeding effective policy learning.

Response to Q1

The desired goal distribution is a part of the problem formulation of GCRL. We contend that this distribution, which should be a known condition, does not restrict GCPO. Even if the demonstration goal distribution significantly differs from the desired goal distribution, GCPO can still train a reasonably good policy. For a detailed analysis, please refer to our response to the second sub-problem in the first question.

Response to Q2

GMM estimates a distribution with a weighted sum of multiple Gaussian kernels, $\sum_{i=1}^M \pi_i N(x|\mu_i,\Sigma_i)$, and estimate parameters with the Expectation-Maximization algorithm. Due to space constraints, we will include detailed calculation in Appendix.

Response to W1

Thank you for the suggestion. We've thoroughly read the papers, with four focusing on curriculum learning and one on NMR. The key contribution of GCPO is its on-policy GCRL training framework, where self-curriculum is vital for efficient online learning. Thus, other curriculum learning methods could replace MEGA within GCPO. We will discuss these works in the related work section. The episode-based RL research offers valuable guidance for enhancing GCPO's suitability for NMR problems. We will explore this possibility in the future work.

Response to L1

Please refer to Q1 in Global Author Response.

Q1 & W1: How were the baselines and your method's hyperparameters tuned?

We employ Grid Search to search the following hyperparameters of the evaluated algorithms:

The other parameters are set to StableBaselines3 defaults. Taking the network architecture of BC as an example, once the optimal set of parameters has been searched, we maintain all other parameters constant and demonstrate the policy performance when using different network architectures with the following table:

As [128, 128] achieved the best performance, we maintain this architecture for all experiments on BC.

W2: the lack of ablations on the quality of demonstrations is a major limitation. How does GCPO perform on non-expert demonstrations?

Firstly, we supplement experiments on demonstration quality.

In Section 4.2, $\mathcal{D}_E$ covers 10264 goals. We generate trajectories for these goals with all the policies trained in our experiments. For a specific goal, we retain only the shortest trajectory. These generated demonstrations are denoted as $\mathcal{D}'$.

We use two metrics to measure demonstration quality: Trajectory length $I_l$. Since we employ (-1, 0) sparse rewards, this implies that shorter trajectories yield a higher cumulative reward. Control smoothness $I_s$. In control problems, minimal control gains is expected to reduce wear on actuators. Hence, we refer to [1] to define the control smoothness. Trajectory length and control smoothness each describe certain characteristics of demonstrations from the distinct perspectives of reinforcement learning optimization and optimal control, respectively.

It can be observed that: From an RL perspective, $\mathcal{D}'$ is of higher quality because the trajectories are shorter, leading to a higher expected cumulative reward. From a control perspective, $\mathcal{D}_E$ is better because the trajectories are smoother.

The performance of the BC policy $\pi_{BC}$ and the GCPO policy $\pi_{GCPO}$ trained on these two sets of demonstrations is:

The results reveal that $\pi_{BC}$ closely aligns with the demonstrations on both quality metrics, indicating that demonstration quality has a direct impact on BC. Additionally, the BC policy trained on $\mathcal{D}'$ has a slightly higher success rate, which we speculate is due to $\mathcal{D}'$ being more suitable for RL (the network architecture and training hyperparameters used to generate $\mathcal{D}'$ are the same as those for the BC policy). However, after the self-curriculum learning, the GCPO policy corresponding to $\mathcal{D}_E$ performs better and exhibits a shorter trajectory length. This suggests that the influence of demonstration quality on GCPO's online learning may not be as direct as pre-training, and further research is required to understand this relationship.

In summary, on one hand, it is challenging to define demonstration quality suitable for RL through a few metrics [2]. This is a research direction that deserves further exploration. On the other hand, demonstration quality does affect GCPO pre-training. How demonstration quality potentially influences the online self-curriculum learning of GCPO remains an intriguing question for further exploration.

Secondly, GCPO is capable of training well-performed policies from non-expert demonstrations. Please refer to the response to Q2 in the Global Author Response for a detailed analysis.

[1] Mysore S, Mabsout B, Mancuso R, et al. Regularizing action policies for smooth control with reinforcement learning[C]//International Conference on Robotics and Automation. 2021.

[2] Belkhale S, Cui Y, Sadigh D. Data quality in imitation learning[C]//Advances in Neural Information Processing Systems. 2024.

Dear Reviewer U5WJ, Do you have any further concerns? Please let us know, we will try our best to address them quickly. We sincerely anticipate your response.