Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization for Goal-Conditioned Reinforcement Learning

... discuss the relationship between MSR and TRPO, PPO methods in designing the MSR loss.

We denote the policy at the t-th iteration as $\pi_{\theta_t}$, with $\theta$ denoting policy parameters. $\pi_{\theta_t}(\cdot|s,g)$ maps state $s$ and goal $g$ to an action distribution.

TRPO and PPO's key principle is to limit policy updates by constraining the KL divergence between policies in consecutive training iterations, $E_{g\sim p_{dg}}D_{KL}(\pi_{\theta_{t-1}} (\cdot|\cdot,g),\pi_{\theta_t}(\cdot|\cdot,g))$. In contrast, MSR restricts the policy's action distribution variation across goals within the same iteration, $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')}D_{KL}(\pi_{\theta_t}(\cdot|\cdot,g),\pi_{\theta_t}(\cdot|\cdot,g+\epsilon))$.

In essence, TRPO, PPO, and MSR use KL divergence for policy constraints but focus on distinct optimization objectives. This clarification will be added to our paper.

Would MSR still work for other reward types?

MSR is effective for all reward types:

1. Theorem 3.4 makes no assumptions about the reward function. The return gap remains bounded, albeit with a complex expression. For clarity, we derived a simpler bound under the distance-based reward, facilitating understanding of the relationship between $|J(\pi_{g_1})-J(\pi_{g_2})|$ and $d_{KL}(\pi_{g_1}(\cdot|s),\pi_{g_2}(\cdot|s))$.
2. In our experiments, we used both distance-based rewards (in Reach and VVC) and sparse rewards (in Push). Results demonstrate MSR's effectiveness across these different reward settings.

Does MSR introduce significant training overhead? How does MSR impact convergence speed?

MSR adds training overhead without influencing inference, due to added regularization only in policy optimization. For on-policy RL, where the main cost is environment sampling, MSR's effect is negligible, with only a 2.14% increase. In off-policy RL, which leverages replay buffer data, the impact is more pronounced: a 21.14% increase for MSR-GC-SAC and 16.02% for MSR-HER. Figs. 13, 17, and 21 reveal MSR's convergence rate is on par with baselines. Overall, despite the slight added training overhead, MSR markedly improves policy continuity in achieving goals.

Note: Results come from PPO (1e7 steps), SAC (1e5 steps), and HER (1e5 steps) on VVC.

Why the optimal value of λ is quite small?

We address your concerns from three aspects:

1. MSR is designed to lower $D_{KL} (\pi_{g_1}(\cdot|s), \pi_{g_2}(\cdot|s))$ without being excessively small. While 0.001 may seem trivial, our ablation studies confirm it sufficiently meets our objectives.
2. We've included a plot showing training loss trends with $\lambda = 0.001$. MSR loss is comparable to entropy and critic losses and an order of magnitude less than actor loss, demonstrating its impact on policy optimization despite the small $\lambda$.
3. Small coefficients for auxiliary losses are standard in RL. The primary goal is to maintain the RL objective's dominance, with other losses in a supporting role. For instance, entropy loss in PPO typically falls within $[10^{-5}, 10^{-2}]$, as seen in the Stable Baselines3 Zoo.

Are the bounds of Eq.(1) and (2) tight?

Eq.(1) and (2) can be unified as $|J(\pi_{g+\epsilon})-\frac{1}{1-\gamma}\sum_{(s,a)}\rho_{\pi_g}(s,a)r(s,a,g+\epsilon)| \le |\frac{2\sqrt{2}R_{\text{max}}}{(1-\gamma)^2}\sqrt{D_{KL}(\pi_{g+\epsilon}(\cdot|s),\pi_g(\cdot|s))}|$. We examine the tightness of this bound up to a constant with a particular MDP instance. In the case of $g_1$ with policy $\pi_{g_1}(\cdot|s_0)=(0.6,0.4)$:

1. For $\pi_{g_2}$ identical to $\pi_{g_1}$, the left-hand side equals $\frac{0.2\gamma}{1-\gamma}$, while the right-hand side is $\frac{0.4\gamma}{1-\gamma}$, showing a constant factor difference.
2. When $\pi_{g_2}$ is slightly different, with $\pi_{g_2}(\cdot|s_0)=(0.55,0.45)$, the left-hand side becomes $\frac{0.15\gamma}{1-\gamma}$, and the right-hand side is $\frac{0.4\gamma + 0.2}{1-\gamma}$, again differing by a constant factor.

We will include the detailed proof in the Appendix of manuscript.

(1) Is Figure 4(b) converged?. (2) The lack of explanation for Figure 7's results ...

Thank you for your suggestion. We have increased the training steps for HER and MSR-HER on Push and updated Fig.4 and added a convergence analysis in the manuscript. Fig.7 illustrates that a larger MSR strength $\lambda$ results in a stronger constraint on $D_{KL}(\pi_{g+\epsilon}(\cdot|s),\pi_g(\cdot|s))$ (values approach 0). We will provide a detailed analysis of the experimental data and corresponding conclusions in the manuscript.

We sincerely appreciate your valuable feedback and careful review of our paper. We address your concerns in the following:

My only remark is that a higher number of random seed varying trials would improve their statistical evaluation of the experimental results, as the current number (5 trials) yields relatively large statistical error bars on the individual KPIs of Table 1.

Thank you for your suggestion. We have increased the number of random seeds from 5 to 10, re-conducted the training and testing, and updated Table 1. The table below presents the results on VVC, showing that the variance of the obtained results has decreased to some extent.

Additionally, we have expanded our discussion to include the statistical significance of the results. Taking $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$ on VVC as an example, the table below presents the T-test's $\alpha$ values for the evaluation results of policies trained with 5 different random seeds between HER and MSR-HER. All the $\alpha$ values are below 0.05, indicating that MSR-HER can train policies with a smaller $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$ compared to HER.

Note: We trained multiple HER and MSR-HER policies with distinct random seeds (numbers in parentheses indicate the seed index), each evaluated over 12100 episodes. As the T-test necessitates i.i.d. data for both groups being compared, and the test outcomes from different seeds do not belong to the same distribution, we can only conduct pairwise comparisons between the policies trained by HER and MSR-HER with varying random seeds.

We are continuing to increase the number of random seeds to obtain more reliable experimental results. Once again, we thank you for your valuable suggestion.

However, the authors must improve the readability of axis labels and (color) legends used in their figure, as they are currently far too small or sometimes missing/unreadable at all. This is a mandatory issue for any final version provided if accepted for publication. Some of the plots in the main text are borderline small, so I strongly recommend finding a solution to make central results more visible.

We sincerely thank the reviewer for highlighting the issue regarding the readability of our figure axis labels and legends. To enhance the readability of all figures, we have taken the following steps:

1. We have increased the font size of all axis labels and legends to ensure they are easily readable.
2. For figures utilizing color legends, we have not only enlarged the text but also ensured that the color distinctions are more pronounced and clearly labeled.

If indeed GC-PPO fails entirely at solving the general RL tasks in the mentioned benchmarks, it is questionable to consider any of those results to draw conclusions about the (dis-)continuity properties and total return accumulation (as partly admitted by the authors). Maybe those results should be marked or put into brackets in Table 1, to prevent false conclusions.

As suggested, we have marked the results of GC-PPO in Table 1 with italics to indicate that these results should be interpreted with caution due to the baseline algorithm's performance limitations. Additionally, we have included a description in the title to explain this notation. Thank you once again for your valuable insights.

W1: Theory To Practice

In our work, we aim to address the issue of ensuring that if a policy can achieve a goal $g$, it should also be capable of achieving goals in the vicinity of $g$, which we denote as $g+\epsilon$. From the perspective of cumulative rewards, our objective is to minimize $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')}|J(\pi(\cdot|\cdot,g))-J(\pi(\cdot|\cdot,g+\epsilon))|$.

Our method is informed by the following insights:

1. We show analytically that $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')} |J(\pi(\cdot|\cdot,g))-J(\pi(\cdot|\cdot,g+\epsilon))|$ can be bounded by $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$, as concluded in Theorem 3.4, Corollary 3.5 and 3.6. This led us to consider constraining $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$ during policy optimization.
2. Theorem 3.3 suggests that the difference between $\pi(\cdot|\cdot,g)$ and $\pi(\cdot|\cdot,g+\epsilon)$ should not be entirely eliminated, implying that $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$ should not be optimized to be too small.

Based on these findings, we develop the margin-based policy self-regularization method, as outlined in Eq.(6). The $KL$ term in the equation corresponds to the first point, and the max operation corresponds to the second point.

W2: Empirical Evaluation

We address your concerns from the following three aspects:

Firstly, we believe that our experimental design aligns well with our theoretical derivations.

In conjunction with our response to W1, our method aims to reduce $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')}|J(\pi(\cdot|\cdot,g))-J(\pi(\cdot|\cdot,g+\epsilon))|$ by constraining $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$, thereby achieving the objective of enabling the policy to achieve not only $g$ but also goals in the vicinity of $g$. When designing our experiments, we sought to answer the following questions:

1. Can MSR reduce $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$?
2. whether a reduction in $KL[\pi(\cdot|\cdot,g)||\pi(\cdot|\cdot,g+\epsilon)]$ leads to a decrease in $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')}|J(\pi(\cdot|\cdot,g))-J(\pi(\cdot|\cdot,g+\epsilon))|$?
3. can MSR enhance the policy's cumulative rewards, $J(\pi)$?

The first two questions directly correspond to our theoretical derivations, while the third question assesses the practicality of our method, indicating that improving the continuity of the policy's goal-achievement can also enhance its ability to obtain cumulative rewards. Therefore, we consider our experimental design to be well-aligned with our theoretical development.

Secondly, our focus is on improving policy continuity for goal achievement by reducing $E_{g\sim p_{dg},\epsilon \sim(-\epsilon',\epsilon')}|J(\pi(\cdot|\cdot,g))-J(\pi(\cdot|\cdot,g+\epsilon))|$, not on reaching farther goals, long-horizon reasoning, or robustness to environmental perturbations.

Thirdly, our Section 4.4 ablation study with $\lambda=1$ shows $KL(\pi_g||\pi_{g+\epsilon})$ nearing 0 (Fig.7(a)), but $J(\pi)$ is lowest (Fig.7(b)). This observation aligns with Theorem 3.3, emphasizing the need to balance the reduction of $KL(\pi_g||\pi_{g+\epsilon})$ to maintain goal-achievement continuity without overly compromising performance.

W3: Contribution and Utility

The KL divergence is commonly used as a measure of distance to constrain policies in policy optimization. In the context of multi-goal settings, we compare MSR method with two KL-based policy constraint approaches:

1. TRPO aims to prevent the policy from changing too much during optimization, thus it achieves this objective by constraining the KL divergence between policies in consecutive training iterations.
2. The Offline-to-Online method aims to prevent the policy from forgetting the knowledge learned offline during online learning, by constraining the KL divergence between the offline-learned policy, $\pi_0$, and the policy currently being learned online, $\pi_{\theta_t}$.
3. MSR, on the other hand, seeks to enhance the policy's continuity of goal-achievement ability by limiting the divergence between policies corresponding to adjacent goals.

In summary, although all these methods utilize KL divergence, from the perspective of optimization objectives, MSR is fundamentally distinct from other KL-based methods.

[1] Schulman, John, et al. Trust region policy optimization. ICML,2015.

[2] Baker, Bowen, et al. Video pretraining (vpt): Learning to act by watching unlabeled online videos. NeurIPS,2022.

Thank you for your thorough review and valuable feedback on our work. We address your concerns in the following:

W1: Many of the figure captions could be improved.

We greatly appreciate your suggestions. In accordance with the principles of accuracy and conciseness, we have re-formulated the titles of all figures and tables. Taking Figure 6 as an example, which illustrates the policy discrepancy between adjacent desired goals, $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$, under various $\beta$ settings, supporting our ablation analysis on $\beta$, we have revised its title from:

"Ablation study on $\beta$. $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$ between policies for adjacent desired goals. The interpretation of the coordinate axes and the data collection methods are analogous to those described in Fig.1. Results come from MSR-GC-SAC on Reach over 5 random seeds."

to:

"Policy discrepancy between adjacent desired goals, $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$, under different $\beta$ settings. The meanings of the coordinate axes and the evaluation methods are consistent with those in Fig.1. Results are derived from MSR-GC-SAC on Reach across 5 random seeds."

W2: It would be beneficial to verify that the claims made in Section 4 (that are based on Table 1) are valid with respect to statistical significance.

We appreciate your suggestion. Taking $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$ on VVC as an example, we have conducted a statistical significance analysis between HER and MSR-HER. The table below presents the T-test's $\alpha$ values for the evaluation results of policies trained with 5 different random seeds. All the $\alpha$ values are below 0.05, indicating that MSR-HER can train policies with a smaller $D_\text{KL}(\pi_{g+\epsilon}, \pi_g)$ compared to HER.

Note: We trained multiple HER and MSR-HER policies with distinct random seeds (numbers in parentheses indicate the seed index), each evaluated over 12100 episodes. As the T-test necessitates i.i.d. data for both groups being compared, and the test outcomes from different seeds do not belong to the same distribution, we can only conduct pairwise comparisons between the policies trained by HER and MSR-HER with varying random seeds.

Additionally, we noted the large variance in the results of Table 1. We have increased the number of random seeds from 5 to 10 and re-conducted the training and testing, updating Table 1. The updated table for the VVC shows a reduction in variance for the results.

Once again, we appreciate your suggestion and will include the discussion on statistical significance in our manuscript.