Diffusion-based Curriculum Reinforcement Learning

W1: The introduction dumps too much related work together [...]

We will restructure the introduction to separate and clarify the related work.

W2: [...] Why can't another method (like a VAE, or GAN) do this through modelling the state-visitation distribution? [...]

We acknowledge that other methods could effectively model the state distribution. However, we are cautious not to speculate on which method would yield the most successful output without further comparative analysis, which may be an interesting line of future research.

We believe that the noising/denoising mechanism of the diffusion model is particularly beneficial for the following reasons:

• Denoising (lines 5-7 in Alg. 1): Gaussian noise is incrementally reduced by the neural network according to a specific variance schedule. This process is inherently imperfect: due to the neural network's limitations in precisely matching the sampled noise, a small degree of randomness remains. We believe that this residual noise introduces a subtle variability in the curriculum goals, which aids exploration by generating slightly varied goals.
• Noising (lines 8-9 in Alg. 1): Original data sampled from a distribution are intentionally corrupted with Gaussian noise based on the sampled timestep k. The “noised” data are then processed by the neural network, and the loss is calculated (see Eq. 10 of the paper).

W3: [...] it is not clear from the text what problem the authors are trying to solve through the graph construction and the optimization of the curriculum goal [...]

We employ two different strategies for generating curriculum goals:

• Bipartite Graph Optimization Strategy: In this strategy, as outlined in the paper and also utilized in baseline methods like HGG and OUTPACE, we sample a mini-batch $b$ from the replay buffer $B$. This batch contains many states from different timesteps, which we provide to the curriculum goal generator, i.e., the diffusion model. The diffusion model generates a distribution of curriculum goal candidates $G_c$ from which we select the optimal curriculum goal $g_c$ using bipartite graph optimization. This graph maximizes the cost function given in Eq. 12.

  • Graph Construction: The vertices of the bipartite graph can be divided into two disjoint sets $V_x$ and $V_y$, where $V_x$ consists of the generated curriculum goals and $V_y$ includes the desired goals sampled from the desired goal distribution. The graph is not fully connected as every pair of distinct vertices in our graph is not connected by a unique edge, but each vertex from $V_x$ is only connected to vertices in $V_y$.
  • Purpose of Eq. 12: The objective here is to select K curriculum goals that are most diverse and beneficial for training, ensuring that the goals selected cover a broad spectrum of possible scenarios. We then randomly sample a single $g_c$ from these K goals for each episode.

• Single State Strategy (see Sec. B of the supp. mat.): This method involves feeding only the state from the last timestep into the diffusion model, which then generates a single curriculum goal. The rationale behind using the last timestep is the assumption that the final state is closer to the desired goal. However, this does not always hold true as agents might not always progress linearly towards the goal, sometimes moving backwards or sideways. In other words, if the last state does not accurately represent progress towards the desired goal, the generated curriculum goals might not optimally guide the agent, potentially leading to decreased sample efficiency and slower learning rates. This effect is demonstrated in Fig. 4 in Section B of our supp. mat.

W4: Fig. 1 overlaps with table 1 and contains too many overlapping lines to draw a conclusion [...]

We will revise Fig. 1 by reducing the number of baseline methods shown and displaying only the baselines that match our proposed method. Additionally, we will relocate some of the comparisons to the supp. mat.

W5: [...] states that there is a difference between OUTPACE and DiCuRL, however, neither method statistically significantly outperforms the other [...]

We acknowledge that our analysis might have focused excessively on the baselines’ performances. We will revise the discussion accordingly and move part of the discussion from the Appendix to the main text (given the possibility of including an extra page in case of acceptance).

Concerning the comparison between OUTPACE and DiCuRL, we have conducted a statistical analysis using the Wilcoxon rank-sum test to compare the no. of timesteps needed by the two methods to achieve a success rate greater than 0.99 across five different seeds for training. Here are the detailed test results for three specific environments:

• PointSpiralMaze: p=0.04
• PointNMaze: p=0.44
• PointUMaze: p=0.04

For PointSpiralMaze and PointUMaze, there is statistically significant evidence to reject the null hypothesis (p<0.05), suggesting that DiCuRL statistically outperforms OUTPACE in these environments. Conversely, for PointNMaze, this is not the case. We note however that with 5 samples this analysis may be limited.

W6: It is unclear from Fig. 3 at what point during training this plot was made [...]

We will add a color bar to Fig. 3 to indicate the timestep corresponding to each different color of the curriculum goals. You can refer to the new plot layout in Fig. 15 of the attached PDF, which shows the intermediate goals for OUTPACE.

Technical and minor comments

Due to space limitations, we cannot provide detailed replies. However, we acknowledge these comments and we'll do our best to handle them in the final version of the paper.

Q1: Could the authors revise the current version and improve upon (most) of my critiques, then I'd be willing to raise my score.

We did (and will do) our best to address all the comments.

Q2: Will the authors share code?

Of course! Please see the link reported at L264.

Thanks a lot for engaging in this discussion. We really appreciate your comments.

W3 [...] this is important and must be mentioned and cited.

We will clarify in the revised manuscript that the baselines (OUTPACE and HGG) also utilize the Minimum Cost Maximum Flow algorithm [a].

[a] Ravindra, K. A., et al. (1993) Network Flows: Theory, Algorithms, and Applications

• [...] Could the authors comment on…
  $\bar{g}$ represents the mean of the generated curriculum goals $\mathcal{G_c}$. It is true that $\bar{g}$ could potentially fall inside a wall, as our algorithm does not specifically prevent this because we assume that our algorithm is environment agnostic. However, we mitigate this issue through the use of diffusion loss (Eq. 10) in our total loss formulation (Eq. 11). This approach ensures that our generated curriculum goals are representative of the state-visitation distribution. Since the agent may collide with maze walls (it does not go through) during exploration, the state-visitation distribution typically does not include locations inside the walls. Consequently, the goals selected by minimizing the objective function are usually outside the walls.
  Despite these measures, the algorithm is not infallible. As illustrated in Figures 3a, 5a, and 6a, a minor amount of goals may still be generated inside the walls (actually, this same problem occurs also in OUTPACE).

• [...] How does this objective transfer to tasks [...]
  Indeed, the current objective function may not be suitable for tasks where distance over states are not applicable, such as in Atari games. However, as detailed in Section B of our supp. mat., we have demonstrated that our algorithms are also effective without relying on this specific objective function.

• [...] Can the term inside the square root of Eq. 12 become negative?
  In our implementation, we calculate the square of the difference $(g_i -\bar{g})^2$, to ensure non-negativity (see L200-201 in our repo https://anonymous.4open.science/r/diffusioncurriculum/hgg/hgg.py). However, it appears we inadvertently omitted the exponent in the paper. We will correct this to accurately reflect the calculation as $(g_i -\bar{g})^2$ in Eq. 12.

• [...] Is there prior work on other methods? [...]
  Indeed, Eq. 12 can be designed more effectively and represents a promising area for future research.As you suggested, integrating methods based on uncertainty or entropy is a viable approach. For instance, the baseline method OUTPACE effectively employs curriculum goal generation that targets states of uncertainty, as demonstrated in Figure 2 of the OUTPACE paper. We are open to exploring similar enhancements to our model to further refine its capabilities by incorporating such uncertainty or state-occupancy based scoring mechanisms in future work.

• [...] After line 12 in Algorithm 2, where is $g_c$ used? [...]
  In our algorithm, $g_c$ is utilized in several key parts: it serves as input to the policy network (line 7) and the Q network (line 13), and it is also used in the computation of the AIM reward (line 14). Once $g_c$ is determined, it defines the goal that the agent aims to achieve. At the end of the episodes, we assess the algorithm's success rate by sampling a desired goal from the goal distribution $\mathcal{G}$ and observing how many of the test rollouts are successfully completed by our agent in achieving the given desired goal.

• [...] Line 12 algorithm 2, shouldn't $g_c$ be a set $\mathcal{G_c}$?
  Indeed, at the end of the process, we select a single curriculum goal, referred to as $g_c$. More specifically, given the set $\mathcal{G_c}$ in Eq. 12, we select K curriculum goals that minimize the equation. We then randomly sample a single $g_c$ from these K goals for each episode. We will revise Line 12 in Algorithm 2 to more clearly reflect this process and ensure that the line is accurate and unambiguous.

W2 [...] I think the paper would greatly benefit [...]

We will certainly add this to the final version of the paper.

W4 [...] figure 4 in the appendix B [...]

We chose to crop the plot titles and included the necessary information in the captions under each subplot for clarity. We will adjust the layout in the final version of the paper.

W5 I believe that doing statistical tests [...]

Thanks a lot for pointing us to this reference. Honestly we were also a bit hesitant in showing the statistical analysis because of the small sample size, as we admitted in the previous reply. That’s why in the paper we reported the results in terms of curves and descriptive statistics on the number of timesteps needed to reach a given success rate, as also done in other papers in the field.

Concerning the way we discussed the results in the results section, we will do our best to revise the text to make it read less speculative, highlighting the similar performance of the methods whenever appropriate.

Thank you!

W1: The first paragraph of the introduction is unnecessarily long [...]

We will restructure the introduction to ensure it is more concise and better organized, which we believe will significantly improve the clarity and readability of the paper.

W2: While the related work section describes several existing works in detail, it fails to differentiate these works from the proposed method exactly.

We address this comment under W6.

W3: [...] I feel it is entirely possible to extend the proposed method to the general RL setup, where only the current state is given. This will greatly increase the applicability of the proposed method.

We recognize that the necessity for giving a curriculum goal can restrict the applicability of our approach in real-world scenarios where such goals are not readily available or definable.

Indeed, extending our method to a general RL setup, where the system operates solely based on the current state without explicit goal definitions, is a valuable direction for future research. Inspired by your suggestion and valuable references, we are considering an approach similar to that described in [a]: in this approach, a desired goal is defined using text encoding, and then a planner generates a series of future frames that illustrate the actions. Control actions are then derived from this generated video, enabling the agent to navigate without the need for specifically predefined goals. Additionally, we could adapt our policy (possibly our Q function using diffusion models), as demonstrated in [b]. Alternatively, we might explore learning directly from the environment using techniques such as RGB video, as proposed in [c], or through point clouds.

W4: [...] It would be a lot more convincing if the evaluation was also conducted in other domains, such as robot arm manipulation, locomotion, and games [...]

Given the time constraints, we have carried out additional experiments on two robot manipulation tasks, although it would be entirely possible to apply to other domains such as indeed locomotion and games as well as discrete state and action spaces. Please see our General comment and its attached PDF (Fig. 16). These experiments hopefully demonstrate the applicability of our method to tasks that are quite different from maze navigation.

W5: According to Fig. 1, I am not entirely convinced that the proposed method performs significantly better than the baselines. Also, the plotting scheme makes it difficult to interpret when many curves overlap.

We will provide clearer plots in our supp. mat. by creating different subplots and plotting together groups of 2 to 3 baseline methods. We hope that comparing only 2 or 3 baseline methods in each subplot will make the comparison clearer and easier to understand.

Concerning the comparison between OUTPACE and DiCuRL, we have conducted a statistical analysis using the Wilcoxon rank-sum test to compare the no. of timesteps needed by the two methods to achieve a success rate greater than 0.99 across five different seeds for training. Here are the detailed test results for three specific environments:

• PointSpiralMaze: p=0.04
• PointNMaze: p=0.44
• PointUMaze: p=0.04

For PointSpiralMaze and PointUMaze, there is statistically significant evidence to reject the null hypothesis (p<0.05), suggesting that DiCuRL statistically outperforms OUTPACE in these environments. Conversely, for PointNMaze, this is not the case. We note however that with 5 samples this analysis may be limited.

W6: The related work section focuses on existing works in curriculum RL yet fails to discuss many works that use diffusion models for RL or imitation learning, including but not limited to [...]

We acknowledge our oversight and appreciate your detailed suggestions. We will thoroughly review these studies and will update the related work section to discuss them extensively, highlighting how they relate to our research.

W7: While Algorithm 2 is titled RL Training, Lines 15-21 are for evaluation/testing, which is a bit confusing.

Thank you for pointing this out. We will update our algorithm title to "Algorithm 2: RL Training and Testing”.

W8: L282: It seems that a non-break newline is used here, which gives no space between this paragraph and the next paragraph starting from Line 283.

Thank you for noticing this layout issue. We will fix it in the revised version of the paper.

[a] Du, Y., et al. (2024), Learning universal policies via text-guided video generation

[b] Chi, C., et al. (2023), Diffusion policy: Visuomotor policy learning via action diffusion

[c] Ko, P-C, et al. (2023), Learning to act from actionless videos through dense correspondences

Thank you for the rebuttal with additional experiments, which address some of my concerns, including fixing Algorithm 2, and the significance of the results.

Thanks for acknowledging our rebuttal and for pointing out the significance of the results.

Clarity & minor errors: Given that NeurIPS does not allow updating submissions during the rebuttal period, it is difficult for me to assume that the authors would completely fix the issues. I would still say the paper in its current form is not ready for publication.

As far as we can see, the point of the NeurIPS rebuttal process is to engage in a scientific discussion with peers to receive suggestions on how papers can be improved to meet the quality standards of the conference. From the conference website: “Authors may not submit revisions of their paper or supplemental material, but may post their responses as a discussion in OpenReview. This is to reduce the burden on authors to have to revise their paper in a rush during the short rebuttal period.” The entire spirit of the rebuttal process is based on the idea that authors can revise an accepted paper right after the rebuttal period (not in a rush). After all, if only papers that are already good as they were to be accepted, what would be the point of the rebuttal phase?

Given this, as said we honestly promised that we will fix those issues, and we will do it. It would be against our professional ethics not to do that.

Limited to goal-conditioned RL: I appreciate the discussion of how the proposed method can apply beyond the goal-conditioned RL scenario. However, without seeing the results, it is not convincing.

We limited our experimentation to goal-conditioned scenarios to align our experimental setup with that of our baselines, and compare against state-of-the-art results. Testing our method on tasks beyond goal-conditioned scenarios has never been in the scope of this paper, but as we said this can certainly be an interesting direction for future research for the whole community.

Related work: While the authors made a promise to discuss how their work differs from the references I provided, I am unsure how this will go without seeing the actual revised paper, which, again, unfortunately, is not possible during the NeurIPS rebuttal.

We are a bit puzzled also by this comment, to be honest. We, as authors, are obviously devoted to maintaining a highly ethical scientific conduct, as we have always done. As said in our previous reply, we will analyze these works and include them in our discussion in the revised version of the paper (as a matter of fact, we have started working on it, please see below our reply on the evaluation).

Evaluation is limited to the Maze navigation: I really appreciate the additional results of FetchPush and FetchPickAndPlace. However, I still believe the evaluation is limited. As I suggested, including navigation tasks with discrete state and action spaces, locomotion tasks, and image-based games would significantly strengthen the contributions of this work.

We politely disagree on this. Our evaluation setup is actually well aligned with most of the works from the state of the art in the field, including the papers introducing our baselines and the papers suggested by the reviewer in the previous comment.

Considering the papers introducing the baselines we used in our experiments:

• ACL [18] → 3 tasks (synthetic language modelling on text generated by n-gr models, repeat copy and the bAbI tasks)
• GoalGAN [14] → only Ant Maze tasks
• HGG [19] → 4 robot manipulation and 4 hand manipulation tasks
• ALP-GMM [17] → BipedalWalker
• VDS [16] → 4 Robot manipulation tasks, 9 hand manipulation tasks , 3 maze tasks
• CURROT [23] → Bipedal Walker and Maze tasks
• GRADIENT [13] → FetchPush and Maze tasks
• OUTPACE [12] → 3 maze with points agent, 1 maze with Ant, 2 robotic tasks

Concerning the papers suggested by the reviewer:

• Learning Universal Policies via Text-Guided Video Generation → combinatorial robot planning tasks (real robotic system as well)
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion → 5 Robotic manipulation tasks (real robotic system as well)
• Learning to Act from Actionless Video through Dense Correspondences → 6 different robot manipulations tasks + 4 different tasks + 3 real world tasks
• Goal-conditioned imitation learning using score-based diffusion policies → 3 different robotic tasks (no real world task)
• Diffusion Model-Augmented Behavioral Cloning → 1 Maze, 1 FetchPick, 1 Hand, 1 cheetah, 1 walter, 1 AntReach
• Imitating human behaviour with diffusion models → CLAW environment, Kitchen environments, CSGO

I have mixed feelings about this work [...]

We sincerely appreciate your engagement in the rebuttal, although of course we would hope for a score improvement. We are aware that the paper can be improved and as said we will do our best to do that if the paper were to be accepted.

Dear Reviewer,

Thanks again for the engaging discussion. Please note that we have thoroughly revised the Introduction and the Related Work sections (see the comments we posted under the General comment section).

Specifically, in the Introduction we clearly highlighted the novel aspects of our algorithm (see the Contributions paragraph), and reorganized the overall text to make it more effective (splitting the first paragraph in a more coherent way). In the Related Work section, according to your suggestions, we included the references you kindly provided and added a brief paragraph highlighting the limitations of current works and the distinctive elements of our paper.

We do hope that the revised versions of Section 1 and Section 2 address your concerns regarding Claritiy and Related work. We would really appreciate it if you could acknowledge this reply.

Best regards,

The authors

W1: The introduction section should be improved in terms of writing [...]

We agree with the reviewer's suggestions: we will restructure the introduction to ensure it is more concise and better organized.

W2: [...], the paper does not demonstrate the curricula generated by OUTPACE [...]

In the attached PDF, we show the generated curriculum goals of OUTPACE for all maze environments (Fig. 15). Additionally, we have added a color bar to indicate which colors of the curriculum goals correspond to which timesteps. These figures will be included in the final version of the paper. Furthermore, we will update Fig. 3 in the paper to include the color bars.

W3: [...] the empirical validation is insufficient to conclude that DiCuRL can outperform state-of-the-art [...]

We have additionally tested our approach on two robotic manipulation tasks. Please refer to our General comment and the attached PDF (Fig. 16) for more details.

W4: The roles of loss components related to the Q-function and AIM reward [... ]

We have conducted an ablation study to explore the individual contributions of the Q function and the AIM reward when integrated with the diffusion loss $L_d$, as outlined in Eq. 11 in the paper. The success rate results from this study are presented in Fig. 13a in the attached PDF.

Concerning the roles of the Q function and AIM reward, here’s our intuitive explanation:

• Loss $L_d$: Minimizing this component helps us accurately capture the state distribution. It ensures that our generated curriculum goals are representative of the state-visitation distribution.

• Q-function and AIM reward: The Q-function predicts the cumulative reward starting from a state, following the policy, while the AIM reward estimates how close an agent is to achieving its goal. We integrate both terms in the loss function by inverting their signs because our objective is to maximize Q and the AIM reward. By doing this, the diffusion model focuses on generating goals that not only minimize $L_d$ but also maximize the expected Q and AIM rewards.

This approach ensures that our generated curriculum goals are neither overly simplistic nor excessively challenging and progress towards the desired goal.

We have provided a detailed analysis with various visualizations for the PointUMaze environment in Sec. D in our supp. mat. If further details are required, we can provide a similar analysis for the PointSpiralMaze environment as well.

Q1: How do Q and AIM rewards differ in a goal-conditioned environment [...]

To demonstrate the effects of using either the AIM reward or the $Q_\phi$ function in conjunction with the $L_d$ component in Eq. 11 in the paper, we have provided illustrative examples in the attached PDF. In particular, Fig. 13c and 13d display the generated curriculum goals using $L_d$ with only $Q_\phi$ and $L_d$ only with the AIM reward, respectively. The generated curriculum goals reveal that omitting the AIM reward results in suboptimal performance, whereas the absence of the $Q_\phi$ function leads to the agent's inability to accomplish the task.

Additionally, we have implemented our method within a sparse reward, goal-conditioned RL framework across two different robotic manipulation tasks. We compared our method with HGG [a] and HER [b], as detailed in Fig. 16 in the attached PDF.

In this setting, the curriculum goals are integrated into our policy and the Q network (see line 7 in Algorithm 2 for the policy and line 10 in Algorithm 1 for the Q function). We utilize the AIM reward for both training RL algorithms and generating curriculum goals. This reward function, also a feature of OUTPACE, is a trainable neural network parametrized by $\varphi$ that is initialized randomly and trained simultaneously with the RL algorithms. It is specifically trained to minimize the Wasserstein distance between state visitation and the desired goal distribution, as detailed in Section 3.3.

The distinction between training RL algorithms using the sparse (binary) reward and the AIM reward is significant. For instance, consider an agent at position (x,y) with a goal at (m,n). In a sparse reward setting, such as that used in the HER methodology, the reward is calculated based on the Euclidean distance between the agent's position and the goal. If this distance is greater than a predefined threshold (set to 0.05m in Gymnasium-Robotics), the agent receives a reward of 0; otherwise, it receives a reward of 1. In other words, If the agent reaches a given goal within a given threshold distance, the agent receives a positive reward, otherwise a non-positive one. In contrast, both our method and OUTPACE utilize a neural network-based reward—the AIM reward—which provides a continuous reward value. To illustrate this concept, we have included a 2D color map of the AIM reward function for the PointUMaze environment, showing how the reward evolves during different training episodes. These visualizations can be found in Fig. 8a, 10a, and 12a of our supp. mat. Additionally, the changes in the AIM reward for the PointSpiralmaze environment are illustrated in Fig. 13b in the attached PDF.

Q2: What is g_d that initializes g_c in Algorithm 2?

$g_d$ is the desired goal, and we initialize $g_c$ with $g_d$. At the beginning of training, our diffusion algorithm has not yet been run, so we initially provide the agent with $g_d$. Then, our curriculum goal generation algorithm generates curriculum goals for the agent.

Q3: What do colors in figures illustrating curricula stand for specifically?

We will add a color bar to Fig. 3 to illustrate the corresponding timesteps for each different color of the curriculum goals. You can refer to the new plot in Fig. 15 of the attached PDF, which shows the intermediate goals for OUTPACE.

[a] Andrychowicz, M., et al. (2017), Hindsight experience replay

[b] Ren, Z., et al. (2019), Exploration via hindsight goal generation

W1: The approach is quite complicated [...]

We carried out additional experiments on two robot manipulation tasks, please see the General comment and its attached PDF (Fig. 16) for more details.

W2:They missed citing a rich literature [...]

We will carefully review these papers and integrate them appropriately into the related work section to ensure a comprehensive discussion of existing research.

W3:The reward function for the Maze envs is not provided [...]

We are using a trainable reward function both for training the RL algorithm and for generating curriculum goals. The trainable reward function, see Eq. 4, is learned by minimizing the Wasserstein distance between state visitation and the desired goal distribution. Given that we are working with maze environments, using Euclidean distance would not be appropriate for a reward function. Following the same reward approach as OUTPACE, we use the AIM reward function trained simultaneously with the RL algorithms. As demonstrated in the supp. mat. and Fig. 13b in the attached PDF, the reward function converges to the desired goal as training progresses. Therefore, we integrated the reward function into our curriculum goal generation mechanism, as it is useful in Eq. 11 and has the potential to guide the agent toward the desired goal. In summary, we do not use a sparse or pre-defined dense reward equation. Instead, we utilize a trainable reward function which is randomly initialized and specifically trained to minimize the Wasserstein distance between state-visitation and the desired goal distribution.

Q1: In Fig. 3, what do the colours represent?

We will add a color bar to Fig. 3, which illustrates the corresponding timesteps for each different color of the curriculum goals. You can refer to our new plot in Fig. 15 in the attached PDF to have a preview of the modified figure.

Q2: [...] How would DiCuRL + fixed start state fare against SAC only + random start states?

Q3: How does SAC only perform in the comparisons in Fig. 1?

To address jointly Q2 and Q3, we removed the curriculum goal generation part from the code and provided only the final desired goal to agents in the maze tasks, which corresponds to training the agent solely with SAC. We obtained the first set of results by starting the agent in a fixed initial state at $(0,0)$. For the second set of results, we uniformly sampled the initial position of the agent randomly in the environment. To avoid starting the agent inside the walls of the maze, we performed an infeasibility check, such that if the initial state sampled was inside the walls, we continued sampling until a feasible initial state was found. We compared our DICURL approach with both SAC+fixed initial state and SAC + random initial state across all maze environments. The success rate is shown in Fig. 14 of the attached PDF. We observe that including curriculum goals significantly improves the success rate in simpler tasks such as PointUMaze, helping the agent achieve the task successfully across different training seeds. Without the curriculum (SAC + fixed initial position), the agent struggles to achieve complex tasks consistently, resulting in high variance in performance. With SAC + random initial state, the agent often reaches the desired goal, at least in some scenarios. This success can be attributed to trial and error (without requiring curriculum goals) because the random starting positions help the agent avoid becoming stuck in maze walls, thus enhancing its ability to navigate the environment effectively.

Q4: How important is the AIM reward? [...]

You are absolutely right; this is why we need to use different coefficients. To illustrate the effects of the Q and AIM reward functions, we conducted an ablation study. Please see our General comment and its attached PDF (Fig. 13a) for more details.

Q5: [...] can the AIM reward be substituted with simpler intrinsic motivation rewards [...]

Yes, we believe that both RND and TD-error can potentially be substituted with the AIM reward. Indeed, in the AIM reward paper [a], the authors compare this approach with RND and show that RND performs worse compared to the AIM reward in certain tasks. Based on this evidence, we preferred to use the AIM reward for our method.

Q6: [...] How does DiCuRL compare against SAC + HER?

Given the timeframe we had, we couldn’t test SAC+HER on the maze tasks. Instead, to demonstrate that our approach can be extended to robotic manipulation and sparse reward tasks, and compare it to HER, we used the official repository of [b], where the authors compare their approach with HER in a sparse reward setting. As detailed in the General comment, we implemented DiCuRL using HER settings in sparse reward robotic manipulation tasks. Please note that we used DDPG instead of SAC, which is another off-policy RL algorithm. All methods, including ours, use sparse rewards for training DDPG. However, since our method is based on the AIM reward, we only use the AIM reward for generating curriculum goals. You can find the comparison in Fig. 16 of the attached PDF.

Limitations: Please suggest how this work can be extended to challenging environments with larger state-action spaces.

To demonstrate that our proposed method can be extended to more complex environments such as robotic manipulation tasks, we compared our approach with HGG and HER. The success rates are shown in Fig. 16 of the attached PDF. We will be happy to include these additional experiments -along with a brief discussion of these new results- in the final version of our paper.

[a] Durugkar, I., et al. (2021), Adversarial intrinsic motivation for reinforcement learning

[b] Ren, Z., et al. (2019), Exploration via hindsight goal generation

Dear Reviewer,

Please note that we have revised the Related Work section as well, including the references you suggested (see the comments we posted under the General comment section).

Best regards,

The authors

Dear Reviewers and Chairs,

Please find below our revised version of the introduction (we post this under the General comment because changes on the intro have been asked concurrently by FHis and CrFc. The intro has been restructured to be more effective (we removed the bullet points at the end and merged them into the Contributions paragraph, and we balanced the length of all paragraphs). Note that we are also working on the related work section to include a critical analysis of the papers suggested by c6He and CrFc, discussing the differences w.r.t. our proposed method.

1 Introduction

Reinforcement learning (RL) is a computational method that allows an agent to discover optimal actions through trial and error by receiving rewards and adapting its strategy to maximize cumulative rewards. Deep RL, which integrates deep neural networks (NNs) with RL, is an effective way to solve large-dimensional decision-making problems, such as learning to play video games [1, 2], chess [3], Go [4], and robot manipulation tasks [5, 6, 7, 8]. One of the main advantages of deep RL is that it can tackle difficult search problems where the expected behaviors and rewards are often sparsely observed. The drawback, however, is that it typically needs to thoroughly explore the state space, which can be costly especially when the dimensionality of this space grows.

Some methods, such as reward shaping [9], can mitigate the burden of exploration, but they require domain knowledge and prior task inspection, which limits their applicability. Alternative strategies have been proposed to enhance the exploration efficiency in a domain-agnostic way, such as prioritizing replay sampling [8, 10, 11] or generating intermediate goals [12, 13, 14, 15, 16, 17, 18, 19]. This latter approach, known as Curriculum Reinforcement Learning (CRL), focuses on designing a suitable curriculum to guide the agent gradually toward the desired goal.

Various approaches have been proposed for the generation of curriculum goals. Some methods focus on interpolation between a source task distribution and a target task distribution [20, 21, 22, 17]. However, these methods often rely on assumptions that may not hold in complex RL environments, such as specific parameterization of distributions, hence ignoring the manifold structure in space. Other approaches adopt optimal transport [13, 23], but they are typically applied in less challenging exploration scenarios. Curriculum generation based on uncertainty awareness has also been explored, but such methods often struggle with identifying uncertain areas as the goal space expands [15, 24, 12].

Some research minimizes the distance between generated curriculum and desired outcome distributions using Euclidean distance, although this approach can be problematic in certain environments [19, 25]. Other methods incorporate graph-based planning, but require an explicit specification of obstacles [26, 27]. Lastly, approaches based on generative AI models have been proposed. For instance, [14] uses GANs to generate tasks of intermediate difficulty, but it relies on arbitrary thresholds. Alternatively, [28, 29, 30] apply diffusion models in offline RL settings [14, 28, 30].

Despite these advancements, existing CRL approaches still struggle with generating suitable intermediate goals, particularly in complex environments with significant exploration challenges. To overcome this challenge, in this paper, we propose DICURL (Diffusion Curriculum Reinforcement Learning). Our method leverages conditional diffusion models to dynamically generate curriculum goals, guiding agents towards desired goals while simultaneously considering the $Q$-function and a trainable reward function based on Adversarial Intrinsic Motivation (AIM) [31].

Contributions Unlike previous offline RL approaches [28, 29, 30] that train and use diffusion models for planning or policy generation relying on pre-existing data, DICURL facilitates online learning, enabling agents to learn effectively without requiring domain-specific knowledge. This is achieved by three key elements. (1) The diffusion model captures the distribution of visited states and facilitates exploration through its inherent noising and denoising mechanism. (2) As the $Q$-function predicts the cumulative reward starting from a state and a given goal while following a policy, we can determine feasible goals by maximizing the $Q$-function, ensuring that the generated goals are challenging yet achievable for the agent. (3) The AIM reward function estimates the agent's proximity to the desired goal and allows us to progressively shift the curriculum towards the desired goal.

We compare our proposed approach with nine state-of-the-art CRL baselines in three different maze environments and two robotic manipulation tasks simulated in MuJoCo [32]. Our results show that DICURL surpasses or performs on par with the state-of-the-art CRL algorithms.