Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay

Q1. Experiments: i) Comparison to external baselines of existing continual learning and offline RL methods; ii) Evaluation on more testbeds like Meta-World, CausualWorld, RLBench, or CompoSuite; iii) CuGRO's scalability with the number of tasks in continual learning.

A1: i) As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space. Hence, following the classical DGR method [1], we compare the performance of the RL model trained with variants of replay methods: Oracle, None, and Noise. For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity. For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator. For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism. We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios. For example, [2] belongs to the parameter isolation kind, [3, 4] needs to replay real samples of past tasks, and most of the existing methods [2, 3, 5] tackle the online RL problems other than the offline setting.

ii) These testbeds are usually used to test traditional online RL algorithms and has not been applied to offline scenarios yet. Compared with online continual RL, offline continual RL is less studied, and there is no standard dataset for it. Generally, the MuJoCo environment, as seen in the D4RL dataset [6], is a popular testbed for offline RL. The MuJoCo environment is also commonly used in offline meta-RL for testing [7, 8]. Hence, we evaluate CORL algorithms on MuJoCo environments with offline datasets in this paper.

iii) We follow the generative replay paradigm, e.g., DGR [1] and DDGR[9], where the scalability primarily depends on the modeling fidelity of the generator. This rationale underpins the choice of diffusion models for our method, mirroring the research motivation of DDGR. In DDGR, diffusion models demonstrate better fidelity compared to GANs, and exhibit promising superiorities for continual learning based on generated replay.
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we believe that this approach will see further developments in the future.

In summary, it is important to evaluate CORL methods on more testbeds, or to conduct empirical comparisons to existing methods for demonstrating their respective pros and cons, or to modify these online continual methods to adapt to offline settings. We leave these for future work.

[1] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[2] Gaya et al., Building a subspace of policies for scalable continual learning, ICLR 2023.

[3] Maciej Wolczyk, et al., Disentangling transfer in continual reinforcement learning, NeurIPS 2022.

[4] Sibo Gai, et al., OER: Offline experience replay for continual offline reinforcement learning, ECAI 2023.

[5] David Rolnick, et al., Experience replay for continual learning, NeurIPS 2019.

[6] Justin Fu, et al., D4RL: Datasets for Deep Data-Driven Reinforcement Learning, arXiv:2004.07219, 2020.

[7] Eric Mitchell, et al., Offline meta-reinforcement learning with advantage weighting, ICML 2021.

[8] Haoqi Yuan, et al., Robust task representations for offline meta-reinforcement learning via contrastive learning, ICML 2022.

[9] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

Q4: Introduction: i) What new tasks emerge overwhelmingly' means; ii) The motivation for why CORL is special seems to be all about RL in general, and not specifically about offline RL; iii) There's always the question of whether the size of the model might surpass the size of replay buffers.

A1: i) It means that in a continual learning setting, new tasks keep emerging in a sequential manner.

ii) Our method does not operate in an online mode and is solely suitable for offline scenarios. Under the offline RL paradigm, we model behavior policies using a diffusion model.

iii) The size of the model will not exceed the size of the replay buffer. Taking the HalfCheetah-Vel environment as an example, we only require an additional state generative model to realize generative replay. The state generator has about 24M parameters, and each new task has about 1M samples where each sample ($s,a$) has a dimension of 26. As the number of tasks increases, the size of the replay buffer will continue to expand, but the model size will remain constant. Moreover, we assume that the data from previous tasks is not available, a challenge often encountered in real-world scenarios, making generative replay necessary.

Q5: Sec. 2 and Sec 3.1: i) The authors could consider moving the notation clarification to the beginning of Sec. 2; ii) Move the description of CORL in Sec 2; iii) Why are there two ($s,a,r,s'$) boxes in Figs. 1(b) and 1(c)?

A5: i) Following your suggestion, we have moved the notation clarification to the beginning of Sec. 2.

ii) Additionally, we have described CORL in Section 3.1 and have now relocated it to Sec. 2.

iii) In Fig. 1, in order to clearly show that real samples and generated samples are used together to train the behavior evaluation model, we use two ($s, a, r, s'$) boxes to correspond to the state generator box and the behavior generator box, respectively.

Q6: Sec. 4: i) "The generator is the only constraint on task performance" -- how is this an improvement over other continual learning methods? ii) Starting from previously trained models might be better or worse, but certainly not equivalent to joint training on the entire dataset; iii) What is the (final) performance of SAC/TD3 on the collected datasets? iv) Explanations of baselines

A6: i) Continual learning methods can mainly be divided into three categories: regularization-based, parameter isolation, and rehearsal methods. We focus on the rehearsal method, using experience replay, and propose using a generative model to generate replay samples rather than storing samples of old tasks. Therefore, the generator is the only constraint on task performance in our method. Compared with methods that need to store samples of old tasks, our method only requires storing the generative model, which greatly saves memory space.

ii) In the continual learning process, we train the networks using generated samples of old tasks and real samples of new tasks together. If the generative model based on the diffusion model is good enough, it can generate very accurate samples, equivalent to joint training based on real data from all current tasks.

iii) The final performance of SAC/TD3 on the collected dataset is shown in the table below.

| Swimmer-Dir | 1048.38 | | Hopper-Vel | 185.72 | | Walker2D-Params | 164.88 | | HalfCheetah-Vel | -24.75 |

iv) We have explained our baseline in Section 4. Oracle uses real samples of all previous tasks to sequentially train the dual generators, and to train the multi-head critic with behavior cloning. Noise considers the opposite case when generated samples do not resemble the real distribution at all. We omit the state generative model and use random noises as replayed state samples. The behavior generator and multi-head critic are sequentially trained using these randomly generated states. Oracle stores real samples of all old tasks, which is the best case in the replay method, and the performance of our method is very close to Oracle. Additionally, from the experimental results of 'None' and 'Noise,' we can see that the performance decreases when the state generative model is removed or there is no replay. Figure 3 shows the average performance including current and past tasks. Table 1 shows the final performance on all past tasks after training on the final task.

Q7: Typos/style/grammar issues.

A7: We sincerely thank the reviewer for their careful reading and have revised the paper according to your suggestions.

Q2. Clarity of approach: i) Change of the MDP distribution from task to task; ii) Claim of multimodal behaviors within one task and across tasks; iii) The weighting of losses across tasks; iv) About test loss in Eqs. (11) and (14); v) The order of training model components; vi) About behavior cloning.

A2: i) One task corresponds to an MDP, and tasks are generated from a specific MDP distribution in continual learning. We have explained the details of sequential tasks in Sec. 3.1 and Appendix B.

ii) The behavior policy of a single task may be a multimodal distribution. When considering multiple tasks, the distribution of all behavior policies might become even more multimodal. Hence, we use a diffusion model to fit the behavior policies of all tasks. In classification tasks, opposite predictions are resolved using a task indicator. Similarly, in RL, the diffusion model mimics behavioral strategies, while the decision-making is guided by the Q-function.

iii) In Eqs. (10) and (13), we present the general form of training loss and set $\beta$ to facilitate training. During the training process, the value of $\beta$ is 0.5.

iv) The goal of Eq. (8) is to learn a continual policy that maximizes the expected return over all encountered tasks, which aligns with the training objectives of Eqs. (10) and (13). The test loss in Eqs. (11) and (14) evaluates the training effectiveness of the diffusion model, analogous to the validation loss in supervised learning. The setting of the test loss is consistent with DGR [1].

v) We decouple the policy into a behavior generative model and an action evaluation model, and use the state generative model along with the behavior generative model to achieve generative replay. During the continual learning process, we use a continually updated state generation model to cover the state space of all tasks, enabling it to reconstruct the cumulative state space of all tasks based on task encoding. The order of training model components is actually shown in Fig. 1.

vi) In our method, action sampling is guided by Q-values, analogous to classical value function methods where the policy/behavior is derived from value functions. Hence, cloning the Q-function implicitly clones the behaviors, thus we call the process as behavior cloning.

Q3: Clarity of preliminaries: i) The description of advantage-weighted regression; ii) The relationship of Eqs. (3)-(5); iii) The description of the diffusion models in Sec 2.2 misses mentioning the existence of a forward diffusion process that adds noise to an (observed) action; iv) The description prior to Eq. (7); v) What does re-sampling mean toward the end of Sec 2.1?

A1: i) Following your suggestions, we have further explained $Q_\theta$ and $Q^\pi$ and their relationship.

ii) Eq. (4) is the general modeling form of offline RL, Eq. (5) is the decoupling form proposed by SfBC [10], and $Q_\theta$ is the same in Eqs. (4) and (5).

iii) In Section 2.2, we describe the forward diffusion process of the diffusion model. Diffusion models are generative models that begin by defining a forward process to gradually add noise to an unknown data distribution, and then learn to reverse this process.

iv) Eq. (7) is based on the fundamental concepts of offline RL.
$R_i^{(0)}$ is the vanilla return of trajectories. The first part of Eq. (7) offers an implicit planning scheme within dataset trajectories that mainly helps to avoid bootstrapping over unseen actions and accelerate convergence. The middle part of Eq. (7) enables the generalization of actions in similar states across different trajectories.
This is our basic approach, and you can refer to SfBC for details [10].

v) The 'first' sampling means sampling from the diffusion-based behavior generative model. Then, we evaluate the sampled actions with the trained critic network. The learned policy is a combination of the behavior generative model and the critic network. Finally, during execution, actions are 're-sampled' from the learned policy. More details can be found in [10].

[10] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

Q1: Incorporating diffusion models in offline RL for generative reply is straightforward, and the computation cost should be rigorously discussed.

A1: In the continual learning setting, the category of generative replay methods [1, 2] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks). That is, they eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional generative model, e.g., GANs [1] or diffusion models [2]. Taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has a dimension of 26. Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs.

In literature, current diffusion-based RL methods [3-8] could face the challenge of increased computational costs due to the training and sampling of diffusion models. For example, the diffusion-planning methods [4, 8] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples. We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [9, 10]. Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle the limitations in future work as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Q2: The convergence guarantee, in Eqs. (10)-(11) and (13)-(15), of training a diffusion model via using the data from the last diffusion model, especially when we only have access to offline data with a limited size or large state and action space.

A2: Under the generative replay paradigm [1, 2], we train the diffusion models using a mixed distribution of real data from the new task and pseudo data generated from previous generators. The convergence regarding Eqs. (10), (11), (13), and (14) align with the diffusion model. In Eq. (15), the first term follows a planning-based Bellman operator used in offline RL [3], and the second term corresponds to behavior cloning. Hence, the convergence regarding Eq. (15) is consistent with the specific Bellman operator [3] and the supervised stochastic gradient descent (SGD). The convergence of diffusion models is independent of the training data. Continually training the diffusion model with data generated by the previous iteration of the model only affects the quality of the training data and does not influence the model's convergence. Hence, the convergence remains consistent with the vanilla diffusion models. Additionally, the convergence of the model is independent of the dataset size.

[1] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[2] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

[3] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[4] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[5] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[6] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[7] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[8] Fei Ni, et al., MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023.

[9] Cheng Lu, et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.

[10] Qinsheng Zhang, et al., gDDIM: Generalized denoising diffusion implicit models, ICLR 2023.

Q3: Missing continual RL benchmarks (e.g., continual world) and baselines (e.g., [11]).

A3: As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space. Hence, following the classical DGR method [1], we compare the performance of the RL model trained with variants of replay methods: Oracle, None, and Noise. For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity. For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator. For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism. We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios. For example, [11] belongs to the model expansion kind, [12, 13] needs to replay real samples of past tasks, and most of the existing methods [11, 12, 14] tackle the online RL problems other than the offline setting.

'Continual world' is currently used to test traditional online RL algorithms and has not been applied to offline scenarios yet. Compared with online continual RL, offline continual RL is less studied, and there is no standard dataset for it. Generally, the MuJoCo environment, as seen in the D4RL dataset [16], is a popular testbed for offline RL. The MuJoCo environment is also commonly used in offline meta-RL for testing [8, 16, 17]. Hence, we evaluate CORL algorithms on MuJoCo environments with offline datasets in this paper.

On the other hand, it is important to evaluate CORL methods on more testbeds like the continual world in the future. It is also significant to conduct empirical comparisons to existing methods for demonstrating their respective pros and cons, or to modify these online continual methods to adapt to offline settings. We leave these for future work.

[1] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[8] Fei Ni, et al., MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023.

[11] Gaya et al., Building a subspace of policies for scalable continual learning, ICLR 2023.

[12] Maciej Wolczyk, et al., Disentangling transfer in continual reinforcement learning, NeurIPS 2022.

[13] Sibo Gai, et al., OER: Offline experience replay for continual offline reinforcement learning, ECAI 2023.

[14] David Rolnick, et al., Experience replay for continual learning, NeurIPS 2019.

[15] Justin Fu, et al., D4RL: Datasets for Deep Data-Driven Reinforcement Learning, arXiv:2004.07219, 2020.

[16] Eric Mitchell, et al., Offline meta-reinforcement learning with advantage weighting, ICML 2021.

[17] Haoqi Yuan, et al., Robust task representations for offline meta-reinforcement learning via contrastive learning, ICML 2022.

Q3: Efficiency trade-off: Training diffusion models for each different task might not be sample-efficient nor computation-efficient. Therefore, the authors might want to provide more information on the feasibility of this approach and the computation resource usage for implementing the experiments (e.g., how long to train the model).

A3: In the continual learning setting, the category of generative replay methods [8, 9] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks). That is, they eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional generative model, e.g., GANs [8] or diffusion models [9].

For sample efficiency, our method operates in an offline mode, and the size of the offline dataset is fixed. Hence, the sample efficiency is not a primary consideration.

For computation efficiency, taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has the dimension of 26. As the number of tasks increases, the number of encountered samples will continue to expand, while the model size remains constant. Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs. The advantage of generative replay becomes more remarkable for scenarios where massive datasets are needed to train a task or the continual learning setting contains a large number of sequential tasks. Additionally, the generative replay method can produce pseudo data as much as needed, which can also improve the robustness and flexibility of the model.

In literature, current diffusion-based RL methods [1-5] could face the challenge of increased computational costs due to the training and sampling of diffusion models. For example, the diffusion-planning methods [3] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples. We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [10, 11]. Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle the limitations as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Q4: I was wondering about the scalability of the model. If having more tasks degrade the model's performance?

A4: We follow the generative replay paradigm, e.g., DGR [8] and DDGR[9], where the scalability primarily depends on the modeling fidelity of the generator. This rationale underpins the choice of diffusion models for our method, mirroring the research motivation of DDGR. In DDGR, diffusion models demonstrate better fidelity compared to GANs, and exhibit promising superiorities for continual learning based on generated replay.
Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we believe that this approach will see further developments in the future.

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[3] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[8] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[9] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

[10] Cheng Lu, et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.

[11] Qinsheng Zhang, et al., gDDIM: Generalized denoising diffusion implicit models, ICLR 2023.

Q1: On the necessity of using diffusion models to learn behaviors from prior tasks, and the reason against the utilization of other models such as behavior cloning, GAN, VAE, etc. If applying other types of generative models to replace the diffusion model will yield similar performance?

A1: As repeatedly stated in our original paper, we have discussed the advantages of the diffusion model and the motivation for adopting it to tackle the CORL challenge. Due to the strong distributional expressivity and the ability to fit multimodal distributions, diffusion models have been explosively developed in the RL community recently [1-5], such as using diffusion models for behavior policy modeling [1], for target policy modeling [2], and for model-based trajectory modeling [2]. In the first paragraph of Sec. 3.2 (bottom, page 4), we elaborate on the necessity of using the diffusion model to learn behavior from prior tasks as: "This perspective rooted in generative modeling presents three promising advantages for CORL. First, existing policy models are usually unimodal Gaussians with limited distributional expressivity, while collected behaviors in CORL become progressively diverse as novel datasets keep emerging. Learning a generative behavior model is considerably simpler since sampling from the behavior policy can naturally encompass a diverse range of observed behaviors, and allows the policy to inherit the distributional expressivity of diffusion models. Second, RL models are more prone to deficient generalization across diverse tasks. Learning a unified behavior model can naturally absorb novel behavior patterns, continually promoting knowledge transfer and generalization for offline RL. Third, generative behavior modeling can harness extensive offline datasets from a wide range of tasks with pretraining, serving as a foundation model to facilitate finetuning for any downstream tasks. This aligns with the paradigm of large language models, and we reserve this promising avenue for future research."

On the other hand, other models such as behavior cloning, GANs, and VAEs can also be considered as alternatives to the diffusion model. In this paper, we choose the powerful diffusion model due to its promising properties discussed above. As the state-of-the-art architecture in the generative modeling community, the diffusion model has achieved great success in high-fidelity data generation, such as its superiority over GANs [6] and stable diffusion [7].

Q2: Experiment: i) Show how diffusion models contribute to continual learning performance; ii) Clarify the training data of the diffusion models; iii) Will noisy data degrade the performance of the diffusion models and the CuGRO model as a whole?

A2: i) As stated in our experiments, our primary concern is to validate whether our diffusion-based generative replay mechanism can realize high-fidelity replay of the sample space. Hence, we set three baselines as Oracle, None, and Noise. For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity. For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator. For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism. In summary, these observations verify the contributions of our diffusion models to the continual learning performance.

ii) The collection details of offline datasets are explained in Appendix B. As stated in Sec. 3.3, the diffusion models are trained using a mixed data distribution of real samples and replayed ones from the previous generator.

iii) We only use the noisy data in the Noise baseline, where the state generative model is replaced with Gaussian noise. In CuGRO, we train the diffusion models with a mixed distribution of real data from the new task and pseudo data generated from previous generators.

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[3] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[6] Prafulla Dhariwal et al., Diffusion models beat GANs on image synthesis, NeurIPS 2021.

[7] Robin Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.

Q3: Comparison to previous continual RL algorithms in addition to diffusion-based model generator pipeline.

A3: As we stated in the paper, continual learning methods can be categorized into three kinds: regularization, parameter isolation (model expansion), and rehearsal. Each of these three types has its own pros and cons, which aligns with the No Free Lunch Theorem in machine learning research. For example, regularization methods can continually absorb new knowledge with a constant network capacity, while comprising additional regularization terms may lead to a tradeoff on the accomplishment of old and new tasks. Model expansion methods can leave existing knowledge totally undisturbed via parameter isolation, while it consumes increasing network resources.

As we are the first to leverage expressive diffusion models to tackle the understudied CORL challenge, our primary concern is to validate whether our dual generative replay mechanism can realize high-fidelity replay of the sample space. Hence, following the classical DGR method [9], we compare the performance of the offline RL model trained with three variants of replay methods: Oracle, None, and Noise. For Oracle, the generated data is replaced with real data of past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high-fidelity. For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator. For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism.

We did not compare our method to several existing continual RL methods due to the difference in the intrinsic characteristics and applicable scenarios. For example, [11] belongs to the model expansion kind, [12] and [13] need to replay real samples of past tasks, and most of the existing methods [11, 12, 14] tackle the online RL problems other than the offline setting. On the other hand, it is significant to conduct empirical comparisons to these methods to demonstrate their respective pros and cons, or to modify these online continual methods to adapt to offline settings. We leave these for future work.

[9] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[11] Jean-Baptiste Gaya, et al., Building a subspace of policies for scalable continual learning, ICLR 2023.

[12] Maciej Wolczyk, et al., Disentangling transfer in continual reinforcement learning, NeurIPS 2022.

[13] Sibo Gai, et al., OER: Offline experience replay for continual offline reinforcement learning, ECAI 2023.

[14] David Rolnick, et al., Experience replay for continual learning, NeurIPS 2019.

Q1: Training two separate diffusion models can be computationally expensive. Exploring concatenating {s,a} and diffusing with one model could improve efficiency.

A1: Thank you for your insightful comments. We train two separate diffusion models since we adopt the diffusion-based generative behavior modeling method [1] as our backbone offline RL algorithm. This kind of backbone is well suited for continual learning problems, because training a unified behavior model can continually absorb new behavior patterns to promote forward knowledge transfer for the offline setting, and sampling from this generative model can naturally encompass a progressive range of observed behaviors. Then, for continual learning, we need to adopt another generative model to synthesize pseudo samples of past tasks under the paradigm of deep generative replay [9, 10].

On the other side, this architecture with two diffusion models could indeed increase computational costs. In literature, current diffusion-based RL methods [1-5] could face the challenge of increased computational costs due to the training and sampling of diffusion models. For example, the diffusion-planning methods [2, 6] also need to train two separate diffusion models, one to generate the trajectory data and another to predict the cumulative rewards of trajectory samples. We believe that the fast development of diffusion models in the generative modeling community can help tackle the above challenge for RL problems, such as sampling acceleration methods [7, 8]. Our method is the first step that leverages expressive diffusion models to tackle the CORL challenge, and we will continue to tackle its limitations as stated in the original paper (bottom, page 9): "Though, our method requires two diffusion models to synthesize replayed samples, which could be further improved by sampling acceleration methods or developing one diffusion model for unifying the state and behavior modeling. We leave these directions as future work."

Q2: How does the continual training cost of updating diffusion models for each new task and sampling from them trade off with the memory savings of condensing previous tasks?

A2: In the continual learning setting, the category of generative replay methods [9, 10] needs to maintain two separate models: the solver (the backbone RL algorithm) and the generator (which mimics the data distribution of past tasks). In our method, we eliminate the dependency on accessing data of past tasks at the cost of maintaining an additional diffusion-based state generator. Taking the HalfCheetah task as an example, the state generator has about 24M parameters, and each new task has about 1M samples where each transition sample ($s,a,r,s'$) has a dimension of 26. As the number of tasks increases, the number of encountered samples will continue to expand, while the model size remains constant. Training the state generator with 1M samples roughly costs 2.5 hours on an NVIDIA RTX 3090 GPU, which could be further reduced to 50 minutes via distributed training on 4 GPUs. The advantage of generative replay becomes more remarkable for scenarios where massive datasets are needed to train a task or the continual learning setting contains a large number of sequential tasks. Additionally, the generative replay method can produce pseudo data as much as needed, which can also improve the robustness and flexibility of the model.

Further, the pros of generative replay are not limited to memory savings only. In many practical applications, we cannot access real data due to privacy and security concerns, and generative replay is one of the few methods that is effective under these conditions. This challenge makes generative replay methods more necessary in real-world scenarios.

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[3] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[6] Fei Ni, et al., MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023.

[7] Cheng Lu, et al., DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.

[8] Qinsheng Zhang, et al., gDDIM: Generalized denoising diffusion implicit models, ICLR 2023.

[9] Hanul Shin, et al., Continual learning with deep generative replay, NeurIPS 2017.

[10] Rui Gao, et al., DDGR: Continual learning with deep diffusion-based generative replay, ICML 2023.

Q1: Is it the improved performance sourced from your proposed method not from the diffusion and multi-head mechanism?

A1: As stated in our experiments, our primary concern is to validate whether our diffusion-based generative replay mechanism can realize high-fidelity replay of the sample space. Hence, we set three baselines as Oracle, None, and Noise. For Oracle, the generated data is replaced with real data from past tasks to demonstrate that our diffusion-based generator can mimic the data distribution with high fidelity. For Noise, the state generator is replaced with Gaussian noise to demonstrate the effectiveness of our diffusion-based state generator. For None, the state generator is removed and catastrophic forgetting is incurred, which demonstrates the necessity of our generative replay mechanism. In summary, these observations verify the contributions of our diffusion models to the continual learning performance.

Q2: Why did you utilize the diffusion model?

A2: A1: As repeatedly stated in our original paper, we have discussed the advantages of the diffusion model and the motivation for adopting it to tackle the CORL challenge. Due to the strong distributional expressivity and the ability to fit multimodal distributions, diffusion models have been explosively developed in the RL community recently [1-5], such as using diffusion models for behavior policy modeling [1], for target policy modeling [2], and for model-based trajectory modeling [2]. In the first paragraph of Sec. 3.2 (bottom, page 4), we elaborate on the necessity of using the diffusion model to learn behavior from prior tasks as: "This perspective rooted in generative modeling presents three promising advantages for CORL. First, existing policy models are usually unimodal Gaussians with limited distributional expressivity, while collected behaviors in CORL become progressively diverse as novel datasets keep emerging. Learning a generative behavior model is considerably simpler since sampling from the behavior policy can naturally encompass a diverse range of observed behaviors, and allows the policy to inherit the distributional expressivity of diffusion models. Second, RL models are more prone to deficient generalization across diverse tasks. Learning a unified behavior model can naturally absorb novel behavior patterns, continually promoting knowledge transfer and generalization for offline RL. Third, generative behavior modeling can harness extensive offline datasets from a wide range of tasks with pretraining, serving as a foundation model to facilitate finetuning for any downstream tasks. This aligns with the paradigm of large language models, and we reserve this promising avenue for future research."

On the other hand, other models such as behavior cloning, GANs, and VAEs can also be considered as alternatives to the diffusion model. In this paper, we choose the powerful diffusion model due to its promising properties discussed above. As the state-of-the-art architecture in the generative modeling community, the diffusion model has achieved great success in high-fidelity data generation, such as its superiority over GANs [6] and stable diffusion [7].

[1] Huayu Chen, et al., Offline reinforcement learning via high-fidelity generative behavior modeling, ICLR 2023.

[2] Zhendong Wang, et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[3] Michael Janner, et al., Planning with diffusion for flexible behavior synthesis, ICML 2022.

[4] Bingyi Kang, et al., Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[5] Anurag Ajay, et al., Is Conditional Generative Modeling all you need for Decision Making? ICLR 2023.

[6] Prafulla Dhariwal et al., Diffusion models beat GANs on image synthesis, NeurIPS 2021.

[7] Robin Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.