Prioritized Generative Replay

Thank you very much for your review.

Q1: “How robust is PGR to errors in the learned dynamics model? Are there ways to mitigate the impact of inaccurate dynamics predictions on the curiosity-based relevance function?”

A1: Like most works in model-based RL, PGR is indeed susceptible to errors in the learned dynamics model of the curiosity-based relevance function. However, in practice, we show that PGR is far more robust than other approaches, especially those which directly plan through the dynamics model.

Specifically, we perform an experiment comparing MAX[1] and Dreamer-v3[4] to PGR when the learned dynamics are misled by noisy observations. We inject Gaussian noise into some fraction of the observations in transitions fed to the world model in Dreamer-v3, as well as the observations in transitions fed to the exploration model ensemble in MAX. We similarly pollute the observations in transitions to the ICM module in PGR. For more details, we refer the reviewer to Appendix C. We reproduce results here for ease of viewing:

As we observe, PGR continues to outperform MAX and Dreamer-v3 under these noisier conditions. This is because Dreamer must learn the joint distribution over $p(s, a, s’, r)$ entirely, and thus any partial model errors propagate throughout generations, and also affect policy learning. In general, (with or without intrinsic reward,) planning directly through a world model with inaccurate dynamics will naturally lead to poor performance. Contrast this with PGR, which more loosely couples the learning of environment dynamics (via intrinsic curiosity) from the rest of the generative world model (diffusion over $(s, a, s’, r)$ tuples). Concretely, PGR learns the dynamics $p(s’ | s, a)$, and conditions on dynamics error $e$ to generate $(s, a, s’, r | e)$. This means that even with an imperfect dynamics model, we may obtain a poor signal through $e$, but the generated transitions $(s, a, s’, r)$ still remain faithful to the ground truth environment dynamics! Furthermore, densification means PGR also grounds synthetic data with real data, allowing the policy to be more robust to any erroneous synthetic generations.

More generally, one way to mitigate the impact of inaccurate dynamics predictions for PGR is to leverage a more robust relevance function $\mathcal{F}$. For example, one could use a variational approach to intrinsic rewards such as VIME [4], with better calibrated uncertainty, to either a) directly do conditional generation of more robust transitions, or b) to downweigh generated samples when the dynamics are highly uncertain.

Q2: “PGR scales better at r=0.75 than SynthER but neither benefits from 0.875. We would think the trend would be consistent? What is the intuition behind this?”

A2: There is nothing better than ground truth data directly obtained in the “real world”. While generative replay is a useful mechanism to densify past real-world online experience, further work is needed to completely replace real environment interactions with synthetic ones.

We show in Figure 5 of the main text that our diffusion model is a reasonable approximation to the simulator dynamics, but this MSE is still non-zero. When the synthetic data ratio is too high, it is plausible that the model begins fitting to the biased distribution of the diffusion model. Thus, overly relying on synthetic data may result in policy collapse.

Note that we did not tune the hyperparameter controlling the frequency of the inner loop when varying the synthetic data ratio $r$. We hypothesize that 10K iterations per inner loop might no longer be optimal for higher synthetic data ratios, since more regularly grounding the diffusion model in the real simulator dynamics becomes more critical if we rely more heavily on the generated data for policy learning.

Dear reviewer 8yrp,

Thank you for the constructive feedback and suggestions. We greatly appreciate you updating your score. If there are any additional clarifications we can provide, please let us know. Thank you once again.

Thank you very much for your review.

Q1: “While the method shows improved performance, it is a bit simple as it combines existing elements in diffusion models and RL to propose the solution.”

A1: To the best of our knowledge, our work is the first to cast widely-used prioritized replay in online RL through the lens of conditional generation. Moreover, we provide a novel and central role to curiosity, as a useful prioritization signal during conditional data generation. This is a novel use case well outside its usual stage in RL methods, which have historically only leveraged it as intrinsic reward for better exploration.

The novelty of our approach does not lie in using a diffusion model (conditional or unconditional) to capture the replay buffer. As we emphasize through lines 410-432 in Section 5.2 of the main text, knowing what to condition on is actually more important than choosing to do conditioning (or generation at all) in the first place. Moreover, why and how conditioning works are also equally valuable questions to answer, and we provide generalizable and non-obvious insights into both these elements.

Q2: “Is the method compatible with different kinds of exploration bonuses? If so, how do you think they would compare?”

A2: We show in our general response that the contributions in PGR go beyond explicit exploration bonuses such as intrinsic curiosity or RND [1], or methods which implicitly promote exploration such as bootstrapped DQN [4] or NoisyNets [5].

These latter methods are readily compatible with PGR, requiring only light modifications to the underlying baseline REDQ architecture. We reproduce the results in this comparison below (experimental details can be found in Appendix B):

Our results indicate densification via PGR is orthogonal to and more useful than promoting exploration alone. For a more comprehensive discussion, we direct the reviewer to our general response above.

Q3: “How do you think the method would do when simply having diverse samples does not imply usefulness? An example is the noisy tv problem.”

A3: We also show in our general response that PGR can effectively condition on relevance functions based on RND [1], pseudo-counts [2] or episodic curiosity [3], which all demonstrated reliably evading stochastic noise sources in prior work.

The relevant discussion can be found under the first question in the above general response, with experimental details in Appendix A. We also direct the reviewer to our response to reviewer 1k48. Specifically, we argue that the search space for desirable relevance functions $\mathcal{F}$ is not that large, and that prediction error-based intrinsic curiosity (ICM) serves as a good default choice. In the event that ICM is insufficient due to stochastic noise sources such as noisy TVs, this problem has been shown in [1] and [3] to be easy to diagnose, and we demonstrate in our general response above that swapping in a more appropriate $\mathcal{F}$ restores the benefits of PGR.

Finally, we remark that we do not generate diverse samples because they imply usefulness. Rather, our first-order goal is to generate samples that are relevant and useful (e.g. reduce early overfitting of the Q-function.) And as a possible mechanistic explanation for this property, we empirically show that these samples are actually more diverse than those generated via the unconditional baseline.

Thank you very much for your review.

Q1: “Discussing different ways to approach generative replay such as mentions of other generative models (e.g. variational auto encoders, gaussian mixture models) and why they were not used.”

A1: We believe that VAEs and GANs have issues which make comparison to baselines harder and would only obfuscate our contribution. In particular, GANs are sensitive to hyperparameters and much harder to train [1]. VAEs do not have the generation quality that is required to achieve comparable results to methods like SynthER. A baseline level of generation quality is especially important in the online setting, when initial data is noisy.

To verify this, we replace the underlying diffusion model in SynthER and PGR with an unconditional and conditional VAE, respectively. We design our VAE to approximate the capacity of the diffusion model in both SynthER and PGR (~6.8 million parameters). Specifically, we use a ResNet-style encoder and decoder with 4 and 8 layers, respectively. Each residual layer has bottleneck dimension 128, with a latent dimension of 32, resulting in 2.6 million encoder and 5.3 million decoder parameters. Note that this model architecture is also highly similar to the residual MLP denoiser used in the diffusion models of SynthER and PGR. Results on three tasks (state-based DMC-100k) are shown in Appendix D, with results reproduced below for ease of viewing:

We conclude that a) indeed conditional generation continues to outperform unconditional generation, but b) overall performance is much lower than either SynthER or PGR, arguably performing within variance of the REDQ baseline. We believe this performance gap well justifies the focused use of diffusion models.

Q2: “Formatting of Tables 1, 2, and 3.”

A2: We have reformatted the paper to better accommodate Tables 1, 2 and 3. Thank you for the detailed comment!

Q3: “What exploration method does the agent use? Could the exploration method be improved instead of the sample generation to improve diversity of samples? Would a combination of both a better exploration and this method be the optimal and a possible solution?”

A3: PGR does not rely on any particular exploration method. Indeed, better exploration can be orthogonally combined with PGR as a plausible approach.

Note that while better exploration alone may improve the diversity of samples, we show in Appendix B that the densification offered via generative replay is independently and critically important to PGR’s improved policy learning results.

As we illustrate in our general response above, PGR offers distinct advantages over explicit exploration bonuses, and potentially can be combined with implicit exploration bonuses. While we do not look at exploration in this work, we believe that these initial findings have now opened up an interesting intersection between generative replay and exploration for future works in online RL.

[1] Salimans et al. “Improved Techniques for Training GANs.” arXiv 2016.

Thank you very much for your review.

Q1: How is this method novel with respect to prior work that uses intrinsic rewards on rollouts from a learned dynamics model?

A1: The PGR paradigm enables 1) generative densification of online experience and 2) conditional guidance via relevance functions. We show that each of these benefits uniquely positions our work amongst prior work.

Firstly, we note that densification is different from planning through the generative model entirely, as in e.g Dreamer [2][4]. In particular, Dreamer analytically backprops the gradients of dynamics through entire trajectories “generated” in latent space. This means that its generative world model is the bottleneck for learning. We show that in the presence of noisy/imperfect observations, approaches akin to this are less desirable than PGR.

Specifically, we perform an experiment comparing MAX[1] and Dreamer-v3[4] to PGR when the learned dynamics are misled by noisy observations. We inject Gaussian noise into some fraction of the observations in transitions fed to the world model in Dreamer-v3, as well as the observations in transitions fed to the exploration model ensemble in MAX. We similarly pollute the observations in transitions to the ICM module in PGR. For more details, we refer the reviewer to Appendix C. We reproduce results here for ease of viewing:

As we observe, PGR continues to outperform MAX and Dreamer-v3 under these noisier conditions. This is because Dreamer must learn the joint distribution over $p(s, a, s’, r)$ entirely, and thus any partial model errors propagate throughout generations, and also affect policy learning. In general, (with or without intrinsic reward,) planning directly through a world model with inaccurate dynamics will naturally lead to poor performance. Contrast this with PGR, which more loosely couples the learning of environment dynamics (via intrinsic curiosity) from the rest of the generative world model (diffusion over $(s, a, s’, r)$ tuples). Concretely, PGR learns the dynamics $p(s’ | s, a)$, and conditions on dynamics error $e$ to generate $(s, a, s’, r | e)$. This means that even with an imperfect dynamics model, we may obtain a poor signal through $e$, but the generated transitions $(s, a, s’, r)$ still remain faithful to the ground truth environment dynamics! Furthermore, densification means PGR also grounds synthetic data with real data, allowing the policy to be more robust to any erroneous synthetic generations.

Secondly, we are not concerned with intrinsic rewards for better exploration as in [1], which is often the use case for combining intrinsic rewards and model-based RL algorithms [1][5][6]. Using intrinsic reward to guide exploration is fundamentally different from using relevance functions to guide generation. In fact, PGR is not attached to using intrinsic rewards, at all. Our paradigm centers on guidance for generative replay, meaning we can condition on any relevance function with desired properties. For example, we may choose $\mathcal{F}$ to explicitly avoid undesirable degenerate solutions. Concretely, we may have physics-based models which inform us apriori which $(s, a, s’, r)$ transition tuples are more likely to lead to unstable gaits for locomotion tasks, and penalize generating such transitions through an appropriate negative condition. Or, we might want to avoid the boundaries of a workspace and therefore decrease generations of $(s, a, s’, r)$ tuples so that $(s + a)$ is outside some epsilon bubble of the edges of the workspace.

Due to time constraints we do not provide an empirical verification of these experiments. However, we believe that by showing an instantiation of PGR with ICM, and empirically grounding why and how it is superior to other simpler choices, our work opens up these promising new avenues for future research in online RL.

In the meantime, we have softened the language in the paper which might erroneously provide the impression that ICM is the ideal, optimal or only choice for relevance functions.

We thank the reviewer for their timely engagement during the discussion period and efforts to improve our work. We have now directly linked our additional results in the appendix to our experimental section (Section 5.1). We continue the interesting discussion on the reviewer's points below:

Follow-Up to Noisy-TVs and Relevance Functions

Indeed, we acknowledge that Table 5 demonstrates that different relevance functions can lead to different outcomes. Choosing the right relevance function for a particular task or environment is thus an important framework-level hyperparameter.

Our perspective remains that this choice is relatively simple and less heuristic than many other choices commonly made in RL frameworks. We believe the conclusions from Table 5 of Appendix A should be that:

1. ICM is a good default choice for $\mathcal{F}$ (PGR (Curiosity) with no hyperparameter tuning outperforms the PPO baseline)

2. ICM failing is a fundamental problem in exploration, endemic to many prediction-error based metrics. This problem is reasonably easy to visually diagnose (inspection of behavior learned by the PPO + Curiosity agent, as documented by Savinov et al. [3]) and quantitatively diagnose (performance of PGR (RND) also suffers).

3. Choosing other more robust relevance functions such as ECO restores the advantages of PGR, allowing it to outperform the baseline that directly adds an ECO exploration bonus to model-free PPO.

Finally, we buttress our claim that $\mathcal{F}$ can be chosen to inform us apriori of transition tuples that are physically more/less desirable, and promote/penalize generating such transitions through appropriate conditioning. We point out that non-linear MPC-based controllers offer exactly the physically-grounded relevance functions we describe. Concretely, consider the task of quadruped locomotion. Then, the cost functions described in the optimal control problems formulated by Equations 9 or 10 of Corberes et al. [10] offer a concrete way to "score" motion trajectories. Specifically, during online learning of a neural network-based controller, given a particular state (e.g. proprioceptive measurements such as center-of-mass linear/angular velocity/acceleration) and control parameter (e.g. ground reaction forces), we can obtain a "cost" penalization term via a relevance function adopting the form of Equation 9 or 10 of Corberes et al. [10]. This is exactly the scalar conditioning term our diffusion model will accept. We can use a similar prompting strategy as described in Section 4.3 of our main text to densify the trajectories which exhibit the lowest $p\%$ of "cost," for some hyperparameter $p$. This line of reasoning extends to other domains (e.g. grasping/manipulation), where constraints on forces/dynamics are well understood from a model predictive control perspective (e.g. force closures and grasping kinematics in [11]).

Thus, PGR can also condition on relevance functions outside traditional measures of priority in PER literature or exploration bonuses in exploration literature. Again, in the interest of time, we cannot replicate the experimental setups of either [10] or [11], and so ascertaining the utility of PGR under these specific settings is not done. However, we believe it is clear that such formulations are an interesting nascent area of future work that PGR sets the foundation for.

[10] Corbères et al. Comparison of predictive controllers for locomotion and balance recovery of quadruped robots. ICRA 2021.

[11] Gold et al. Model predictive interaction control for force closure grasping. CDC 2021.

2. Contextualizing Exploration Bonuses Within the PGR Framework (Reviewers 1k48, Yyu8, ir9Q)

Our contributions are orthogonal to exploration. We do not make any claims about improved exploration, nor is this a focal point of our problem setting. Nonetheless, we show here that existing work in enhancing exploration can be easily and effectively integrated into the PGR framework.

We first direct the reviewers to Figure 3b of the main text, which offers initial evidence that PGR goes beyond simple exploration bonuses. In particular, providing either the unconditional replay baseline (SynthER) or the model-free RL baseline (REDQ) with an exploration bonus, via intrinsic curiosity, underperforms PGR. We provide quantitative numbers in Table 6 of Appendix B, along with a repeat of this experiment using RND [1]. We reproduce the results here:

For further experimental analysis, we also add two baselines which modify neural network architectural inductive priors to implicitly improve exploration. In particular, we adapt both Bootstrapped-DQN [4] and NoisyNets [5] to our REDQ baseline. For training details and results we refer reviewers to Appendix B (c.f. Table 7). We reproduce the results here:

We argue that densification under PGR is empirically orthogonal to and more useful than simply promoting exploration.

We provide some simple intuition on this conclusion, which also further distinguishes our exploration-agnostic contributions: PGR densifies the relevant subset of existing data to improve learning dynamics, whereas exploration is primarily concerned with improving the distribution of this data in the first place. We do not make this conclusion in the main text, as DMC/OpenAI Gym benchmarks may be too facile for exploration-centric conclusions. Nevertheless, taken together, the comparative experiments above strongly suggest that generative replay in PGR is effective and orthogonal to exploration bonuses.

[1] Burda et al. “Exploration by Random Network Distillation.” arXiv 2018.

[2] Bellemare et al. “Unifying Count-Based Exploration and Intrinsic Motivation.” NeurIPS 2016.

[3] Savinov et al. “Episodic Curiosity Through Reachability.” arXiv 2018.

[4] Osband et al. “Deep Exploration via Bootstrapped DQN.” NeurIPS 2016.

[5] Fortunato et al. “Noisy Networks for Exploration.” ICLR 2018.