Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning

How do you ensure synthetic data doesn’t introduce bias into the critic’s learning process? Are there methods to filter or correct this data?

We performed sweeps over the synthetic-to-real data ratio (Figure 12) and found that introducing any amount of synthetic data into off-policy learning in DMC leads to a substantial degradation in policy performance. Moreover, in our Dyna-style experiments using perfect synthetic data (i.e., no model error), we showed in Figure 6 that learning still fails to outperform the base off-policy methods on which these algorithms are built. These findings suggest that the failure is not due to model inaccuracies or data quality, but rather due to algorithmic issues intrinsic to how synthetic data is integrated.

While methods such as those proposed in [1,2] attempt to filter or adapt synthetic rollouts, our results imply that such corrections are insufficient to fix the core algorithmic shortcomings. Even with perfect data, the performance gap remains, underscoring the limits of current Dyna-style designs.

Addressing this challenge remains an active area of research for us, and we welcome further discussion on promising directions for improving synthetic data integration in off-policy learning.

Besides periodic resets, have you considered other methods to address plasticity loss, such as regularization or dynamic architectures? Yes. While we did not frame them explicitly in terms of plasticity, we explored several techniques targeting related issues. Layer normalization was employed in prior work [3] as a method that simultaneously mitigates overestimation and stabilizes learning dynamics - both of which relate to plasticity loss. Additionally, we experimented with unit ball normalization [4], which was proposed to combat overestimation and thereby also helps address plasticity indirectly. We did not include these results in the main text as they did not substantially outperform the more commonly used layer norm baseline.

We did not pursue further architectural or regularization strategies because our literature review found no alternatives that clearly outperform these methods. Periodic resetting, layer norm, and unit ball normalization appear to represent the state-of-the-art approaches for maintaining plasticity in high-update-ratio settings. If we have overlooked a promising direction, we would sincerely appreciate any further insight from the reviewer.

Have you tested these algorithms on other benchmarks or real-world applications to see if the performance gap persists? We did not. Our motivation was to critically re-examine claims of sample efficiency made by prior dyna-style methods (e.g., ALM, MBPO), which were accepted largely based on Gym benchmarks. These methods were widely adopted under the assumption that they outperformed their model-free counterparts in sample efficiency. Our work revisits that conclusion using a comparable benchmark - DeepMind Control Suite (DMC) - which shares the same MuJoCo physics backend.

This choice minimizes the likelihood that differences in performance are due to changes in dynamics simulation, and therefore emphasizes the impact of the Dyna-based algorithmic changes introduced in ALM and MBPO. Our results demonstrate that the original claims made about those methods do not generalize even within this extremely similar experimental setting. While evaluating additional domains might provide further insight into generalization challenges, we chose to focus on a clean and controlled comparison to highlight foundational flaws in current Dyna-style designs. Introducing more benchmarks could risk diluting the clarity of this central message.

Citations:

[1] B. Frauenknecht, A. Eisele, D. Subhasish, F. Solowjow, and S. Trimpe, “Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption,” Jun. 21, 2024, arXiv: arXiv:2405.19014. doi: 10.48550/arXiv.2405.19014.

[2] Y. Li, Z. Dong, E. Luo, Y. Wu, S. Wu, and S. Han, “When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter,” Oct. 16, 2024, arXiv: arXiv:2410.12160. doi: 10.48550/arXiv.2410.12160.

[3] C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney, “Understanding plasticity in neural networks,” Nov. 27, 2023, arXiv: arXiv:2303.01486. Accessed: Jun. 29, 2024. [Online]. Available: http://arxiv.org/abs/2303.01486

[4] M. Hussing, C. Voelcker, I. Gilitschenski, A. Farahmand, and E. Eaton, “Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence,” Mar. 09, 2024, arXiv: arXiv:2403.05996. Accessed: Apr. 08, 2024. [Online]. Available: http://arxiv.org/abs/2403.05996

We thank the reviewer for their helpful suggestions to improve the clarity and presentation of our paper. We will incorporate the suggested changes to:

• More precisely describe the subset of OpenAI Gym environments used in our experiments.
• Cite ALM upon first mention and reduce repeated citations later on to avoid redundancy.

What would be the result of Figure 12 if the model is perfect? This is an excellent question and one we are also curious about. As depicted in Figure 6, we generated a single data point for both humanoid-stand and hopper-stand at the 0.05 synthetic ratio using perfect transitions (i.e., querying the ground truth environment). This process required approximately 4 GPU-days and substantial CPU usage, due to the environment transitions becoming the bottleneck. Unlike synthetic rollouts from learned models (which are efficiently parallelizable via JAX), real environment rollouts suffer from parallelism limitations that drastically slow down runtime.

Extrapolating from our timing, generating the full version of Figure 12 with perfect transitions would require roughly 40 GPU-days across multiple machines. Given the resource cost and limited additional insight expected beyond what is already shown in the paper, we opted not to pursue the full experiment at this time. However, if we succeed in optimizing environment rollouts via vectorized environments in the future, we agree this would be a worthwhile avenue to explore.

That said, our preliminary results suggest that even with perfect transitions, performance does not recover to the level of pure real-data training, regardless of synthetic-to-real ratio. This aligns with our central claim that the degradation arises not solely from model bias but from algorithmic limitations in how synthetic data is incorporated.

For MBPO/ALM on OpenAI Gym that surpass SAC (or methods like DreamerV3/EfficientZeroV2 on DMC), do you also observe the underestimation issue? Thank you for this insightful question. We interpret it as: "Do value underestimation issues persist in model-based methods when they outperform SAC in Gym or state-of-the-art model-free methods in DMC?" Please let us know if we have misunderstood.

In our experiments, we did not observe value underestimation in ALM or MBPO when evaluated on OpenAI Gym. This suggests that underestimation is largely a DMC-specific phenomenon in our setting. As discussed in the paper, even with a perfect model, underestimation is much less pronounced in DMC compared to a learned model, implicating model error as a contributor to Dyna failing, but not the only issue. However, Section 4.2 further shows that MBPO still fails to match SAC in DMC even when underestimation is not prominent, indicating that underestimation, while influential, is not the sole factor - nor a "smoking gun" - behind the performance gap.

For the left column of line 425 - could you clarify the update count after model reset? Certainly. During our investigation of model plasticity, we explored multiple reset strategies inspired by prior work [1]. We found that increasing the reset frequency up to a point improved performance. The strategy we ultimately adopted - resetting every 20k environment steps - aligns with the reset frequency recommended in [1] for a replay ratio of 128. Empirically, this approach worked better in our setting than the more conservative reset frequency from [1] prescribed for a replay ratio of 20 (every 2.56 × 10⁶ gradient steps, equivalent to ~128k env steps), or intermediate strategies between the two.

We will clarify this detail in the revision to aid reproducibility and interpretability.

Citations: [1] P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville, “Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier,” 2023.

On comparisons with other model-based RL algorithms: We appreciate the reviewer’s suggestion regarding DreamerV3, PlaNet, and PETS. Our study focuses on Dyna-style algorithms MBPO and ALM, which use synthetic rollouts in proprioceptive, state-based settings. Pixel-based methods such as DreamerV1-V3 and PlaNet abstract away low-level issues central to our study. From the perspective of pixels, two hopper environments that move and look similar are effectively indistinguishable, whether simulated in Gym or DMC. In contrast, state-based methods must engage with environment-specific dynamics and state representations, which our results show can vary significantly despite a shared MuJoCo backend. For this reason, pixel-based approaches fall outside the scope of our analysis.

DreamerV3, although impressive on proprioceptive tasks in addition to pixel-based, significantly differs from simpler Dyna-style methods in complexity and design philosophy. Understanding its success relative to MBPO and ALM, as noted in Section 2.1 and our FAQ, is valuable but warrants a dedicated analysis. Additionally, DreamerV3 has not been evaluated on OpenAI Gym tasks, complicating comparisons due to benchmarking inconsistency (as discussed in our paper).

Similarly, PETS and TD-MPC diverge from the Dyna paradigm by employing learned models for online trajectory optimization (e.g., model predictive control), rather than using synthetic rollouts for policy/value training. Our work targets hallucinated experience in Dyna-style pipelines.

Ultimately, while broader comparisons are important, evaluating methods that differ in observation modality or depart from the Dyna structure falls outside our intended scope. Our goal is to identify previously overlooked failure modes in a prominent subclass of model-based RL - Dyna-style methods - under realistic, proprioceptive conditions.

On the possibility that more sophisticated models could close the performance gap: We thank the reviewer for the insightful question regarding whether more sophisticated model architectures or training techniques (e.g., meta-learning) could close the performance gap. While such advances may indeed improve results, they would only reinforce our central claim: standard Dyna-style methods like MBPO and ALM exhibit fundamental limitations when used “out of the box,” even with extensive tuning. These methods were originally presented as simple, sample-efficient augmentations to off-policy RL, relying on short synthetic rollouts and standard neural network dynamics models. Yet across 15 diverse DMC environments, they consistently underperform their base off-policy RL learners - and sometimes even random policies - revealing deeper algorithmic issues.

On environmental differences between DMC and Gym: We appreciate the reviewer’s point that differences in reward structures, termination conditions, and dynamics between OpenAI Gym and DMC may influence performance. While we address these factors (Section 2.2, Appendix C), our analysis in the paper is deliberately algorithm-focused. We show that the base algorithm, SAC, performs well in both testbeds, while its Dyna-style counterpart (e.g., MBPO) fails in DMC. This contrast isolates synthetic rollouts - not environment differences - as the primary source of degradation, challenging the assumption that Dyna-style methods reliably enhance strong off-policy learners across domains. We agree that further work should explore how environment-side factors interact with Dyna-style methods. These are important questions, but beyond the current paper’s scope.

On hyperparameter sensitivity: As described at the end of Section 4.1, we performed extensive hyperparameter sweeps on key environments (e.g., hopper-stand, humanoid-stand), varying model size, retraining frequency, learning rates, and gradient steps. While these reduced model error in some cases, they did not produce consistent or meaningful gains in policy performance beyond a random policy in many cases. Sweeps over the off-policy algorithm's hyperparameters yielded similarly negligible effects. This insensitivity supports our central claim: the failure of Dyna-style methods arises from fundamental challenges in leveraging synthetic rollouts, not suboptimal hyperparameter choices. We therefore did not emphasize sensitivity analyses, as tuning alone does not address the core limitations of these approaches. We are happy to include additional details in the camera-ready manuscript.

Clarification on the “percent model error” metric: We will revise the text to include an explicit formula and explanation of our percent model error metric. Specifically, let $\hat{y}$ denote the model's prediction (next observation and reward) and let $y$ be the corresponding ground-truth target. We define:

Reproducibility:

See response to Reviewer gMX1.

Thank you for your thoughtful and positive review. We appreciate your recognition of our contributions and the strengths of our experimental design and analysis.

Regarding your mention of DreamerV3, we agree that its success in DMC highlights the diversity of model-based RL approaches. For further detail, we refer the reviewer to the section titled "On comparisons with other model-based RL algorithms" in our response to Reviewer TyLB, clarifying why DreamerV3 falls outside the scope of our evaluation of Dyna-style methods.

Regarding the code, we consider the code an integral part of our contributions and are actively refining it. If the paper is accepted, we will certainly release the code alongside the camera-ready version.