Towards General-Purpose Model-Free Reinforcement Learning

Thank you for the review! We respond to each of your comments below.

While it is not possible to evaluate every benchmark, the Minecraft benchmark from Dreamer, which I believe tests long-term credit assignment, is notably missing. It could be great to see how this method holds up when reward is sparse, exploration is difficult, and policy gradient variance may be high. Procgen is also missing, which may be one of the harder domains in the Dreamer paper.

We agree that analyzing every benchmark provides real value in understanding the performance of an algorithm and its strengths and weaknesses. Since we can only evaluate on some subset, we chose the most common benchmarks in the RL literature as a starting point. Admittedly, as mentioned in Section 6 (discussion and conclusion), we would not expect MR.Q to perform well on sparse reward environments such as the Minecraft benchmark, as it does not consider any explicit exploration mechanism, nor does it contain any history-based component.

We have added some discussion on the limitations of MR.Q to the conclusion to expand on this (Section 6 - Limitations). Despite this, we hope that our 118 environment evaluation setup can convince you of the strength of MR.Q across a varied set of challenging domains.

The model-based representations are not particularly novel. For example, consider VariBAD (Zintgraf et al., 2020), which reconstructs reward and dynamics functions.

We agree that model-based representations are a well considered area of research, our related work attempts to cover as many closely related papers as possible but it is not surprising that some slipped past us. We have included VariBAD as a reference in the related work.

To comment on novelty: we believe that model-based representations are an important research area in RL, and therefore this direction is worth contributing to. Even if our approach is not significantly different from existing approaches, we believe that our key contribution to this area is demonstrating their effectiveness as a general tool across many benchmarks.

One piece of novelty is that the reconstruction of reward and transitions use a linear function of the same latent encoding. However, experiments show that a non-linear value reconstruction is useful, and it may be the same for the model-based objectives as well. Are there experiments testing this?

We agree that this is an interesting design choice. We have tried similar experiments during development. An interesting choice here is balancing model size, for example, additional layers to improve reward or dynamics estimation could also be spent on improving capacity elsewhere in the model. We have added an design experiment that replaces the linear MDP prediction layer with 3 MLPs (Non-linear model in Section 5.2 - Design study). We find that this variant overall performs slightly worse than the linear version, suggesting that the benefit from being approximately linear outweighs the benefit of additional capacity.

It would be great to see comparisons to other baselines here that use contrastive learning as well, e.g., Neural Predictive Belief Representations (Guo et al., 2018).

While more baselines will always provide value, considering our number of environments, we had to be selective about which baselines to use. Since our emphasis was on raw performance, we tried to include only highly competitive baselines (SOTA or near SOTA). We believe there are many new settings to tackle in future work, and contrasting against other representation learning methods may provide interesting insight or unlock new approaches in these settings, but we have to leave something for the future!

I am also concerned that the method cannot handle general POMDPs. Is it true that only the state and action are passed the policy and not history? If you are only using a latent that reconstructs the reward and dynamics, this statistic must condition on history, and is not general enough for any POMDP, but only a distribution of MDPs, as in meta-RL (Zintgraf et al., 2020).

The method is indeed intended for the MDP setting rather than POMDPs. All of the benchmarks we evaluate with only consider the MDP setting (or are approximated with short histories). The representation learning scheme can be extended to consider the history by using recurrent or transformer architectures, but again we leave this important addition to future work.

Thank you for the review! We respond to each of your comments below.

The authors argue that the benefits of model-based approaches may stem from their learned representations, rather than from their planning abilities. While this may be true to some degree, we assume that this is highly specific to the domains they conduct experiments on. In particular, Gym and DMC environments and most Atari games do not require much planning to be solved. However, we know that in other domains planning plays a critical role. Therefore, we believe it is important for the authors to clarify that the benefits of model-based representations alone depend on the domain at hand, to avoid any misconceptions.

We certainly agree. We tried to be careful with our wording (e.g., “suggesting”) in our comments in Section 6, but you are correct that this is highly dependent on the set of benchmarks. It is also worth noting that our results don’t necessarily show any certain performance downside to planning, rather we aim to highlight the effectiveness of the representation itself. We will adjust the writing to reflect the importance of the environment when considering the benefits of different methods (Section 6 - Model-based and model-free RL).

The authors refer to their method as “generalist”, which is a term commonly used for multi-task agents in the literature. However, all conducted experiments are in single task settings. Instead, “generalist” in the context of this paper refers to being robust to different environments (reward scales, dynamics, with the same hyperparameters). We therefore strongly encourage the authors to rethink this terminology to avoid confusions.

Thank you for pointing out this oversight. We have adjusted “generalist” to “general” (as used by DreamerV3) or “general-purpose” algorithm. Let us know if you think there is a better term which captures this sentiment.

The results for Dreamerv3 are surprisingly bad on continuous control tasks. This suggests that this baseline is purely tuned for these environments.

We were also surprised by the poor performance of DreamerV3 at these tasks. We use the suggested default hyperparameters for DreamerV3, and compare against DreamerV3 on environments contained in their original evaluation. Our results are consistent with the TD-MPC2 paper, which also shows the weakness of DreamerV3 in continuous control tasks. Ultimately, we believe our evaluation is fair and highlights the overwhelming challenge of developing algorithms which are effective across a diverse range of benchmarks. As far as possible reasons, we suspect DreamerV3 was over-tuned for the Atari benchmark (compared to continuous domains), again a consequence of the difficulty of maintaining a high performance across varied benchmarks.

The proposed approach comes with a large list of hyperparameters, which may limits its practical applicability.

We aim to take a transparent approach to reporting all of our hyperparameters, to provide high clarity and maximize reproducibility. Many hyperparameters do not require tuning or are based on default values defined in prior work. Regardless, our results demonstrate that it is possible for a single set of hyperparameters to work across a wide range of environments, so the number of hyperparameters may not be such an explicit downside of the method.

How do the Training FPS of MRQ. in Figure 1 compare to other non-planning baselines on DMC-Visual (TD7) and Atari (Rainbow, DQN)?

MR.Q is more expensive than TD7 and Rainbow, due to the larger model size (for TD7) and the additional training of the model (for Rainbow). These costs can often depend on the implementation details, but we estimate MR.Q runs at about half the speed.

Thank you for the review! We respond to each of your comments below.

Unfortunately, the paper does the opposite and provides no detail (that I could find) regarding the hyperparameter settings of the baseline algorithms, nor how or if they were tuned. This is a critical weakness of the paper, as the possibility of untuned baselines undermines the claimed performance improvements.

Were the baselines tuned, and how was this done if so?

Do you have evidence that A) MR.Q is less hyperparameter sensitive than baselines, or B) the best single set of hyperparameters for MR.Q achieves higher performance than the equivalent for a baseline?

Thank you for highlighting this oversight, as it is important to state clearly how the baseline hyperparameters were chosen. In our work, we stick to author-suggested defaults without tuning and keep them fixed across all benchmarks. We will add clarification on this point in the paper to avoid any ambiguity (Section 5 and Appendix B.4).

This is consistent with the claims of TD-MPC2 and DreamerV3, which both use a fixed single set of hyperparameters across their respective benchmarks. Using new hyperparameters would contradict the claims made by the authors, and we believe that our setup is as fair as possible, given that it matches the authors’ original experimental setup, and we mainly test on environments included in their respective papers.

To be specific:

• For TD-MPC2, we use the hyperparameters used by the authors for DMC and DMC-visual. For Gym we use the episodic branch of the authors’ repo (as discussed in Appendix B.4), which is intended for the Gym environments.
• For DreamerV3, we use the hyperparameters tuned by the authors’ for DMC, DMC-visual, and the author’s results for Atari. For Gym we use the same setup as DMC.
• For all the other baselines, we use the specific hyperparameters designated for their respective benchmark. The sole exception is TD7, which we keep constant between Gym and DMC.

Even if we omit the benchmarks that TD-MPC2 and DreamerV3 do not tune for (Gym), we believe that our results demonstrate that MR.Q’s best single set of hyperparameters is competitive or outperforms the best single set of hyperparameters of TD-MPC2 and DreamerV3 with lower computational overhead.

Another example is that the paper seems to maintain model sizes from the original implementations of each algorithm. However, the size of the model is not defined by the algorithm, so it would be much more convincing to normalize the parameter count between algorithms and show that MR.Q achieves superior performance at the same size, or has a better scaling curve.

We believe that architecture falls under the single set of hyperparameters of the baseline models, and as such we do not modify them. To your point however, MR.Q already uses a smaller model size than both TD-MPC2 and DreamerV3 across all of our 118 environments. We believe that our results demonstrate MR.Q’s superior parameter efficiency than the baselines.

Some key algorithms are missing, most notably PPO. It is not expected that the authors compare to every popular RL algorithm, but PPO is undoubtedly the closest thing to a "general-purpose" RL algorithm in current literature.* A number of recent methods have also claimed to serve as "general-purpose" RL algorithms (e.g. PQN, Gallici et al., 2025). These would make for valuable comparisons in future work but are not required for acceptance.

We agree that PPO is a highly general-purpose algorithm. We chose to exclude PPO from our experimental results because it has not been shown to be a competitive baseline in any of the benchmarks we considered. Our primary objective was to evaluate MR.Q against well-established near-SOTA baselines to demonstrate the effectiveness of MR.Q. The DreamerV3 paper demonstrates PPO underperforms DreamerV3 in DMC, DMC-visual, and Atari. TD7 is shown to outperform TD3 in Gym. TD3 in turn, was demonstrated to outperform PPO in the original TD3 paper.

However, we acknowledge that demonstrating an explicit performance gap by showing PPO results would strengthen the paper. So we have included PPO in our results.

PQN (Gallici et al., 2025) is an interesting comparison, but unfortunately, it appears to be a concurrent submission to this ICLR, which explains why we were not aware of it (but thank you for highlighting it regardless). PQN also does not extend naively to continuous actions, which make up three out of our four benchmarks. However, this does highlight the community’s interest in powerful general-purpose algorithms.

Thank you for the review! We respond to each of your comments below.

What exactly is the objective of the model-based elements mentioned by the authors? It seems that all techniques described are already present in existing model-free reinforcement learning frameworks.

We outline the purpose and benefits of a model-based representation in Section 4, line 171-177. To summarize, the model-based objective provides a richer learning signal than value-based learning, which may help learning stability or generalization and also helps abstract away uninformative information in the original observation space. The actual learning objective is described in Equations 10 and 14.

While it is true that some of the techniques we describe are inspired by existing model-free algorithms, we demonstrate that the correct combination of these concepts leads to a powerful final algorithm. Let us know if we misunderstood your question, we are happy to clarify further.

The authors only conducted experiments on the Gym, DMC, and Atari benchmarks, which are classic but relatively homogeneous environments. Claiming generality might be overstated given these limited evaluation. For example, the widely accepted generalist algorithm PPO has demonstrated its effectiveness across a broad range of tasks. To establish the generality of this method, additional testing on tasks with some heterogeneity from these benchmarks, such as fine-tuning on large models, would strengthen the claim.

We acknowledge that extending our results to some unique settings, such as fine-tuning large models, would offer some concrete additional evidence at the generality of our method. We have added some remarks about this to the limitations section in the conclusion (Section 6 - limitations).

We have chosen to focus on common RL benchmarks as they allow us to leverage well established baselines and experimental protocol. We do not claim that MR.Q is a fully general-purpose algorithm–our goal is to make meaningful progress towards developing more versatile methods, and we believe that our current results represent an important step in this direction.

The theoretical motivation presented by the authors is somewhat confusing and challenging to interpret. Please refer to Question 3 for details.

Equation (7) seems to be derivable directly from Equation (3). Could you clarify the phrase "from this insight"?

Apologies for the confusion. The insight is Theorem 1, Equation 7 is a definition. To be explicit: From this insight (Theorem 1), we connect the definition of value error (Equation 7) to the accuracy of the estimated reward and dynamics (shown in Theorem 2). We have re-written this sentence to eliminate the confusion.

We hope that clarifies any confusion in the theoretical results, but if there are any unclear points, we would love to engage in further discussion to improve the clarity of the paper further.

The authors describe the linear decomposition of the value function as the motivation, yet the final algorithm design and ablation studies prefer using common nonlinear function approximations. The authors claim that Equations (5-6) are inspired by model-based representations, but related ideas have already appeared in existing model-free algorithms. While this work is innovative and effective from an algorithmic design perspective, the theoretical sections seem somewhat retrofitted to fit the empirical results.

While we agree that our key results are empirical, our theoretical work provides important guidance for designing the algorithm. The existing literature offers many similar ideas, making it hard to choose the best approach. Our theory helps identify which design choices (and approximations) are important. As an additional design experiment, we have included results (in Section 5.2) where we replace the linear model with three non-linear MLPs. Although this increases capacity, we find it slightly harms performance. This helps justify that a linear model is a helpful design choice.