Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies

My principle concern with this work, and what limits me from increasing my score, focuses on its significance; I think there are interesting questions about if this is potentially misguided, which I get into below, but overall my main assessment is that a lot of these findings are quite trivial. For instance, it is well known that training a world model on purely expert trajectories will lead to subpar policies due to overestimation bias in OOD states, and in that situation an imitation learning approach may be more expected. Similar, it should not be a surprise that introducing additional data to the agent via online sampling will lead to better performance.

Below, I cover more specific feedback. That said, I think the researchers behind this work are asking an important question and have clearly put a lot of effort into this project. I am keen to discuss this with the authors in more depth as, while I hold strong conviction after reading the paper, I am open to being persuaded that I have misjudged the work. I apologise that the review has ended up longer than expected, though I think many of the points have a degree of overlap and thus am happy for the authors to respond more holistically to the points I raise rather than going through bullet point by bullet point.

• I found lines 19-23 in the abstract very confusing. I think part of my view on the misguidance in this work is the flippant use of 'offline' and 'online' to refer to data collection. For instance, in the abstract it says the below, which seems to suggest that the problem with offline learning is that it collects data online:

We identify a key challenge behind performance degradation of offline agents: encountering Out-of-Distribution states at test time. This issue arises because the data collected online primarily benefits the online agent by learning from its own mistakes, but it leaves many states unvisited. As a result, the offline agent suffers from insufficient coverage of the state space.

• Another point I feel is not addressed is that an offline dataset is generally colelcted prior to policy learning and, as such, is not designed. The paper focuses on finding better ways to shape offline datasets (eg line 28 in the abstract, adding exploration data), ignoring the fact that an offline dataset is presumably pre-collected and there is not ready access to the online environment to collect more data.
• I think it is misinformed to say current efforts predominantly focus on expert data (line 29) when arguably the most common offline benchmark, D4RL, has random, medium and mixed policy datasets in addition to their robotics data.
• I think there are some missing reference examples; most notably, World models by Ha & Schmidhuber is a seminal MBRL paper. I think a relevant recent paper is The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning, Sims et al. (2024).
• I am a bit unclear on Figure 1. We are focused on model-based RL here, but there seems to be no demonstration of the world model in figure 1 - just an agent.
• The explanation of the active agent is very unclear.
• Equation 5 introduces a policy input reconstruction loss, but this does not seem to be used elsewhere in the paper.
• On line 217, it is worth pointing out that Dreamer V3 has the online data collection discussed here. I am not quite sure what the argument is.
• I am unsure what is means in line 234 onwards:

Their mistakes are catastrophic, as their world models and thus their policies have their unique fallacies that cannot be corrected by the task-oriented data with limited state space coverage, which is collected by and catered to the Active agent.

• There is a lot of discussion about how the passive agent experiences worse performance than the active agent in section 3.4, but again this seems quite trivial; the agent is not able to interact with the real-world so effectively does not receive any online feedback about its overestimation bias. Just because it is trained on the same data as the active agent, it is obvious that this lack of correction would harm performance; the reason many don't sample online during training isn't because they think it won't help, but because it is impossible or impractical. -I'm a bit unsure about what is meant by line 300. Again, there is little insight in the fact that the replay buffer trajectories will have a lower reconstruction loss since they make up part of the training signal of the model; this is equivalent to having a higher supervised test loss compared to training.

However, in our observation, the quality of imagined trajectories for all agents is poor with mean world model loss being more than 5 to 10 times higher compared to trajectories in the replay buffer due to accumulated error in open-loop multi-step predictions, as shown in Appendix A.4

• When considering potential mitigations, I again bring up what I believe to be misguidance in this paper. For instance, I think assuming we are rolling out a policy for collecting offline data to do offline RL with (4.1) can be potentially misguided - in this case we could just do online RL. Instead, I think an assumption in offline RL is that there is a precollected offline dataset that we then have to work with. Similarly, I think the fact that an exploration only dataset underperforms is expected, but I think this hinges on the fact that the dataset size is small.
• In section 4.2, it is suggested that offline data is 'often cheaply available', which seems in conflict with the statement in the introduction that 'collecting expert trajectories is expensive'. Again, it should be no surprise that allowing an agent to interact with the real world will increase performance.

I also raise some very minor points and typos below.

• Lots of the bibliography is incorrectly formatted, seemingly including the names of authors from different papers.
• There are a number of 'bulk references' (i.e., a large number of papers cited in a row) without explanation of what each of these papers are or their relevance to the statement. It would be good to disambiguate between these many references.
• In equation 1, $\pi$ is used on both sides of the equation which is unintuitive. The LHS should be labelled $\pi^*$
• In the explanation of Dreamer, $R_t$ is used to define the critic without definition.
• Pages 8 and 9 have a large amount of whitespace. It would look better to remove this via formatting.
• On line 514, there is a missing space after the full stop: 'best facilitates offline training.In'

1. The novelty of the proposed method is limited. The mixed reward function employed relies on an exploration-exploitation trade-off, which is a well-established framework. For OOD detection, the authors utilize an ensemble approach within the world model, combined with a supervised learning method—both of which are commonly applied in offline RL and lack significant innovation.

2. The approach appears to require task-specific parameter adjustments (e.g., weights for exploration and the OOD ratio), which suggests limited feasibility and generalizability of the method across diverse tasks.

3. The lack of pseudocode for the proposed approach limits clarity. I recommend that the authors include pseudocode in the final version to better illustrate their method and make their contributions more accessible to readers.

1. Lack of novelty: While the paper offers several analyses, many of these insights exist in previous research. For example, the issue of “lack of self-correction causing OOD errors” is a well-known challenge that has been extensively discussed in both offline RL and offline MBRL algorithms [1][2]. Similarly, the use of on-policy samples or exploration data to achieve broader data coverage has been widely studied in the offline (or offline to online) model-free setting [3][4]. Simply extending these methods to the model-based setting does not add substantial novelty.
2. Insufficiently effective solution: Despite dedicating a substantial portion of the paper to analyzing issues in offline MBRL methods, the final solution is not effective enough. Whether using exploration data or incorporating on-policy samples, the authors do not provide a principled approach for deciding when and how to incorporate these data. Additionally, the approach relies heavily on hyperparameters in the data strategy, which limits its generalizability across different environments and datasets.
   [1] Off-Policy Deep Reinforcement Learning without Exploration
   [2] MOPO: Model-based Offline Policy Optimization
   [3] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
   [4] Imitation Learning from Imperfection: Theoretical Justifications and Algorithms

1 The paper confirms a widely studied issue, but it does not reveal novel insights. I agree that, many standard offline RL algorithms (e.g. CQL), approach the issue of visiting OOD states from the perspective of Q values. But I do not think the issue if fundamentally different for offline model-based RL, since in both case, the root cause is that a neural network sees samples not in its training set. See the question 1 & 2 for my suggestion on improving the novelty of this paper.

2. Although aimed to provide insights for practitioners, the three benchmarks used in this paper, DMC, Metaworld, and MinAtar, are in fact not very realistic. In consequence, it is questionable if the same conclusions hold when using more powerful simulators (so it is harder to collect "exploratory" data, since trajectories terminate early).