Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models

Weaknesses

 This paper focuses on navigation tasks. While these tasks are challenging, my main concern is to what extent do these studies replicate to other tasks.

---

While we agree that further experiments are needed to confirm scalability of PLDM to other environments, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. Additionally, we test on ant navigation, an environment that has much more complicated dynamics due to the agent being not a point mass but a 4-legged ant, which showcases PLDM's scalability to complex dynamics.

 

 Also quite recently there have been other latent planning style architectures for example...

---

Thank you for pointing out these works. We have added them to the related works section. However, we couldn't directly compare these to PLDM as Latent Adaptive Planner for Dynamic Manipulation does not provide the implementation, and Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference requires reward signal, although only per-trajectory, making direct comparison impossible.

Questions

Besides the decoder-based and Dreamer-style ablations, did you test other reconstruction-free SSL objectives (e.g., BYOL, VICReg) within the same planning loop?

---

In addition to the experiments in the paper, we conducted an additional experiment with TDMPC2 architecture and representation learning objective. Because TDMPC2 relies on rewards to learn a Q-function and a policy, we removed those components, but kept the rest unchanged. We found that TDMPC2 without rewards completely collapses, leading to 0% success rate even with good-quality data. We added inverse dynamics learning to TDMPC2, and found that, while avoiding collapse, such model still achieves 0% success rate with good data.

Can you quantify how much of PLDM’s gain derives from the JEPA encoder versus from having any learned latent model at all? (or provide a rough heuristic as to what you'd expect?)

---

JEPA architecture only stipulates that the representations and dynamics are learned without reconstruction, and therefore encapsulates TDMPC2 as well. As discussed above, we found that VICReg-based objective performs much better, with TDMPC2 objective failing completely. If we compare JEPA vs non-JEPA approaches, we can look at the comparison between PLDM and reconstruction-based representation learning in Table 6 in the paper. We see that reconstruction-free approaches get less than 30% success rate with good-quality data, while PLDM achieves ~97%.

How sensitive is performance to small visual perturbations (Gaussian noise, random crops, mild viewpoint changes)?

---

We hypothesize that unless the models are trained with such perturbation, they will fail. We are currently running experiments to test if PLDM can work with these perturbations without seeing these transformations during training, we will report the results as soon as they are available.

Formatting issues

Thank you for pointing out the typos, we have updated the manuscript accordingly.

Weaknesses

1. Despite answering numerous questions regarding performance, the paper does not adequately delve into why a particular method performs better under specific conditions, limiting the depth of insight. Further investigation into the underlying mechanisms driving observed performance differences would strengthen the analysis.

---

Thank you for pointing that out. As you mentioned, the paper is empirical in nature, with limited analysis of particular reasons for failure. We believe that such analysis is beyond the scope of this work. However, we do analyze why HILP fails on layout generalization, and show that the learned distance function used to train the policy does not generalize well to new layouts (section H).

2. The baseline comparisons are somewhat narrow, focusing exclusively on model-free GCRL algorithms.

---

We do not solely focus on model-free GCRL algorithms, as HILP is not a goal-conditioned method, and is instead an unsupervised RL method. We also tested Dreamer, and reconstruction-based representation learning, and got poor results even with good data, and therefore didn't test on more settings (see Table 6). Additionally, prompted by questions in this review, we conducted experiments with TDMPC2, and found that due to lack of reward information, TDMPC2 representations completely collapse, leading to 0% success rate.

3. The tasks utilized in the experiments lack sufficient variation, as acknowledged in the limitations section.

---

While we agree that further experiments are needed to confirm scalability of PLDM, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. Additionally, we test on ant navigation, an environment that has much more complicated dynamics due to the agent being not a point mass but a 4-legged ant, which showcases PLDM's scalability to complex dynamics.

4. The proposed PLDM (Planning with Latent Dynamics Models) replans at every step, which, as shown in Appendix E, results in approximately four times slower computation. The paper would benefit significantly from an investigation into how performance changes with larger replanning horizons (e.g., replanning every 4th step), addressing a practical efficiency concern.

---

We believe there must have been an understanding. We would like to point out that Appendix E tests exactly that, and shows that replanning every 4th step as opposed to every step, retains 95% of the original performance while being only 20% slower than policy-based methods. Please let us know if this is not what you were looking for, we are happy to answer any further questions.

5. Section 4.8 exhibits an unusual arrangement.

---

While we agree the figures in the appendix aid in understanding this section, we disagree that they are crucial:

• Table 4 essentially shows that all the methods get 100% performance, which is exactly what is said in the main text, making the table redundant.
• Figure 7 is a summarized version of Figure 8 and shows just one layout example. Figure 8 shows trajectories for each method on a separate image, and shows more layouts with different edit distances. While Figure 8 helps in understanding the experiment, we believe Figure 7 is enough to convey the main point. Additionally, Figure 8 is way too large to be able to fit in the main text.

Figure 2 is shown in the beginning of the paper because we believe this is the most interesting result, but we agree that placing it directly in the sction discussing it will be more convenient for the reader. We will move this figure to section 4.8.

Questions

Currently, PLDM replans every step, leading to increased computational cost. Could the authors present an experimental analysis demonstrating how performance (e.g., success rate, trajectory quality) varies with different replanning horizons (e.g., replanning every 2nd, 4th, or 8th step)? This investigation would provide crucial insights into the practical deployability and computational trade-offs of the method

---

See response to Weakness 4.

In Sec. 4.5 the authors come to the conclusion that HILP and PLDM generalize better to out of distribution goals. Did the authors investigate the reasons why this is the case?

---

We hypothesize that GCRL methods fail on out-of-distribution goals due to trajectory stitching. PLDM can easily stitch trajectories from the training data as the dynamics model predicts one step ahead, and thereofre will learn transitions starting from the same state s but with different actions, e.g. $(s, a_1, s_1')$ and $(s, a_2, s'_2)$ if such transitions are present within any two trajectories in the dataset. HILP works well because it is not technically goal-conditioned, but instead uses latent-space direction to condition the policy (see Figure 1 in HILP paper). Covering all existing directions in latent space is easier than all existing goals in the observation space, therefore HILP can reach out of distribution goals as long as they can be reached by following known directions in the latent space. Offline GCRL on the other hand struggles with stitching, as in order to enable stitching, these methods rely on random goal sampling and dynamic programming. The larger the environment, the harder it is to sample reasonable goals to encourage value and policy learning, making such methods fail.

Can the authors explain the difference between PLDM's inferred plan and the actual trajectory in Sec. 4.8?

---

The inferred plan is constructed at timestep t=0. We construct a new plan every time we take a step, therefore the resulting executed trajectory may differ from the first planned trajectory due to randomness in the planning algorithm (MPPI samples candidate trajectories randomly) and due to small differences between model predictions and actual environment observations.

We do not solely focus on model-free GCRL algorithms, as HILP is not a goal-conditioned method, and is instead an unsupervised RL method.

Based on the information provided in the paper, it is not evident that the used HILP version is not a goal-conditioned method. According to the original publication, HILP can be applied in various settings, including unsupervised, skill-based, zero-shot reinforcement learning, or goal-conditioned reinforcement learning. Could the authors clarify which specific variant of HILP was used in their experiments? In particular, does HILP receive the goals (as it seems according to Appendix H)?

While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends.

I agree that the selected environments provide a reasonable starting point and illustrate several important capabilities. However, these environments may not adequately reveal the limitations of PLDM. In fact, they might align more closely with the strengths of PLDM—for instance, planning-based methods often face challenges in environments requiring extensive exploration or involving long-horizon tasks. The used environments do not require extensive exploration nor are the involved task long-horizon. Therefore, the generality of the reported results may be more limited than suggested. What is the authors take on this? Does the choice of evaluating on one environment not limit the overall generality of the results?

We believe there must have been an understanding. We would like to point out that Appendix E tests exactly that, and shows that replanning every 4th step as opposed to every step, retains 95% of the original performance while being only 20% slower than policy-based methods.

Maybe I did not phrase things clearly enough beforehand: In the current experimental setup, PLDM is allocated a computational budget that is approximately four times larger than that of the other baseline methods. From an evaluation fairness perspective, it would be more appropriate to compare the baselines against a version of PLDM that replans every fourth step, thereby matching the computational constraints. As it stands, it is difficult to accurately assess PLDM’s relative performance under equivalent computational budgets, and any conclusions drawn are necessarily speculative.

Yes, your understanding is correct, we used the process defined in section 5.2 of the paper introducing HILP.

We did not use test-time planning from section 5.3 in two-rooms as HILP achieves near-perfect performance there even without it. In diverse maze, HILP fails not due to planning complexity, but due to representations failing to capture the distance, as we showed in Appendix H, so we didn't use test-time planning there either.

Let us know if you have any further questions or concerns.

Is HILP goal-conditioned?

GCRL, CRL, HIQL, GC-IQL methods train a policy that directly accepts current state and the goal state and produces the action, and which therefore can only be applied to the goal-conditioned problems. As is common in the literature, including the paper introducing HILP, we refer to such methods as 'goal-conditioned'. On the other hand, HILP cannot be considered a purely goal-conditioned method, as it is more general and can be used in other settings as we showed in section 4.5, and as was discussed in the original paper. The policy HILP proposes takes as input the current state and a latent variable, which can be derived in different ways, including, but not solely, from a given goal state. The latent variable aims to capture a variety of behaviors, including those not corresponding to reaching a set goal.

We will clarify this distinction in the paper, and say "purely goal-conditioned" when referring to GCRL, CRL, HIQL, and GC-IQL to avoid confusion. Let us know if this addresses your concern.

The generality of the reported results is limited

We agree that these results are cannot be used to conclude, once and for all, that PLDM is better than other methods in all cases. The explicitly stated goal of this work is to carefully test methods applicable to reward-free offline data in settings where we have a lot of control over the data. We find that PLDM works well, and therefore attempt to draw attention of the community to this relatively underexplored paradigm of learning latent dynamics and planning. Our findings---for planning in latent dynamics models for diverse environments---fill a gap in the literature and, we believe, would be of great interest to the community. We agree that further extending our investigation to long-horizon tasks and to tasks requiring exploration would be valuable, but our results stand on their own, and we hope to extend this investigation in follow-up work.

That being said, we appreciate your comments regarding the generality of our findings and propose the following changes:

• We will more explicitly state what types of generalization we are testing in each of our experiments. Specifically, we will state that we only test generalization in terms of environment layouts, and in terms of tasks defined on the same state/action space with the same transition function. We will state that other types of generalization, such as generalization to changes in environment dynamics, to noisy observations, etc., are not evaluated for. More specifically:
  • in line 281, instead of saying "test whether methods can generalize to new environments", we will say "generalize to new obstacle configurations (layouts) within the environment".
  • In line 288: "to test generalization" will be replaced with "to test generalization to new obstacle layouts"
  • In line 310: "generalization" will be replaced with "generalization to new environment layouts"
• In any conclusions we draw from our experiments, we will change the phrasing to specify what kind of generalization we observe. For example:
  • line 258 "planning with a latent dynamics model can generalize well to a new task.", will be replaced by "Assuming the environment dynamics remain fixed, PLDM can generalize to tasks other than goal-reaching simply by changing the planning objective."
  • line 305: "The model-based approach enables better generalization to unseen environment variations than model-free methods." will be replaced with "PLDM enables better generalization to unseen obstacle layouts within the environment when compared to model-free methods."
  • line 314 (the final conclusion box): "Learning a latent dynamics model and using it to plan exhibits robustness to data quality, superior data efficiency, and the best generalization to new layouts and tasks;" will be replaced with "PLDM exhibits robustness to data quality, superior data efficiency, the best generalization to new environment layouts, and excels at adapting to tasks beyond goal-reaching;"
• In limitations, and at the end of section 3.3, we will reiterate the fact that planning is more computationally expensive (about 4x slower in our experiments) than a single run of the policy trained with RL;

We believe that these changes represent a fair representation of the experimental evidence presented in our work and will make it easier for potential readers to clearly understand relative strengths and weaknesses of RL and optimal control.

Weaknesses

1. The evaluation domains have limited difficulty, and it is unclear how much the claims extend to more complex domains.

---

We agree that further experiments are helpful to confirm scalability of PLDM to other environments. However, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. In addition, our results on AntMaze—where the agent is a quadruped rather than a point mass—offer promising early evidence that PLDM can extend to settings with more complex control dynamics.

2. Some claims presented in the paper are overstated. For example, in Section 4.6, the paper studies generalization to new tasks by inverting the sign of the cost function, converting goal-reaching tasks to goal-avoidance tasks. From this one experiment, the paper claims that PLDM "can generalize well to a new task." This is vastly overstated, as the results might stem from the idiosyncrasies of the environment dynamics.

---

Thank you for pointing this out. We will tone down this claim. However, we'd like to emphasize that this experiment serves to showcase that PLDM can be adapted to an entirely different task just by changing the planning objective -- an ability that GCRL methods lack, no matter how simple the task is.

3. The paper only compares to a single instantiation of model-based planning with a JEPA-style model (a dreamer comparison is included in the appendix in limited domains). Again, it is unclear how much the results transfer to model-based planning with other model architectures.

---

While it's true that we only consider a single instantiation of a model-based algorithm, we did try Dreamer-style reconstruction, and found performance very poor even with good-quality data. Additionally, prompted by this review, we conducted experiments with TDMPC2, and found that it performs poorly or collapses completely. Existing model-based methods like Dreamer and TDMPC2 heavily rely on the provided reward information in the data, and fail when the data is not available. Thus, comparing PLDM to these methods is not an apples to apples comparison. We are considering the reward-free offline setting, which, while very important, hasn't been explored in the literature as much as learning from rewards was.

Questions

In the dreamer baseline in Appendix F, what's the planning cost? Is it pixel distance?

---

We used the latent representation space distance, just like we did for PLDM.

Can you include comparisons on more complex domains such as manipulation? An example could be franka kitchen from D4RL, which provides datasets of varying quality and can be cast as a goal-reaching problem.

---

We are running experiments with manipulation, but we are afraid that this will take longer than the duration of the rebuttal. Also, as mentioned above, we believe the existing experiments already support the message we are conveying in this paper.

Weaknesses:

1. Many works have been proposed to show relationships between RL and optimal-control based approaches, and latent dynamics models are also commonly used for planning in the literature, the contribution of the paper is incremental;

---

We agree that the relationship between RL and optimal control has been explored since the very inception of RL, and we do not claim novelty in either being the first to investigate the relationship between RL and control, or to propose planning. However, we argue that our proposed approach is novel because it considers the planning with latent-dynamics in the reward-free setting. Previous works that use planning, such as TD-MPC2, use reward signal. PLDM on the other hand doesn't use rewards and therefore is more flexible in terms of what data it can use, and what tasks it can be applied to downstream. Furthermore, our work includes comprehensive empirical evaluations across multiple datasets with varying data quality, which, as reviewers saB6, 2Ba1, and WZKp noted, provide valuable insights into the strengths and limitations of existing methods in the reward-free offline regime. We would be happy to clarify or expand on any part of our contribution, and if there are specific related works we’ve overlooked, we would gladly cite them upon suggestion.

2. In many cases, the performances, except for the generalization ability of the proposed latent dynamics, are not competitive compared with existing methods;

---

Aside from generalization, PLDM exhibits stronger robustness to data quality that GCRL methods across all experiments. Additionally, although HILP has competitive performance with PLDM on sequence length and data quality, PLDM is significantly more data efficient.

3. These approaches are only verified on two simple navigation tasks, the scalability of the proposed model in some real-world applications is unclear.

---

While we agree that extending our expeirments to more complicated environments would be valuable, we believe we already provide strong enough evidence with experiments spanning both visual and proprioceptive domains, as also mentioned by reviewer 3zum. These simple environments are designed in a way to have a minimally complex setting to stress-test methods for offline reward-free data, and we believe these environments achieve this goal -- we see existing methods fail in unexpected ways. Additionally, granular control of data generation afforded by the simple environments we consider would be much harder in more complicated environments like manipulation.

Questions:

1. Why another benchmark, and not D4RL?

---

While D4RL is indeed a great benchmark for offline RL, it is not suitable for the reward-free context, as each environment there only has one task. For example, the navigation tasks considered in D4RL only have one goal state in the maze. Our Two-Rooms and Diverse Maze environments, while simple, sample the goal location randomly, making it much more challenging.

2. As can be seen from the results, except for generalization, the performance of PLDM is not competitive, and in many cases, HILP even performs better. Then what are the advantages of PLDM compared with HILP? Moreover, as mentioned, in the literature, the use of latent dynamics model for planning is actually not new, and more discussions and comparisons among these latent-state based approaches should be reported.

---

We appreciate the reviewer’s question. While HILP demonstrates strong performance in stitching and on low-quality data, PLDM performs competitively in these settings and brings distinct advantages that make it a valuable contribution. In particular, PLDM is significantly more data-efficient (Section 4.3), and it shows stronger generalization to both unseen tasks (Section 4.6) and unseen layouts (Section 4.8), which is crucial for scalable offline learning. We agree that latent dynamics models for planning have been explored in prior work. However, to the best of our knowledge, PLDM is the first end-to-end method to systematically apply this paradigm in the offline, reward-free setting. We appreciate the reviewer’s suggestion to expand the discussion of related latent-state planning approaches and will include a more detailed comparison in the final version of the paper.

3. As is well-known, the performances of model-based approaches rely heavily on the accuracy of the learned model, then how can we guarantee the accuracy of the model of PLDM? Is it also the reason that these approaches are only compared in the two simple navigation tasks as more complex the task is, the less accuracy of the learned model may be.

---

PLDM relies on model ensembles to measure dynamics model uncertainty (see Eq. 6 in the paper). While this is not a perfect measure of model accuracy, it can serve as its proxy during planning, and limit considered actions to ones where the dynamics model is accurate. We already show this can work on ant navigation (see section 4.7), an environment with relative complex dynamics. Overall, we believe that uncertainty estimation will enable PLDM to scale to even more complicated dynamics.

4. Experiments on some large-scale environments are welcome to show the scalability of PLDM.

---

We agree that further experiments are helpful to confirm scalability of PLDM to other environments. However, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. In addition, our results on AntMaze—where the agent is a quadruped rather than a point mass—offer promising early evidence that PLDM can extend to settings with more complex control dynamics.

Weaknesses

I think that the authors should be more explicit about what "new task" and "new environment" actually mean in terms of the dynamics of the MDP.

---

Thank you for pointing this out. We will add this description to the paper. By new task, we mean any task defined on the same environment, with the same state/action space and dynamics. We picked avoidance task as it's a natural task for navigation, and showcases flexibility of PLDM which GCRL methods don't have. In terms of new environment, we only consider layout perturbations, and assume that there exists a distribution of layouts, some samples from which are seen during training. During inference, unseen samples are used for evaluation.

In general, I think that a lot of the presented experiments favor methods that explicitly learn a dynamics model (which is policy agnostic) rather than a Q-function (which is conditioned on a policy).

---

In this work, we aim to investigate what methods can learn well from reward-free offline data. To do well with this data, methods need to be able to generalize to new tasks. We disagree that this favors dynamics models specifically, but we do agree that this setting points out an important limitation in most commonly used value-based methods in RL: the policy collecting the data may be completely irrelevant to the downstream task, which impedes Q-learning. In this paper, we argue that, instead of Q-learning, a promising way to tap into reward-free offline data is learning a dynamics model and planning (PLDM), as it's better at handling low-quality data, and can be used to solve different tasks after training.

**To the best of my understanding, the hyperparameters for the baselines are not tuned (the authors use the default values from OGBench), while the proposed method uses separate hyperparameter settings per environment. If the baselines were tuned via the same procedure as the proposed method the results would be more convincing **

---

We do tune baseline hyperparameters for diverse maze, see appendix I.1.2 for the exact valus. For two rooms, we tuned hyperparameters for the good-quality data, and used them throughout the rest of the experimetns. We are currently running hyperparameter tuning for experiments with sequence length set to 17, as tuning all methods hyperparameters for each individual setting will be prohibitively expensive. The experiments are still running for GCBC, HIQL, CRL. We didn't tune HILP as it already achieves perfect performance. We did observe a boost in performance for GCIQL. We will update results of GCIQL and other methods as the experimetns complete.

Why not use any model-based RL baselines? For example, TD-MPC2 [1] was shown to work in an offline setup and also uses a learned model, but there are a variety of other papers

---

We ran additional experiments to test TDMPC2 performance in our setting. Since we are in the reward-free offline setting, we removed the reward and q-function learning components from TDMPC2, but kept the rest of the architecture unchanged. We found that TDMPC2 collapses without additional reward supervision and achieves 0% success rate.

Nitpicks

I would highly prefer to have metrics instead of colored stars in Table 1 - it is hard to say what "average performance" or other actually mean

---

Thank you for pointing this out. We will redo Table 1 to have stars denote percentile performance of the methods.

There is a lot going on in Figure 1

---

Thank you for pointing this out, we will simplify it by removing the results part of the figure.

I do not like that the authors motivate their work with a broad comparison of RL and OC in the offline setting, but end up using a single model learning OC algorithm which is also a method proposed in this work. I think that the introduction would work better if authors just said that they propose an OC/model learning method which they will benchmark against RL

---

Thank you for this thoughtful comment. We agree that clarifying the framing in the introduction would improve the paper and will revise it to more clearly state that we are introducing a novel OC/model-learning method. That said, the broader motivation of the work remains to explore the relative strengths and weaknesses of RL and OC in the offline reward-free setting. PLDM was proposed specifically because existing OC methods do not work under these conditions. To support this comparison, we have included Dreamer as an additional OC baseline and added results for TD-MPC2 in this rebuttal.

The first contribution relating to the proposed environments and data generating process could be documented better. Where is the "granular control" described?

---

Thank you for pointing this out. By granular control, we mean the flexibility the proposed environment affords in adjusting data collection properties, such as agent behavior (purely random or biased for one direction), state coverage (limit trajectories to one room), episode lengths, etc. The TwoRooms environment is also GPU-accelerated, meaning that we can quickly collect new datasets and experiment with many variations of the datasets. We will add this discussion to the paper in the sections describing the environments.

Questions

Any hypothesis how would the model-based method perform with a smaller action repeat?

---

Smaller action repeat, e.g. 2 instead of 4, will mean that covering the same distance will require twice the number of steps. This will require a longer planning horizon, which will in turn mean that the model error will accumulate more, leading to worse performance. On the other hand, we hypothesize that long-horizon tasks can be solved more easily with planning if action repeat is increased.

How central is JEPA to the performance here? What would happen with the TD-MPC2 model instead of JEPA?

---

While PLDM and TD-MPC2 use different methods for learning the representation, TD-MPC2 can be considered to be a JEPA, as it learns representations and the dynamics by predicting in the latent space and does not use reconstruction. As other reviewers also pointed out the relevance of TDMPC2, we ran additional experiments to test its performance. Since we are in the reward-free offline setting, we removed the reward and q-function learning components from TDMPC2, but kept the rest of the architecture unchanged. We found that TDMPC2 collapses without additional reward supervision and achieves 0% success rate.

Thank you for your feedback! We are glad you find the experiments and their results convincing.

We appreciate your point regarding the strength of our claims. We reviewed the claims in the paper, and propose the following changes in view of your feedback:

• We will explicitly state what types of generalization we are testing in each of our experiments. We will state that we only test generalization in terms of environment layouts, and in terms of tasks defined on the same state/action space with the same transition function. We will state that other types of generalization, such as generalization to changes in environment dynamcs, to noisy observations, etc, are not evaluated for. More specifically:
  • in line 281, instead of saying "test whether methods can generalize to new environments", we will say "generalize to new obstacle configurations (layouts) within the environment".
  • In line 288: "to test generalization" will be replaced with "to test generalization to new obstacle layouts"
  • In line 310: "generalization" will be replaced with "generalization to new environment layouts"
• In any conclusions we draw from our experiments, we will change the phrasing to specify what kind of generalization we observe. For example:
  • line 258 "planning with a latent dynamics model can generalize well to a new task.", will be replaced by "Assuming the environment dynamics remain fixed, PLDM can generalize to tasks other than goal-reaching simply by changing the planning objective."
  • line 305: "The model-based approach enables better generalization to unseen environment variations than model-free methods." will be replaced with "PLDM enables better generalization to unseen obstacle layouts within the environment when compared to model-free methods."
  • line 314 (the final conclusion box): "Learning a latent dynamics model and using it to plan exhibits robustness to data quality, superior data efficiency, and the best generalization to new layouts and tasks;" will be replaced with "PLDM exhibits robustness to data quality, superior data efficiency, the best generalization to new environment layouts, and excels at adapting to tasks beyond goal-reaching;"
• In limitations, and in the end of section 3.3, we will reiterate the fact that planning is more computationally expensive (about 4x slower in our experiments) than a single run of the policy trained with RL;

We believe that these changes will make the claims more in line with the experimental evidence presented in our work, and will make it easier for potential readers to clearly understand relative strengths and weaknesses of RL and OC.

Let us know if these changes address your concerns.

Overall, I think this is an interesting work, and I am generally leaning towards acceptance. I am satisfied with the number of experiments and their results. However, the descriptions of what is being done were sometimes lacking - I understand that authors are ready to improve these in the final version.

Given above, the only thing holding me back is the use of strong language throughout the paper that sometimes feels like overclaiming the benefits of using classic control versus RL. I would be much more confident in the paper if the authors slightly toned down the commentary of the results. Can authors propose something (e.g. toning down the 'generalization' claims, more open discussion of computational costs)?