Imagine Within Practice: Conservative Rollout Length Adaptation for Model-Based Reinforcement Learning

Thank you for your helpful comments on our paper!

• Weakness1: It is difficult to explicitly show experimentally when CRLA increases trajectory length. We attempt to illustrate how CRLA works and its effectiveness using visualization results in 4.3. Figure 5 shows how CRLA works by truncating the trajectory at key time steps, suggesting that our method is able to determine when truncation is necessary. But it is difficult to show under what conditions CRLA is favored to increase the rollout length, since our method is a conservative method. Figure 4 shows the average step size of the unfolding during training, but does not show the maximum length in the trajectories. In fact, in our experiments, trajectories reaching the maximum rollout length existed on almost all environments at any period of training.
• Weakness2: Although our algorithm still shows performance degradation on 12 games, this is due to the fact that we use an automatic threshold setting method which makes it only necessary to set the same threshold parameter $\beta$ for all 26 environments. Our method does not change the policy update rules, but only adjusts the rollout length. We could adjust the thresholds individually for those environments where performance degradation occurs and be able to guarantee that no performance degradation occurs (by simply narrowing the adjustment range so that it is closer to 15), but this does not seem elegant enough. Our method uses the same parameter $\beta$ to achieve a boost in 16 out of 26 Atari games, which we believe reflects the robustness of our approach.
• Weakness2: Our approach is very easy to implement based on Dreamerv3. Our code is based on (dreamerv3-torch)[https://github.com/NM512/dreamerv3-torch] for development and we are currently organizing the code. We will submit the code in a few days.
• Question1: We believe that CRLA, as a conservative rollout strategy, does have the potential to truncate promising trajectories. While World models do generate some well-generalized trajectories, there is still the possibility of early truncation. This can be adjusted by setting the threshold $\alpha$. Although our approach is a conservative strategy, since the judgment condition is the js-divergence of the actor and conservator‘s outputs, there is still a chance that the actor will not be truncated even if it has a probability of sampling unpracticed actions. In other words, the actor still has the ability to explore.
• Question2: It is difficult to balance sample efficiency and sample quality very precisely. This is still an open problem in the MBRL field. CRLA does have the potential to cause some good exploration trajectories to be terminated early, but it does not completely eliminate the possibility of exploration. As answered in question 1, although our approach is a conservative strategy, it is actually a soft constraint rather than a hard constraint since the judgment condition is the js-divergence. As long as the js-divergence is within the threshold range, it is considered safe for the current step to rollout, at which point the actor still has the probability of sampling actions that have never been practiced. This gives our approach flexibility and preserves exploration capabilities to some extent. We believe that this concern can be alleviated as long as the ability to explore is preserved.

Thank you for your helpful comments on our paper!

• Weaknesses:

  • Since the true sampled action distribution in continuous action space is difficult to estimate, we would like to focus on this issue in future research. But that doesn't stop us from focusing on discrete action space tasks. There are some model-based methods that also focus on discrete action spaces, such as some Transformer-based methods like IRIS.
  • Since the y-axis is not scaled in Figure 7, some games with larger scores will appear to have limited improvement visually, and we show the percentage improvement in Figure 3 for a more intuitive comparison.

• Question1: Thanks a lot for providing valuable revisions. It is true that there is no rigorous theoretical analysis in the paper because there is currently no theoretical analysis specifically for Dreamer. We believe that the analysis of MBPO is also applicable to Dreamer to some extent because Dreamer also uses branch rollout to obtain imaginary trajectories, although Dreamer's training is on-policy while MBPO is off-policy. And although MBPO learns the dynamical model directly in the observation space, the analytical approach is also applicable to Dreamer because Dreamer reconstructs the observations and not just the rewards as TDMPC does. We hope to improve the theoretical analysis in our future work, and we also hope that more scholars will pay attention to the theoretical analysis of model-based methods in latent space.

• Question2: Since a cold start is detrimental to the training of the conservator, which may lead it to overfitting early on, we start training the conservator after 10k steps (40k frames) of interaction with the environment. Since the conservator's output is close to the average distribution at the beginning, which results in the computed js divergence being almost always less than the threshold. So the rollout length will be large, and as the conservator is trained, it will gradually be able to predict the historical action sampling distribution and adjust the rollout length. Because CRLA is judged with respect to the current policy, as the policy is updated, it may deviate from the historical action sampling distribution estimated by the conservator. The rollout length decreases gradually during this time. This is due to the fact that our approach is a conservative strategy that conservatively truncates the rollout as soon as there is a large change in policy with respect to the conservator. As the policy stabilizes, the sample distribution in the replay buffer also stabilizes, resulting in the output of the conservator stabilizing. At this time, the output of the conservator will be close to the output of the actor, which leads to a rise in the rollout length. However, since 100k interactions are not enough to make the policy converge completely, the average rollout length can only show a decreasing trend. It is worth noting that, according to the experimental observation, there are still a significant number of trajectories that reach the maximum rollout length in each rollout of the batch data. We will try to prove the above conjecture experimentally and reply later on.

Thanks for the response! I have carefully read the response as well as other reviewers' comments. However, I don't find the response by the authors convincing at all. For the performance comparison, the authors claim that "some games with larger scores will appear to have limited improvement visually ." Could you then plot the learning curve (and perhaps an aggregated plot across all games) with respect to the Atari Human-normalized score? For the theoretical analysis, if the authors claim that they are providing theoretical justifications for their algorithms, I suggest the author rigorously present a version of their algorithms under the MBPO framework and carefully analyze the performance bounds. Otherwise, the version now only explains the intuition. Therefore, I would like to maintain my score.

Thank you for your helpful comments on our paper!

• Weakness1:

  • Thank you very much for the valuable revisions. As you said, our presentation is indeed not rigorous enough. We will revise the presentation and add relevant citations.
  • Due to limited time and computational resources, we will conduct experiments on four games with linearly growing rollout length. We will add the experimental results in a follow-up reply.

• Weakness2: We have taken note of these papers, but we believe that our approach and theirs have different emphases, despite the similarities in motivation. Our approach focuses on how to use the world model more safely for rollout by adjusting the rollout length, provided that model errors cannot be eliminated. Moreover, these two methods are based on a similar framework to MBPO, which requires storing imagined data and performing off-policy training, and cannot be directly migrated to dreamerv3, which performs real-time rollout and on-policy updating without saving imagined data. We briefly explain the reasons below.

  • For M2AC, it needs to evaluate imagined rewards based on uncertainty. This requires the simultaneous training of multiple dynamical models, which is computationally prohibitive for Dreamer based on image inputs. And since the transition occurs in a latent space, ensemble's models need to be unified in one latent space, which is obviously difficult to do.
  • For PDML, its motivation is to learn more accurate dynamics models. It is essentially a data sampling method to achieve better model learning by modifying the sampling probability of different data. This is fundamentally different from our approach. Our approach considers how to utilize the model more safely and efficiently by adjusting the rollout length when the model error cannot be completely eliminated. Moreover, PDML needs to save its model parameters after each policy update, and hundreds of checkpoints need to be saved throughout the training process, which requires additional storage. And dreamerv3 is updated once for one step of interaction with the environment, which is clearly not applicable if we want to use PDML's method to estimate the policy shift, which requires saving tens of thousands of model parameters.

• Weakness3: It is true that latent world model do not need to explicitly predict the next observation, but it is not protected from model errors. A key reason is that Dreamerv3 uses actor-critic architecture and needs to estimate the value of the latent state. Inaccurate latent state estimation can lead to problems with value estimation. And since the training of world model is based on limited real data, while the training of actor and critic are based entirely on imagined trajectories, this leads to the quality of imagined trajectories directly affecting the training of the actor and critic.

  It is an oversight on our part not to add the relevant citation, which we will add. DMVE [1] shows the performance curves of Dreamer for different rollout length. When the rollout length is larger than 15, the performance shows a decreasing trend. And when the rollout length is 30, Dreamer almost completely loses its performance. We think this is enough to support our view.

• Weakness4: Since the true sampled action distribution in continuous action space is difficult to estimate, we would like to focus on solving this problem in future research. However, this does not affect our aim to solve the problem in discretized action space. In fact there are many related algorithms focusing on discretized action space, such as IRIS [2] and TWM[3].

• Weakness5: Since the y-axis is not scaled in Figure 7, some games with larger scores will appear to have limited improvement visually, and we show the percentage improvement in Figure 3 for a more intuitive comparison. Taking into account the variance of Atari, the data in Figure 3 are the average performance gains over 10 trainings, with each training performance test being the average result of 100 full episodes at the end of the training. This gives a more accurate reflection of the performance of the policy.

Reference:

  [1] Wang et al. "Dynamic Horizon Value Estimation for Model-based Reinforcement Learning". arXiv:2009.09593. 
  [2] Micheli et al. "Transformers are Sample Efficient World Models". ICLR2023
  [3] Robine et al. "Transformer-based World Models Are Happy With 100k Interactions". ICLR2023.

Thank you for your helpful comments on our paper!

• Weakness1:

  • Since we focus on discrete action spaces, Atari100k is currently one of the most popular benchmarks, and several model-based methods have focused on Atari100k to test the effectiveness of the algorithm, e.g., IRIS[1], TWM[2]. Since Atari100k contains 26 games, its a difficult enough challenge to demonstrate the effectiveness of the algorithm.

  • Although our algorithm still experienced performance degradation on 12 games, this is due to the fact that we used an automated threshold setting methodology which made it only necessary to set the same threshold parameter beta for all 26 environments. Since our algorithm does not essentially change the policy update rule, but only adjusts the rollout length, we could adjust the thresholds individually for those environments that show performance degradation, but this does not seem to be elegant enough.

  • Our algorithm uses the same parameter $\beta$ to achieve a boost in 16 out of 26 Atari environments, which we believe reflects the robustness of our algorithm.

• Weakness2: The difficulty in estimating the true action sampling distribution in continuous action space makes it difficult to use our algorithm for continuous action space. However, this does not prevent us from focusing mainly on the discrete action space in this work. In fact some model-based approaches also focus on the discrete action space, such as IRIS. We leave improvements on continuous action spaces for future work.

• Weakness3: We apologize for the lack of rigor in our description and we will revise the description. We would like to emphasize the simplicity of the way of computation rather than the computational saving. Since our algorithm computes the respective rollout lengths for each rollout trajectory, to better utilize the GPU for batch inference, we chose to rollout the maximal lengths directly instead of removing the truncated trajectories. Due to the smaller number of parameters in the conservator network, the algorithm only takes about 20% more time compared to dreamerv3 (9.5h for dreamerv3 and 11.5h for CRLA), with about 3% more GPU memory usage.

• Weakness4: Thank you very much for the proposed changes, we will make further revisions to this section.

• Questions1: For the interval of rollout length, we tried a larger range of rollout but observed some performance degradation. This may require setting different thresholds for different environments, but we think this is not robust. Since it is difficult to accurately perform a longer rollout for environments based on image inputs, we empirically set the rollout length to [5,16]. This better emphasizes the balance of data quality and sample efficiency of our algorithm compared to a fixed step size of 15. We add a reference which illustrates the performance degradation of Dreamer after unfolding steps larger than 15 [3].

• Questions2: This is a great suggestion and thank you very much for bringing it up. Due to limited computing resources, we will be testing this on four games, and the results of the experiment will be responded to later.

Reference:

  [1]Micheli et al. "Transformers are Sample Efficient World Models". ICLR2023
  [2]Robine et al. "Transformer-based World Models Are Happy With 100k Interactions". ICLR2023.
  [3]Wang el al. "Dynamic Horizon Value Estimation for Model-based Reinforcement Learning".