Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning

• The motivation of the proposed ADM rollout should be clarified further. In the proposed rollout in ADM (Algorithm 1), when predicting the future state such as $s_{m+1}$, $k=2$, it essentially is based on the information of $s_{m-1},a_{m-1:m}$. When $s_{m}$ is available in the given sequence, it seems not clear why the proposed ADM can perform good. Moreover, when the state transition and policies are stochastic, the proposed rollout essentially requires to consider the stochasticity in $s_m$ and $a_m$ when making prediction for $s_{m-1}. What is the intuition of using this rollout for the sake of the performance.
• The proposed rollout algorithms also make prediction based on the previous prediction, e.g., when predict $s_{m+2}$, it may require $s_{m+1}$ (which is obtained in the previous steps). It is thus very important to clarify why this method can effectively mitigate the bootstrapping errors comparing with the conventional approach.
• The backtracking length $m$ is a newly introduced parameter, and the ablation studies show that it affects the performance a lot. How to choose such a parameter in practice for specific task? Will it work for all different tasks?

• Clarity: The paper is overall hard to follow with poor writing and inconsistent notation. For example, sometimes $\hat{s}$ is used to refer to states predicted by the dynamics model, sometimes $s$ is used to refer to predicted states. $s$ is also used to refer to ground-truth states. In addition to various typos this makes the paper hard to follow.
• Soundness: The paper does not give a convincing reason for why their proposed method works better, nor does it include ablations or experiments that could elucidate why it performs well. There also isn't any theoretical analysis that might be helpful to understand it better.

Some important implementation choices are not justified either. To list the most major issues:

• As written in L152, and shown in Figure 1, the state $s_t$ is repeated as input for all GRU/ model steps. This is reasoned to prevent error accumulation of using model predictions as input. However the paper assumes access $m$ step sequences which are then used to predict the next $H$ steps. Why aren't all $m$ steps used as model inputs?

• Using a random input length during training is reasonable to learn a model that performs well for different input lenghts. However, when using the model to train a policy it would seem more natural to average across lenghts to obtain a better prediction, or choose the longest length. Why is a random length used?

• Experiments: Hyper-parameter choice is not discussed and only 5 seeds are used. For most baselines, the results were copied from the original papers. As hyper-parameter choice and implementation details are important for RL methods, this is not sufficient, especially when applying the algorithms to a new benchmark (NeoRL).

Minor issues:

• Theorem 3.5 follows rather straightforwardly from Assumption 3.3 and does not provide any new insights. It is also not specific to the ADM and could apply to any model-based RL method.
• parantheses should be placed around the author name for citations, not just around the year number.
• What the authors refer to as "bootstrapping prediction" issues seems more like error accumulation to me. Using the term bootrstrap can be a bit confusing in an RL context, but this is a minor point.
• Algorithm 2 and 1 should probably combined and both be in the main text, as they are important to understand the approach.
• L241: "explore beyond the boundaries of the risky regions" should be "beyond the boundaries of the safe regions"?
• L352: should this be $\hat{s}_{t+m}$? If not, there can be no compounding error.

• The main motivation of this work is that of alleviating autoregressive error accumulation. Nevertheless, ADMs rely on a recurrent architecture that implicitly bootstraps on previous hidden states. It is not entirely clear to me why this would not also result in error accumulation. Empirical performance provides some support to the method's effectiveness, but are there any principled reasons why ADMs suffer less from error accumulation? In my opinion, this one of the main issues with the current version of this work. The authors report a Theorem (3.4), but it appears to be a general result. How does it relate to the backtracking nature of the proposed method?
• The second issue is in the limited comparison to recurrent forward models. Recurrent world models are at the core of SOTA model-based algorithms [1,2]. The only empirical comparison to recurrent models is, as far as I understand, in Figure 2, which is not easy to interpret. The RL evaluation does not include any recurrent architecture. This raises the following question: is the improvement in RL performance due to any-time dynamic sampling, or to the choice of architecture? For instance, how would the method compare in online RL to a RSSM [1] of similar size, using standard autoregressive sampling?
• A third main issue is an insufficient discussion of the limitations of this method. A fundamental issue which is not mentioned in Section 6 is that the uncertainty estimation method proposed appears to mix aleatoric and epistemic uncertainty. This is in general suboptimal: how would if perform in an environment in which the optimal policy traverses stochastic parts of the MDP?

[W1] ADM uses multiple-step rollouts, initializing from a state several steps back in time to predict the trajectory, thus potentially disregarding states closer to the current time step. This may increase compounding errors in rollouts compared to ensembles, which operate with single-step rollouts based on the current state.

[W2] The reference formatting in the main text appears incorrect; minor revisions are recommended.