It's true that our approach is related to the state divergence minimization literature, as we discuss in the paper. Our contribution is to provide a simple and performant instantiation of divergence minimization imitation learning, with the twist that we can do it through simple on-policy RL in a learned world model.

The paper you point out, f-IRL, evaluates on 4 state-based environments, so we don't understand why 6 pixel-based environments is insufficient to substantiate our claims.

RE: Appropriate metrics of covariate shift. This is why we included the latent distance plots, to point out how typical metrics such as action prediction accuracy fail to be predictive of the expected environment return, whereas our latent distance metric is predictive.

Thank you for pointing out the writing issues, we will address these.

RE: "Many of the key formulas (e.g., ELBO objective) are placed in the Appendix, while the authors spend many spaces on related work part" We placed these details in the appendix because they are not novel, the world model is exactly that introduced in the Dreamerv2 paper.

RE: "However, no formal theorem/lemma/proposition is presented in the paper." This is incorrect. Appendix A gives a proof of the return bound.

RE: "The evaluations are limited. The authors only conduct experiments on 5 Atari tasks and 1 continuous control task." Frankly, most imitation learning papers report results on 3-4 low-dimensional state-based environments. We are surprised by this pushback, we feel our evaluation is comprehensive and in more difficult environments than many related works.

RE: IQ-learn and OTR. Thanks for pointing these out. It appears IQ-learn uses online interactions in the Atari environments? They are only fully offline in the state-based environments, if we've understood their paper correctly? It's difficult to directly compare since the experts we imitate have different performance profiles across the environments, but their results indeed look comparable, except for the online interaction part. The OTR paper is also interesting, but it appears they don't test in visual environments, only state-based. Indeed, the difficulty is getting a useful measure of state in visual domains to do some kind of divergence minimization or optimal transport approach. It would be interesting to run an OT approach on the world-model latent space! It seems people think visual imitation learning is trivial, but if you look at many of these papers, they do not test in pixel-based environments.

RE: Lambda return and inner-RL details: These are exactly the Dreamer policy learning procedure, there's nothing exotic or particular here. Indeed we imagine that using more modern approaches such as PPO for the inner RL would probably result in a stronger agent. But we are focusing here on the conceptual problem of doing imitation using world models, not the RL details in the inner loop. The fact that DITTO is this strong without an optimized inner RL loop is probably a good sign.

RE: "I do not think it is difficult to extend these algorithms into the pixel-based variants, do you think these methods can have poor performance in the pixel-based tasks, given that we introduce an encoder to these algorithms?" Well, yes. If you look at SQIL and DRIL, you'll see that adapting standard algorithms like GAIL to the visual setting results in terrible performance. Pixel-based imitation learning is understudied. We hope our work helps to highlight this area of research.

Regarding the literature review being inadequate: your mentioned [2] and [3] evaluate on 3, and 4 low-dimensional state-based environments, respectively. We evaluate on 6 pixel-based environments. There are indeed many related works in the imitation learning literature, but to maintain focus on the hard aspects of partial observability which we would like to tackle, we do not consider works which have so far only demonstrated success in state-based environments. Instead, we adapt mature methods like BC and GAIL to the world model setting. If you compare our adapted baselines to recent methods, you'll find they perform competitively.

RE: "The paper is not well-organized. Some important details are missed in the main body, e.g., the training objective of the world model." Since these details are exactly identical to prior works (adapted straight from Dreamer), we do not consider them significant contributions and placed them in the Appendix.

RE: "Some offline model-based RL baselines are missed" We are not doing offline RL. There are no rewards in our trajectories, so these methods simply do not apply.

The DICE [1] paper evaluates on 4 state-based environments. Again, we evaluate on 6 pixel-based environments, and do not compare with that method since it hasn't been demonstrated in this setting. AIRL is essentially a variant of GAIL, and evaluates again only on low-dimensional state-based environments.

RE: "How does DITTO compare with other model-based reinforcement learning methods that do not use imitation learning?" DITTO does not address the same problem as MBRL methods.

RE: "How does DITTO perform on different types of environments, such as continuous control or navigation tasks?" We included results on a continuous control task in the appendix - bipedal walked from pixels alone. Please take a look!

[1]: I. Kostrikov et al. Imitation Learning via Off-Policy Distribution Matching. In ICLR, 2020.

RE: "The novelty of the method seems limited, which looks like a simple combination of Dreamer and GAIL / BC." How so? GAIL uses a learned, adversarial setup for policy learning. BC simply clones actions. We discuss why both of these methods fail to produce robust imitation policies, and introduce a simple, novel intrinsic reward which can be optimized with off-the-shelf on-policy RL methods. We demonstrate that out method outperforms world-model augmented versions of GAIL and BC.

RE: "The experiment setup is not convincing, it is too simple and all results are only evaluated on a small set of simple tasks" These are standard environments for visual imitation learning. Please see e.g. [1][2], and compare their baseline results with ours. Most imitation learning papers evaluate on much simpler, state-based tasks.

RE: "there is no clear relation between the formulations of Eq. (7) and Eq. (8)" As we stated in the paper, the result holds for any distance function. We chose the particular form of equation 8 based on its empirical performance. The theoretical result does not depend on the details of the distance function. Please see the Appendix and the cited related work for details.

RE: "In Fig. 2, the latent distance is defined by 1−rint, where rint is also defined by yourself. It might be unfair to compare this indicator. What are the results of more common distance measures? Or could you give reasons why this measure is better than other measures?" The point here was to demonstrate that the latent distance measure we introduce is more predictive of final performance, compared to more common metrics of imitation performance, e.g. action prediction accuracy. We expect the on-policy latent distance to be more predictive of real world performance for reasons laid out in the paper, and we indeed find that to be the case.

RE: "Is there any ablation study for different reward functions? If training with Eq. (8) is exactly better than with other common distance measures, it can also partly answer Question 3." We would be happy to add ablations for other distance measures, e.g. L2, and so on. In early testing we found that this modified cosine-distance vastly outperformed all others. The focus of our work is not on the particular form of the distance function, but on the concept of minimizing on-policy state divergence using a learned model.

[1]: Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreement-regularized imitation learning. In ICLR, 2020.

[2]: Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In ICLR, 2020.

1. We are not making any claims about the bias or generalization capabilities of the learned model itself. The point is that the return difference between expert and learner is bounded by the divergence of their induced distributions. Our method is about getting an estimate of the learner's distribution in the model, and explicitly penalizing the divergence from the expert over multiple time-steps, to address compounding error problems. We show that the world model latent space provides an excellent emasure of state distribution divergence. We show that minimizing this divergence using on-policy RL results in state-of-the-art performance in the offline setting (competitive with online methods)

2. Equations 5 and 6 are related in the following way: equation 5 gives a bound on the return difference in terms of the state distribution divergence. Equation 6 shows how the true state distribution divergence is bounded by the divergence in the model. Therefore, bounding the divergence in the model bounds the true state divergence, thereby bounding the return difference.

RE: "There are several Atari games chosen in the experiments, but the reason for choosing them remain unclear." As we stated in the paper, these are the benchmarks used in [1], a well-known SOTA method in the community, as well as other top-performing imitation learning methods, e.g. [2]. In addition to those Atari experiments, we also gave results in a pixel-based continuous control environment. Please see the appendix for those results.

RE: "it is difficult to understand what kind of internal feature space that has been learned by the model [...] one plausible reason is that the learned model feature space has strong generalization ability." Indeed - our results support this view.

[1]: Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreement-regularized imitation learning. In ICLR, 2020.

[2]: Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In ICLR, 2020.

1. Equation 5 establishes the relationship between the expert and learner return gap, and bounds it by their state distribution divergence. Our method bounds this state distribution divergence in the latent space of a learned world model. We give a proof that this bounds the return difference. The learner is penalized by a novel intrinsic reward which we introduce to minimize the covariate shift during training episodes - this is the central idea of the algorithm.

2. We give theoretical results for the exact distribution matching form of the DITTO algorithm, but for computational cost reasons approximate it to match the nearest latent from the initial expert trajectory, not from any expert trajectory. This biases the algorithm towards the mode of the distribution. We speculate that this is beneficial in certain environments (i.e. that the mode expert behavior outperforms the full expert behavior), as discussed in [1].

3. We are not doing RL, so comparing with MBRL algorithms wouldn't really make sense. We're merely using the machinery of RL in the inner loop to induce imitation learning. The point is to use an online/on-policy algorithm inside the world model. Offline MBRL methods are tackling a different set of issues and are not relevant here. There is no "offline reward" here. There are no rewards in our expert trajectories.

[1]: F. Sasaki and R. Yamashina. Behavioral cloning from noisy demonstrations. In ICLR, 2021.

Done!