We thank the reviewer for their comments and suggestions. Below, we will address the main questions and concerns raised by the reviewer.

How do you select the hyper-parameters for different experiments?

We refer the reviewer to Section “Hyperparameter Selection” in our common response where we clarify how hyperparameters were selected.

What are the original image sizes for Minecraft Dungeons and Minecraft?

Our collected data in Minecraft Dungeons and the filtered dataset in Minecraft contain images of resolutions 1280x720 and 640x360, respectively. We added this information to the environment descriptions in the paper.

Why does the paper use FocalNet as classification supervised pre-trained encoders? ImageNet pre-trained models such as ViT / DeiT are also popular, which share the same architectures as in the other categories (language contrastive / self-supervised pre-trained) and thus are easy to be compared.

We agree that there are popular and well-performing ViT (and ViT variations) models trained on ImageNet classification. While these models are architecturally more similar to other pre-trained and end-to-end trained models, we decided to instead include FocalNet since these models have recently been proposed and have been shown to outperform ViT-based models in the ImageNet benchmark [1].

Although the paper offered many insights and potential analysis from the emprical observations, the paper lacks enough decisive conclusions.

We thank the reviewer for their detailed thoughts. We revised the description of the discussed conclusions and point the reviewer to the colour-coded parts in the revised paper.

The conclusions are drawn without controlling some critical variables. For example, the effect of network size is overlooked in the paper.

End-to-end visual encoders mostly have a comparable number of parameters and therefore do not allow clear conclusions about network size, but we evaluate three models of varying sizes for CLIP, DINOv2 and FocalNet. For the model size of these pre-trained encoders, we do not observe any clear relationship with online performance. While larger DINOv2 models perform best in Minecraft, the same trend does not hold for CLIP and FocalNet where encoders with fewer parameters perform better. Results for Minecraft Dungeons do not allow any clear conclusions either. The same inconclusive trend continues for pre-trained visual encoders when reducing the amount of training data. Therefore, at least for pre-trained visual encoders, we can confidently say that the number of parameters does not seem to be a clear indicator for online evaluation performance. We included a brief description of these conclusions to our revised paper (see colour-coded sentences).

How about unfreezing the visual encoders during training? It may address the last point I raised as the visual encoder can adapt to new input size during fine-tuning.

We refer the reviewer to Section “Fine-tuning of Pre-Trained Visual Encoders” in our common response. In short, fine-tuning pre-trained visual encoders would come at a considerable computational cost that is not feasible at the breadth of our empirical study.

In the last paragraph of section 5.5, it states that the pre-trained visual encoders fail to generalize when the input image size shifts. …

We agree with the reviewer that our experiments in CS:GO allow no clear conclusions as to why pre-trained visual encoders fail. To investigate our hypothesis that the image processing might be detrimental to DINOv2, we compare the performance of four pre-trained visual encoders (one for each of the four families CLIP, DINOv2, FocalNet, Stable Diffusion) with three different types of image processing in Minecraft. For details, we refer the reviewer to Appendix E which describes the conducted experiment and findings. In summary, we do not find any evidence that image processing, as applied in CS:GO, is detrimental to the performance of pre-trained visual encoders in Minecraft. This raises the open question of why pre-trained DINOv2 visual encoders are not effective for decision making in CS:GO, in contrast to Minecraft Dungeons and Minecraft. We hope this paper raises awareness of this unique challenge to the ICLR community and will open source all required code to reproduce and build on this result.

For example, there are thousands of tasks in Minecraft as shown by [1], and this paper only tested on the "Treechop" task.

We acknowledge the simplicity of the Treechop task in Minecraft but would like to highlight that the goal of our study is to cover a range of video games. We refer the reviewer to Section “Evaluation Tasks” in our common response where we discuss the selection of tasks more.

We thank the reviewer for their comments and suggestions. Below, we will address the main comments raised by the reviewer.

This paper is more of a baselines paper comparing existing methods. For a baselines paper, I would expect far more extensive experiments across domains and methods.

We would like to highlight that our experiments evaluate a total of 22 different visual encoder configurations across two tasks, each model being evaluated across three seeds in each task, with a partial set of visual encoders being considered in a third task. We believe our evaluation already covers a wide range of commonly used visual encoders. Furthermore, we outline our reasoning for the selection of existing tasks and their complexity in Section “Evaluation Tasks” of our common response.

The domains considered here, while they are “modern video games”, are quite limited. E.g. for Minecraft they only consider the treechop task, which is the most basic thing one can do in Minecraft

and

How do these methods compare in more domains? I would also expect experiments in simpler domains like e.g. atari, coinrun, maybe robotics environments.

We acknowledge the simplicity of the Treechop task in Minecraft but would like to highlight that the goal of our study is to cover a broad range of modern video games. We refer the reviewer to Section “Evaluation Tasks” in our common response where we discuss the selection of tasks more.

Regarding the suggestion to run simpler domains like Atari or Coinrun, we would like to highlight that, first, the focus of our work is on modern video games that go beyond Atari (as stated in the abstract). Second, Atari and Coinrun both offer a programmatic interface and are therefore common environments for reinforcement learning. In contrast, our work focuses on imitation learning in video games where no such interface is available. Third, imitation learning in Atari is already a well-studied problem with a plethora of prior work [1-6]. For Coinrun, few work consider the application of imitation learning due to the lack of datasets of (human) demonstrations. Existing work therefore trains policies from demonstrations collected by reinforcement learning agents [7] which does not align with our focus on imitating human gameplay.

Lastly, our work already acknowledges and cites an existing empirical study on the application of imitation learning with varying visual encoders in robotics environments [8]. Given this study, we focus on the settings of video games which are meaningfully different from robotics since robotics tasks often represent the current state using images akin to the real world. In contrast, video games often have highly stylised visual representations (e.g. Minecraft and Minecraft Dungeons) which are notably different from the real-world images many pre-trained visual encoders are trained on. This represents a significant shift in distribution of encountered images which raised the research question whether visual encoders pre-trained on real-world images can be effective in video games. Our study clearly answers this question affirmatively and provides clear evidence as to which pre-trained visual encoders should be considered for application in complex video games.

auxiliary objectives for representation learning are quite common in reinforcement learning. Given this paper is studying the efficacy of different image encoders, it would seem natural to me to also include auxiliary self-supervised objectives into the end-to-end experiments

As the reviewer rightfully states, these auxiliary objectives are commonly applied in reinforcement learning. However, we train agents purely with imitation learning in domains where no reinforcement learning signal is available. In our evaluation domains, there exists no or only a limited programmatic interface which does not provide all necessary information used in many of these reinforcement learning auxiliary objectives so these approaches are not applicable to our setting or have not been established yet to serve as additional training objectives in our setting.

the authors hold pretrained encoders fixed. It seems natural to also comp

We infer that the reviewer meant to suggest to fine-tune pre-trained visual encoders. In this case, we refer the reviewer to Section “Fine-tuning of Pre-Trained Visual Encoders” of our common response in which we illustrate why we have not considered fine-tuning pre-trained visual encoders in our study. We kindly ask the reviewer to clarify if they meant to make a different suggestion.

We thank the reviewer for their comments and suggestions. Below, we will address the main comments raised by the reviewer.

It is not clear what kinds of image augmentation tricks are used

As stated in Section 3.2, we apply the same image augmentations also applied by VPT [1]. To further clarify, we added a description of the applied image augmentations in Appendix A.1 in the revised paper.

Why the image augmentation method is specific to the game?

We apply the same image augmentation in each game. We find image augmentations to be effective in Minecraft Dungeons and harmful to training in Minecraft which might be surprising but is consistent with prior findings where image augmentations can significantly improve the robustness of decision making agents [2, 3] but also harm performance if it perturbs images too much [4].

Why a pre-trained model (DINOv2) is better than the others? It lacks deep discussions.

We acknowledge that from our experiments, it is difficult to identify the reason why some pre-trained models work better than others. We hypothesise that the self-supervised training objective of DINOv2 leads to more general embeddings than reconstruction (tries to recreate all pixels rather than embed high-level information), classification (tries to only embed information related to predicted ImageNet classes), and language-contrastive learning (embed information correlated with language). This might be particularly important in domains, such as video games, which both visually and conceptually differ notably from the real-world images these models are typically trained on. Language descriptions, classification labels, and reconstruction might be too focused on embedding concepts important in the real world, self-supervised objectives might be more agnostic and, thus, effective in domains which differ from the real world such as video games.

Provides rollout videos for better understanding. Rollout videos are very helpful for readers to understand the challenges of the environments and the effectiveness of the model. It is strongly recommended to include some videos in the supplementary materials.

We thank the reviewer for the suggestion to release rollout videos of agents. These videos can now be accessed at the following anonymous repository: https://anonymous.4open.science/r/imitation_learning_in_modern_video_games_rollouts-4C48/README.md

Insufficient evaluation tasks in the Minecraft domain. In Minecraft, the "Treechop" task is the most basic and simple task. Although it is an important benchmark, however, conducting experiments solely on this task is not enough. It is better to include 2-3 extensive tasks, such as "Hunt animals", "Craft crafting_tables", and "Mine ores", to enhance the soundness.

We acknowledge the simplicity of the Treechop task in Minecraft but would like to highlight that 1. our goal is to cover a range of visually and conceptually diverse modern video games (breadth) rather than focusing on few challenging tasks in a single game (depth), and that 2. Minecraft Dungeons in particular represents a highly challenging decision making task. For a more detailed discussion of this topic, we refer the reviewer to the section “Evaluation Tasks” in our common response.

[1] (VPT) proposed an important foundation model for decision-making in Minecraft, which was trained on large-scale YouTube gameplays with behavior cloning. It yields a good vision encoder that is specified in the Minecraft domain. Although it is cited in the paper, it does not participate in the comparison. I suggest the authors to compare VPT in the experiment. [2] (SAM) is a large-scale pre-trained segmentation model, which has demonstrated strong cross-domain recognition capability. It should be included as a baseline.

We refer the reader to the sections “Discussion of Related Work” and “Segment Anything Pre-Trained Visual Encoder” in our common response where we contrast VPT to our evaluation setup in more detail and discuss the applicability of SAM to our setting. In short, we do not believe that comparisons to VPT would be fair or informative. For Segment Anything, we agree these models appear promising but unfortunately we found its inference cost to be too high for our tasks in which agents have to take actions in real-time.

We thank the reviewer for their comments and suggestions. Below, we will address the main comments raised by the reviewer.

It seems that the task is very simple, such as chopping trees in Minecraft, which is a fairly straightforward task. The author's conclusion is that there is no significant difference between various visual encoders and input image resolutions. However, due to the simplicity of the task, this conclusion is unreliable. Evaluating models like CLIP and DINO on such simple tasks does not effectively demonstrate the differences between modern vision transformers and CNNs. I strongly recommend that the author choose more challenging tasks, such as MineRLObtainDiamondShovel-v0 or MineRLBasaltBuildVillageHouse-v0 etc.

We acknowledge the simplicity of the Treechop task in Minecraft but would like to highlight that 1. our goal is to cover a range of visually and conceptually diverse modern video games (breadth) rather than focusing on few challenging tasks in a single game (depth), and that 2. Minecraft Dungeons in particular represents a highly challenging decision making task. For a more detailed discussion of this topic, we refer the reviewer to the section “Evaluation Tasks” in our common response.

In time-series decision-making tasks, the memory of historical states is crucial for making decisions. For example, VPT uses a transformer to record a history state of 128 frames, while the paper only utilizes LSTM to capture a limited number of frames. This can have negative implications for completing long-horizon tasks using behavior cloning.

We agree with the reviewer that memory of historical states can be crucial for decision making in some long-horizon tasks. However, we would like to disentangle the episodic length of tasks and the memory agents need to solve a task. Some long-horizon tasks require very limited or no memory to solve. For example, DQN was able to achieve superhuman performance in many Atari games without any memory, some of which feature relatively long episodes [1].

Furthermore, we disagree that LSTMs are only able to capture a limited number of frames since they can learn to keep information in its hidden state (memory) throughout an episode as long as it provides value. Impressive prior decision making achievements in long-horizon tasks have been achieved using LSTM models [2]. In contrast, a transformer is always limited to the context of its input frames (dependent on the used context length)

To further stress the relevance of agents maintaining history in our evaluation tasks, we highlight that the Minecraft Dungeons task requires agents to take up to 3,000 actions within a single episode (5mins at 10Hz). We believe that the ability of agents to effectively act in such a long-horizon task demonstrates that agents with LSTMs are sufficient to complete many long-horizon tasks in modern video games.

Recently, the popular technique of Segment Anything has achieved better results in various visual tasks. The author can further compare this model to explore its potential.

We agree that Segment Anything appears to be a promising pre-trained visual model. However, its inference cost makes it infeasible for our tasks in which agents have to take actions in real-time. We refer the reviewer to the section “Segment Anything Pre-Trained Visual Encoder” of our common response for more details.

In the conclusions shown in Table2, the best model of tree pruning only has a success rate of 32%, much lower than VPT's nearly 100%. Does this imply that the visual encoder is not actually the most important module in game playing?

We refer the reader to the section “Discussion of Related Work” in our common response where we contrast VPT to our evaluation setup in more detail.

Furthermore, we politely but strongly disagree that our best models performing notably worse than VPT implies that the visual encoder is not the most important module in game playing. There are many differences between our evaluation setup and VPT (the most notable ones being the scale of data and models, as well as agent architecture) which all confound the comparison, so no such conclusions can be drawn from the discrepancy in results in our paper and VPT.

Some relevant work has not been cited

We thank the reviewer for pointing out these recent, concurrent related works and included these works in Section 2 on related work. We further refer the reviewer to the section “Discussion of Related Work” in our common response where we discuss the work on open-world multi-task control in more detail. Lastly, we would like to point out that the most recent version of 1. and the listed paper 3. were made publicly available on arXiv after the ICLR submission deadline.

We thank the reviewer for their prompt response but, with respect, we firmly disagree with both of their criticisms for the following reasons.

Environment Selection

We notice that the reviewer’s criticisms solely focus on our task selection in Minecraft and would like to highlight that we also evaluate in CS:GO and Minecraft Dungeons - the latter representing an entirely new environment for imitation learning. Minecraft Dungeons is not a variation or task in Minecraft but a completely different game with distinct game mechanics (focus on navigation and combat) and visual perspective (isometric top-down view in contrast to first-person view in Minecraft and CS:GO).

To visualise the uniqueness and complexity of Minecraft Dungeons as well as the Minecraft and CS:GO environments, we would like to point the reviewer to our released rollout videos: https://anonymous.4open.science/r/imitation_learning_in_modern_video_games_rollouts-4C48/README.md

Comparison to VPT

We strongly disagree with the reviewer’s comparisons to VPT and their statement that VPT has the “same structure”. We would argue that adding comparisons to VPT, which would only be possible in Minecraft since VPT has been specifically designed and trained for this environment, would provide no additional value for our study. Given the many differences between VPT and our models, VPT outperforming our models in Minecraft provides no clear insight in addition to our findings.

To expand on our common response, we would like to highlight the following differences:

• >100x parameters: VPT’s main model has 500M parameters vs our model having ~4M (in the policy) + ~94K (in the referenced Impala CNN encoder) parameters
• ~100,000x training data: VPT uses ~70,000h of (filtered) training data vs ~45min training data for our Minecraft experiments
• >3,000x training budget: VPT training uses 6480 GPU days (720 GPUs for 9 days) vs the training of the Impala CNN model using 2 GPU days (a single GPU for less than 2 days)

Even if we focus on the ResNet architecture of the VPT policy model alone, the main VPT model’s ResNet has 40M parameters (the smallest variation has 10M parameters in the ResNet). In comparison, the Impala CNN model highlighted by the reviewer has ~94K parameters. The ResNet encoders considered in our study, which are architecturally more comparable to VPT’s ResNet, only have ~580K parameters which is still substantially less than the ResNet used in VPT.

The insights of our study are not conflicting or in competition with VPT, which serves as a foundation model for decision making in Minecraft, but provide additional and orthogonal findings about the impact of visual encoders for decision making in complex video games more broadly.

Code Quality

Given the significant differences between VPT and our Impala CNN, we disagree with the assessment that the poor performance of the Impala CNN model suggests possible errors in our code. We will open source the code to enable reproducibility and assure the reviewer of the quality of the code.

Segment Anything Pre-Trained Visual Encoder

Reviewers yJPq and JgYZ suggested to additionally consider the pre-trained Segment Anything (SAM) [4] models. We agree with both reviewers that these models have demonstrated impressive generalisation capabilities. We considered including SAM pre-trained visual encoders in our study, but found these models to be too slow at inference time. On hardware available to us (including A6000, GTX 1080Ti, and V100 GPUs), computing embeddings with even the smallest SAM checkpoint took between 0.1 and 0.5 seconds which is too slow for decision making in video games. In CS:GO and Minecraft Dungeons, we aim to take actions at 16Hz and 10Hz, respectively, in real-time. This was not possible with pre-trained SAM models.

Hyperparameter Selection

Several reviewers asked how we selected hyperparameters for our experiments. Hyperparameters such as the learning rate, batch size, and model architecture configurations were determined in prior experiments in Minecraft Dungeons. To keep comparisons fair and focused on the visual encoders, we kept hyperparameters consistent in following experiments. We only reduced the total training duration in tasks which required notably less training (as seen in training loss curves in Appendix B) to limit the computational cost of experiments.

We prioritised minimising conflating variations besides the variation of visual encoders to draw informative conclusions from our experiments. Furthermore, to conduct an extensive hyperparameter search for each considered configuration of visual encoders and tasks would be infeasible for our study where we consider a total of 22 different visual encoder configurations across three tasks.

Revised Paper

Lastly, we would like to highlight several improvements made in the paper based on the reviewers’ feedback. For details, we refer to the revised PDF which includes all changes highlighted in orange colour. Below, we list the most notable changes:

• Prompted by our results in CS:GO, we hypothesised that the poor performance of DINOv2 could be explained by the image processing necessitated by the CS:GO dataset providing images at 280x150 resolution. Following this hypothesis and comments by reviewer bAy4, we now provide an evaluation of four pre-trained visual encoders in Minecraft (Appendix E), comparing the original online performance with the performance observed when alternating the image processing in two ways. We find that pre-trained visual encoders in Minecraft are more robust to alternative image processing as hypothesised from our CS:GO results, and incorporate these findings in Section 5.5.
• Appendix B now includes loss curves for the training of all models in all three games (the paper previously only included the training loss in Minecraft Dungeons).
• Following the suggestion of reviewer JgYZ, Appendix A now includes details on the image augmentation considered when training end-to-end visual encoders.
• Following the suggestion of reviewer bAy4, Sections 5.3 and 5.4 now include further conclusions on the correlation between the number of parameters of pre-trained visual encoders and the online evaluation performance. In short, we observe no clear correlation between the size of pre-trained visual encoders and the resulting online performance, suggesting that larger pre-trained visual encoders might not always be beneficial for decision making in video games.
• Following references provided by reviewers yJPq, JgYZ and bAy4, we incorporated recent work into Section 2 on related work.

Citations

[1] Kanervisto, Anssi, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyou Li, Jianing Shi et al. "Minerl diamond 2021 competition: Overview, results, and lessons learned." NeurIPS 2021 Competitions and Demonstrations Track (2022): 13-28.

[2] Baker, Bowen, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. "Video pretraining (vpt): Learning to act by watching unlabeled online videos." Advances in Neural Information Processing Systems 35 (2022): 24639-24654.

[3] Cai, Shaofei, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. "Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13734-13744. 2023.

[4] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

Discussion of Related Work

Several reviewers mentioned Video Pretraining (VPT) [2] and work on Open-World Multi-Task Control [3] which we would like to further discuss.

First, reviewers ask why VPT reaches higher performance than our trained models in Minecraft and suggest including VPT as a baseline. We would like to highlight that VPT trains a transformer model with 500M parameters on 70,000 hours of filtered IDM-labelled data using 720 V100 GPUs for 9 days. In contrast, we train LSTM models with ~4M parameters (excluding the visual encoder) on less than 1h of filtered data using a single A6000 GPU for a few hours.

Given the vast discrepancy in used data and model sizes, we believe that any comparison to VPT would not be fair since VPT is clearly expected to perform notably better. To identify the contribution of visual encoders to the decision making performance in comparison to the results of VPT, we would need to compare visual encoders at a comparable scale of model size and training data. However, such comparison is infeasible due to its computational cost.

Lastly, we would like to highlight that VPT focuses on Minecraft as an open-ended decision making benchmark, training only a few models at large scale. Instead, our study aims to demonstrate wider breadth by evaluating a total of 22 different visual encoder configurations across three modern video games.

Regarding the work on open-world multi-task control [3], we thank the reviewers for pointing us to this related work which we now include in Section 2. This work leverages the VPT agent to collect demonstration data in Minecraft. First, we agree that this approach of leveraging pre-trained models is interesting. However, we would like to highlight that it relies on released pre-trained agent models which are not available in most video games. In this case, the designed approach would only be applicable in Minecraft since no similar pre-trained agent is available in Minecraft Dungeons or CS:GO. Secondly, we would like to point out that the dataset of this work was only released three weeks prior to the ICLR deadline (as per Github commits) which did not allow us to consider the collected data in our experiments. Thirdly, the work points out the difficulty of multiple tasks in open-ended environments, such as Minecraft, being indistinguishable from each other. In light of this challenge, we believe that the selection of effective visual encoders can be complementary to the proposed approach, since these have to perceive and represent task-relevant and discriminative information across different scenes.

Fine-tuning of Pre-Trained Visual Encoders

Reviewers sq4s and bAy4 suggest to additionally fine-tune the considered pre-trained visual encoders in our evaluation study. We agree that fine-tuning these visual encoders would add valuable comparisons to the evaluation study. However, fine-tuning these models with, in some cases, hundreds of millions of parameters is computationally very costly.

To illustrate the computational cost, we would like to highlight that fine-tuning these visual encoders would not just require notably more compute (in particular VRAM) but also dramatically slow-down training. As outlined in Appendix G, our current experiments with pre-trained visual encoders first compute the embeddings with these frozen visual encoders for our entire dataset before training a policy network on these pre-computed embeddings. This process significantly speeds-up training (we found training to be 10-20x faster on pre-computed embeddings). Fine-tuning would prevent us from pre-computing embeddings for training since embeddings would change as the visual encoders are updated. At such computational cost, an empirical study of our scale (training 10 pre-trained visual encoders across three domains, each with three seeds) would not have been feasible.

Lastly, we would like to highlight that our work to study the efficacy of varying visual encoders (pre-trained and end-to-end trained) for imitation learning in modern video games is largely motivated by the prohibitive cost of training decision making agents in these domains. We believe that our findings, even in the absence of fine-tuned pre-trained encoders, provides significant value to the community and can serve as a starting point for anyone to develop capable decision making agents with limited resources.