Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

We thank the reviewer for their concise review.

Life Information - We do not use life information in any main paper results, as we specify in Section 4.1. Furthermore, we provide a detailed comparison of BTR with and without life information in Appendix I. If your low score was due to doubts about our empirical performance stemming from this matter, we hope you are willing to reconsider now that this has been clarified.

Comparison against baselines - We already compare against SoTa algorithms for Atari 200M in Table 3. We did not include such baselines in Figures 1 and 2 as we wanted to limit this comparison to those algorithms which can be run with widely accessible compute power. However, we are willing to replace Figure 1 with a walltime efficiency curve against Dreamer v3, as we feel this better conveys the message of the paper (MEME did not release this data). We can also add Dreamer v3 and MEME to Figure 2, with a note indicating their walltime to allow for a fairer comparison as we believe that with the appropriate context this will improve the paper. We do, however, argue against the inclusion of sample efficient algorithms (Atari-100K), as these algorithms use drastically different resources (100K frames vs 200M), and critically are benchmarked on a 26-game subset, rather than the full suite, making for an unfair comparison. If you believe this to be pivotal, we can include a figure in the appendix for the 26-game subset. Below is a comparison against SoTa algorithms on the mentioned 26-game subset:

*MEME used a shared server with a TPUv4.

**PQN used life information, making its results appear significantly higher.

***EfficientZero was tested on a server with 8 RTX 3090s, not an A100.

Practical Implications of RL on Desktop PCs - Could you further clarify this point? We are unsure exactly what you are asking.

[1] Gallici, Matteo, et al. "Simplifying deep temporal difference learning." arXiv preprint arXiv:2407.04811 (2024).

Thank you for your detailed and constructive review, and for your appreciation of the evaluation on unique environments.

Comparison against baselines - We already provided a discussion on SoTa algorithms for Atari 200M in Table 3, and two of the listed algorithms (BBF and EfficientZero V2) are sample sample efficient algorithms which only used 100K frames rather than 200M. Furthermore, they were never benchmarked on the full Atari set, but rather the reduced 26 game sample efficient benchmark. Therefore, we cannot include these in Figures 1 and 2. We did not include MEME and Dreamer v3 in Figures 1 and 2 as we did not want to present an ‘apples to oranges’ comparison, since these SoTa algorithms are inaccessible to our targeted users due to heavy compute requirements. However, to avoid any confusion around BTR’s performance, we are willing to replace Figure 1 with a walltime efficiency curve that includes Dreamer v3, as we feel this better conveys the message of the paper. We can also add Dreamer v3 and MEME to Figure 2, with a note indicating their walltime to allow for a fairer comparison as we believe that with the appropriate context this will improve the paper. If this is your main critique of the paper, given that this is simple to remedy, we hope it may be grounds for improving your score.

*MEME used a shared server with a TPUv4.

**PQN used life information, making its results appear significantly higher.

***EfficientZero was tested on a server with 8 RTX 3090s, not an A100.

More Baselines for Wii Games - As you mention, unfortunately this may not be possible during the rebuttal period. Out of interest, what algorithm would you like to see? We found other SoTa algorithms such as Dreamer v3 to be too computationally expensive to run on more demanding environments.

[1] Gallici, Matteo, et al. "Simplifying deep temporal difference learning." arXiv preprint arXiv:2407.04811 (2024).

We thank the reviewer for their review and their appreciation of how our work will be useful for both researchers and practitioners.

Steps vs Walltime – We feel the reviewer has somewhat misunderstood the main point of this paper. As stated in the title, abstract, and contributions, the purpose of our work is to create a high-performance algorithm that is computationally accessible—something that is poorly measured by environment steps. For example, while BTR takes approximately 12 hours to process 200 million frames, Dreamer v3 takes almost 8 days to do the same and requires considerably more RAM and GPU memory, illustrating that environment steps are a poor metric for computational accessibility.

Use of the ALE – The ALE remains the most widely used RL benchmark for good reason, and we feel it is being unfairly dismissed here. The standard evaluation protocol for the ALE [2] (which we follow) includes both no-op starts and sticky actions, which are explicitly intended to prevent agents from exploiting determinism in the environment. Moreover, the ALE gained its popularity due to its extremely diverse set of tasks, a strength we believe is being overlooked. Other SoTa algorithms, such as MEME [3], benchmark exclusively on Atari for this very reason. The ALE also contains numerous sparse-reward environments, which can be found in our Atari-60 results.

Bump in Figure 1 – As discussed in Appendix F, this bump is due to epsilon-greedy exploration being disabled partway through training. While we found exploration beneficial early in training, disabling it halfway through led to improved performance.

[1] Schwarzer, Max, et al. "Bigger, better, faster: Human-level atari with human-level efficiency." International Conference on Machine Learning. PMLR, 2023.

[2] Machado, Marlos C., et al. "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents." Journal of Artificial Intelligence Research 61 (2018): 523-562.

[3] Kapturowski, Steven, et al. "Human-level atari 200x faster." arXiv preprint arXiv:2209.07550 (2022).

We thank the reviewer for their very thorough and constructive review, which clearly took a great deal of time and effort. We appreciate that the reviewer clearly understands the value of producing an accessible and high-performance algorithm.

Complexity of Wii Games - We would like to argue that in particular, Mario Kart Wii is a very challenging environment, more so than it has been given credit for. This environment contains 11 other racers and the use of randomized items, introducing a very high degree of stochasticity into the environment, far beyond the stochasticity introduced in Atari. Furthermore, we tested the agent on the track “Rainbow Road”, commonly known as the game’s hardest track, taking over a minute even for a single lap, and featuring a very complex and noisy observation space (fully 3D, with multicoloured roads and constantly changing 3D backgrounds that resemble the track). Although Super Mario Galaxy uses a single fixed level, we would argue this is unique and more challenging than typical benchmarks due to complex movement mechanics and high-resolution images.

Claim of SoTa for non-recurrent RL - While non-recurrent RL may not be a category of interest, we chose this phrasing as it is indicative of walltime efficiency, since off-policy recurrent algorithms (particularly image-based ones) are very expensive [1]. We thought this better than stating “SoTa for walltime efficiency”, or “SoTa for Desktop PCs”, as these are somewhat tricky to formally verify (though we believe it is important to the field). As you mention, strong baselines heavily rely on expensive recurrent models.

Dreamer v3 and MEME baselines - As you mention, we already included these baselines in Table 3; we did not include these results in Figures 1 and 2 as it would be misleading to compare BTR against algorithms which are inaccessible to our targeted use case (academics, hobbyists, etc). However, to be more transparent about BTR’s place among these algorithms, we would be happy to add Dreamer v3 and MEME to Figure 2 (the box plot), while also stating the walltime required for these algorithms to emphasize the difference. As for Figure 1, we can replace this with a walltime efficiency curve that includes Dreamer v3 as we feel this better aligns with the paper’s contributions (MEME, however, did not release sample efficiency curves to our knowledge). We didn’t include PQN in any main paper figures/tables as they used Life Information, which we discuss at length in Appendix I, and leads to completely non-comparable results and is not standard (Table 3 also uses Atari-60, not Atari-5, and we didn’t perform experiments on Atari-60 using life information).

Wii Game Results – As you mention, the setting is somewhat informal, and thus we find it difficult to make direct comparisons to other benchmarks. We did not intend to use these environments as formal benchmarks (we already rely on Atari and Procgen for that purpose), but rather as a demonstration of what BTR is capable of. Furthermore, we have been unable to find any other work on Super Mario 64 that uses a reinforcement learning approach, and we would be interested if you could reference such work. As for Super Mario Kart, we would like to highlight the difference in complexity between it and Mario Kart Wii, as the latter includes far more complex graphics, tracks, and driving mechanics (e.g., mini-turbos, wheelies, tricks, etc.).

Missing References – We are happy to add the references you suggested to our section on Procgen to provide a more comprehensive discussion. We also found some of the work you referenced [2] very interesting, and it may help improve BTR’s performance on the Procgen benchmark.

Comments and Suggestions – We are happy to address the minor adjustments you suggested in points 1, 3, and 5 of this section. As for point 2, we also considered adding LayerNorm, though it may introduce inconsistencies across the ablations and other figures, and could complicate the narrative of the paper. Regarding point 4, we are currently running additional experiments for the VizDoom environment, which we will add to the appendix.

Questions – We define consistent completion as achieving over a 90% success rate, and we will add this definition to the caption of Figure 3. We believe your remaining questions have been addressed throughout our response.

[1] Kapturowski, Steven, et al. "Recurrent experience replay in distributed reinforcement learning." International conference on learning representations. 2018. [2] Jesson, Andrew, and Yiding Jiang. "Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale." arXiv preprint arXiv:2410.10905 (2024).