Beyond The Rainbow: High Performance Deep Reinforcement Learning On A Desktop PC

We thank you for understanding our core motivation for this work and appreciating what it achieved.

Error Bars

In this work, we computed the error bars using the evaluation episode variance rather than as proposed in [1] with the seed variance (for the mean evaluation performance). Using [1], for the Atari-5, we trained BTR across 3 seeds, finding IQM scores of 7.77, 8.22, and 7.86. However, due to time and computing limitations, we only trained BTR for 1 seed on Atari-60 and ProcGen, meaning we can’t use [1] for error bars. If the reviewer agrees, we will train BTR for Atari-60 and ProcGen with three seeds, allowing us to update the error bars. However, due to the computational and time requirements of these experiments, we could not provide these results within the discussion period though they would be ready before the camera-ready version deadline.

Regarding Figure 2, we did not intend to claim that we achieved state-of-the-art performance compared to prior work. Instead, we claim that we achieved similar performance as the prior state-of-the-art with significantly fewer resources, in particular, walltime and learnable parameters. We will change our wording on Line 083 from “outperformed” to “similar” and Line 274 from “exceed” to “similar”. Does this address the reviewer's concern?

Comparison of prior work on Wii Games

When demonstrating BTR in these new environments, we aimed to showcase BTR’s capability in games that are not normally considered in RL research. Therefore, we didn’t compare to the prior works shown in Figure 1 that achieved significantly lower scores than BTR. If the reviewer believes this is a crucial comparison to include, we are willing to train a Rainbow DQN agent for the Wii games to include in Figure 3.

Ablation Study

We are happy to use human-normalized performance to measure each ablation’s performance and will update Figure 4. Regarding the regression procedure, we are not making claims about the performance of all Atari games from these ablations, rather, we used Atari-5 as a benchmark to better discriminate each ablation’s performance without needing to train on all Atari 60 environments to find similar results. Does this answer the reviewer’s concern?

Dormant Neurons

We really appreciate this comment as we ourselves are skeptical of dormant neurons as a measurement. However, we included it as it has become a popular measure in the literature [1,2,3]. Additionally, in Appendix F, we plot a histogram of activations showcasing the range of activations used by the agent presenting a better overview of the agent’s activations. If you feel these results should not be included, we will remove them at your request.

Claims of the state-of-the-art on a desktop PC

We recognise that this is a difficult claim to make; however, after surveying the literature, we believe, to the best of our knowledge, that it is true. Part of the reason we make this claim is that few algorithms achieve greater performance than BTR, and those that do, in their reported training requirements, make it clear that it would not be possible to train their agent on a desktop PC. In Table 3, we survey three top-performing algorithms, listing their claimed resources used and identifying several reasons for preventing them from being a fair comparison to a desktop PC: the required resources with numerous CPUs / GPUs, excessive training time over 5 days or more recently GPU VRAM to train a model (commercial GPU have significantly less VRAM than research-grade GPUs). Is the reviewer aware of any training algorithm that could reasonably compete with our claim?

[1] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[2] Xu, Guowei, et al. "Drm: Mastering visual reinforcement learning through dormant ratio minimization." arXiv preprint arXiv:2310.19668 (2023).

[3] Qin, Haoyuan, et al. "The Dormant Neuron Phenomenon in Multi-Agent Reinforcement Learning Value Factorization." The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Thank you for this detailed review with productive and interesting questions.

Computation and using hardware accelerated environments

We considered using hardware-accelerated environments, which provide several advantages to researchers, particularly in parallelisation and training speed. However, it is not clear how to hardware-accelerate many real-world problems, e.g., Wii games tested in this work. As we are interested in expanding and investigating RL in a broad variety of challenging environments, we focused on classical CPU-based environments.

We also want to clarify that 12 hours is the time to train an agent for a single environment.

Different Domains

Yes, as stated in Section 4.2 (lines 285 to 287), we use the same algorithm hyperparameters for all domains (Atari, Procgen and Wii games). The only thing we changed is the input resolution (140x114 from 84x84) as Dolphin Emulator’s native resolution is much higher, and the number of parallel environments as we found Dolphin was too memory-expensive for more than 4 on a single 64Gb desktop. To counteract this, we took 16 steps (in all 4 environments) before performing an update, which gives us a similar effect to taking 64 steps in parallel, meaning that no additional tuning is required.

Large Shaded Areas on Plots

The shaded areas are the standard deviation across all evaluation episodes and seeds. For example, with 100 evaluation episodes and 3 seeds, the shaded area is the standard deviation over all 300 episodes. This tends to be high when performing evaluations due to the surprisingly stochastic nature of atari games (i.e., sticky-actions, no-op starts and long episode lengths), increasing the score variance in a single episode.

BTR’s IQM on Atari-5 for each seed is 7.77, 8.22, and 7.86

Rainbow with Vectorization

Adding vectorization to Rainbow is certainly possible, however, it would require extensive tuning to provide a fair and unbiased comparison as BTR’s parameters were tuned to use vectorization (e.g., learning rate, batch size and how often to replace the target network). We are, however, still happy to provide an untuned comparison if you believe this is important.

Comparison to “Simplifying Deep Temporal Difference Learning” (PQN)

We think this is a really interesting comparison, and thank you for raising it. Sadly, PQN does not report their full results for 200M frames and importantly uses the life information parameter that alters the environment to provide more regular termination signals, which has been found to significantly improve training. This makes it an unfair comparison to previous works that do not use it. In Appendix J, we evaluate BTR with and without life information so we can provide a fair comparison. See the table below. Additionally, we note that even without life information, at 200 million, BTR still achieves a higher IQM performance at 7.42 than PQN for 400 million with 3.86 across the Atari-5 environments.

Atari-5 IQM and per-game Scores. For individual games, Human-Normalized scores are reported, with the raw score in brackets.

Beyond performance, the largest difference is in wall time for several reasons. First, we use Gymnasium async rather than EnvPool, which is 1.6x faster [1]. Second, we do not use PyTorch compile or cudagraphs, which have both recently been shown to significantly speed up reinforcement learning code which are the equivalent of jax.jit [2]. Incorporating them should reduce BTR’s walltime. Third, using a large replay buffer will mean that off-policy algorithms like BTR will be slower from repeatedly accessing RAM than on-policy algorithms like PQN or PPO that can be stored in GPU VRAM. On-policy and off-policy algorithms tradeoff sample efficiency and walltime, which we believe the performance impact is worthwhile for BTR.

It is noteworthy that some of BTR’s improvements could be combined with PQN to improve performance, particularly in relation to the neural network architecture. We agree that a comparison against PPO in terms of performance and walltime would be interesting, and we will add one in the appendices of our paper.

[1] Weng, Jiayi, et al. "Envpool: A highly parallel reinforcement learning environment execution engine." Advances in Neural Information Processing Systems 35 (2022): 22409-22421.

[2] https://github.com/pytorch-labs/LeanRL

Thank you for your diligent and detailed response, which summarizes the paper’s strengths and raises productive criticisms.

Details of Adaptive Maxpooling

Thanks for spotting this. We will update the paper to clarify Adaptive Maxpooling and how it works.

Adaptive Maxpooling is identical to the standard maxpooling layer, except the output shape is an argument with the kernel size and stride automatically adjusting for different input resolutions, enabling support for different resolutions with no algorithmic change or needed learnable parameters. Our code simply uses the PyTorch Adaptive Maxpooling 2D layer with a maxpool size of (6, 6) as a drop-in replacement for standard maxpooling.

Citations and comparison against other work

In an earlier draft, we included a comparison against Schmidt and Schmied, 2021 in Figure 4, however, after some investigation found that they used a “life information signal” when evaluating their results. As demonstrated in Appendix J, this significantly impacts performance and is not recommended for evaluation by Machado et al., 2018 [1]. Because of this, we decided not to include these results in the main paper as we believed it would confuse readers as to the reason for the performance difference, however, we did still include a comparison in Appendix A.

With regard to your requested citations, we think these are strong relevant additions to the paper and have added them in Sections 3.1 and 3.2, respectively.

Adam’s Epilson Formula

The parameter 1.95e-5 for Adam’s epsilon comes from Schmidt and Schmied, 2021. We will add a reference to this. While the value was correct, we apologize for a minor mistake in our formula, which should have been 0.005 / batch_size, as opposed to 0.05 / batch_size.

BTR’s hyperparameters on different domains

For new domains, e.g., the Wii games, we did not do any further hyperparameter tuning, keeping the same set across Atari, ProcGen, and Wii games. We wouldn’t be surprised if further improvements could be made to individual environments with further tuning, but we aim to produce a single algorithm that works across many domains, which is in line with DQN’s original motivation. In terms of the most important parameters, we found the learning rate, discount rate, and N (from N-Step TD learning) to be the most significant.

Using A High-End Desktop PC

We agree about specifying desktop components and will specify them in the introduction. We would like to note that research servers with an Nvidia A100, a four-year-old GPU model, cost over \$10,000 each, compared to an equivalent high-end desktop used in this research, which costs less than \\$5,000 in total. Additionally, with future consumer graphics cards, we anticipate the cost of building similar desktops will decrease, making this research more accessible.

Ethical issue with regards to using Wii Games

On the copyright-based and emulator question for using Wii-based games, we view this as similar to the use of Atari games in RL research, where the ROMs are similarly under Atari's copyright, though they are not viewed as an ethical issue for research. Likewise, we have bought the ROMs for the respective games used and are not distributing the ROMs with the research, so we do not believe this is an issue. If the reviewer is unsatisfied with this response, could they clarify in what way they believe there is an ethical issue here that does not exist for RL research using the Atari games? Furthermore, we have discussed using the Dolphin Emulator with the developers, who have cleared us for use in research.

[1] Machado, Marlos C., et al. "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents." Journal of Artificial Intelligence Research 61 (2018): 523-562.

Thank you for your insightful review, which succinctly outlined our paper’s contributions, results, and analysis.

In answer to where the novelty in this work comes from, we identify three sources:

• The process of choosing, integrating, and testing components together is a time-consuming process that requires diligence. In addition to the six improvements included in BTR, several more promising extensions were considered and investigated, though dismissed (see Appendix I for more detail). In publishing the paper, we hope this will prevent future researchers from recomputing the same tests conducted in this paper. While integrating these components, we found it necessary to re-tune hyperparameters, especially when using vectorization with components designed to be used with a single actor (e.g., batch size, learning rate, and frequency of updating the target network).
• We analyze the components across several measures beyond performance, demonstrating their different and sometimes competing effects on action gaps, policy churn, and observational noise. Such analyses have been completed previously, though not across as many measures or as varied improvements.
• We demonstrate BTR beyond the standard testing suite to Wii games, showcasing BTR’s continued strength as an algorithm outside of standard research testing environments. This includes games with far more graphically intensive and challenging domains than what any other algorithm (with similar compute resources) has been shown to handle.

On the challenge of integrating these components, we found it technically complex to simultaneously support Impala, Spectral Normalization, Maxpooling, Dueling Networks, Noisy Networks, and IQN, which has never before been attempted. An additional complexity was implementing a replay buffer that supports vectorization, N-Step TD Learning and Prioritization in a memory-efficient way. This approach stores each frame individually rather than all four frames together, reducing memory overhead from 28GB to 7GB in total for a million Atari observations. To help future researchers and hobbyists, we open-source our code.

We are happy to further elaborate on any of these at your request.

Thank you again to all the reviewers for their time and diligence. In the updated paper we have addressed all the reviewers' concerns, now proving statistically significant performance improvements over even more algorithms. In publishing, we hope this research will be able to increase the wider accessibility of high performance RL.

Following the reviewer’s comments, we have made the following changes, which can be found in our updated PDF.

• We have updated all relevant figures in the paper to use 95% confidence intervals in accordance with the current research standards from RLiable. For Figure 2 and Table 2, we now use 2 and 3 seeds, respectively.
• For Figure 3 with the Wii games, we have added a baseline for Mario Kart with Rainbow DQN, achieving a score of 3.4 at 155 million frames compared to BTR’s score of 54 at the same point.
• We have added a more detailed description of Maxpooling and citations for Spectral Normalization and Hyperparameter tuning, as requested by Reviewer CULH.
• We have clarified “High-end” PC in both the abstract and introduction, as requested by Reviewer CULH.
• We have updated Appendix C2 with the correct formula for Adam’s Epsilon.
• In Appendices A and B, we added comparisons to PQN and PPO as requested by Reviewer GWAQ.
• Figure 4 now includes a comparison to Rainbow + Vectorization, as requested by Reviewer GWAQ.
• Appendix B includes the Optimality Gap, Performance Profile and Box Plots for BTR from RLiable to be at the highest standard of RL research.
• Figure 6 has been moved to the appendix, and some claims about plasticity have been removed due to concerns raised by Reviewer 2n46.

Additionally, we are still running BTR for Atari-60 for an additional two seeds. Due to the confidence interval statistics, we expect this to significantly shrink the confidence intervals in Figure 1 to match closer to the error bars in Figures 2 and 3. Lastly, we will also provide Rainbow DQN comparisons for all Wii games.

We look forward to hearing any more comments regarding the updated PDF.