A Large-Scale Analysis on Methodological Choices in Deep Reinforcement Learning

1. ”One claim in the paper is that the distributional RL may harm early learning, however, comparing C51 to Dueling is not a direct comparison, since it introduces a new technique. A fairer comparison would be C51 + Dueling to Dueling or C51 to DQN.”

---

Please note that this comparison is already currently present in our paper. DER is essentially C51 + Dueling, and the results in Figure 1 and Figure 5 demonstrate that indeed dueling substantially outperforms C51 + Dueling.

---

2. ”How much data is the high data regime results in Figure 1 (right) based on? Since these results don’t seem to correspond to the original paper results.”

---

Figure 1 reports low data and high data regime results which correspond to 100K for low data and 200 million for high data. The results of Figure 1 (right) indeed correspond to the original paper results. This is also explained between line 33 and 38 in the introduction.

---

3. ”Line 416 talks about the performance of DRQ vs Dueling, where are these numbers coming from? (Table or figure?)”

---

The results are reported in Figure 5 and this is explained starting from Line 375.

---

4. ”In Figure 1, shouldn’t dueling be attached to several of these methods, such as Rainbow, OTR, DER, and DrQ, which all use the dueling architecture?”

---

Please read the caption of Figure 1 where it states "A directed arrow from Algorithm A to Algorithm B means that the algorithm B provides comparison against the algorithm A as a baseline."

Thus, Figure 1 highlights that all of these methods failed to provide a comparison to the baseline dueling algorithm even though some of these methods were even built on top of it, read Line 412 and 415. Hence, a baseline algorithm that predates these recent algorithms in fact outperforms these recent algorithms designed for the low-data regime.

---

5. ”Line 507 states “Our paper demonstrates that several baseline Q algorithms perform better than a line of recent algorithms claimed to be the state-of-the-art” but the results only show that Dueling DQN outperforms some other DQN-based algorithms designed for the full Atari setting, in the Atari 100k setting.“

---

This statement you make here is incorrect. The results reported in Figure 1 and Figure 5 further demonstrate that dueling outperforms a wide portfolio of algorithms designed for the low-data regime.

---

6. ”It’s unclear to me what the incorrect conclusions made by prior work are exactly, could you expand on this point? (line 83)”

---

These incorrect conclusions are extensively explained in the empirical section of our paper. While there are many instances of these incorrect conclusions that are extensively discussed in the main body of our paper, here you can find an itemized form of one of these incorrect conclusions.

I. The DRQ paper uses the dueling algorithm and proposes a training method on top of the dueling algorithm.

II. The DRQ paper, although based on dueling, does not provide direct comparison to dueling.

III. Our paper demonstrates that the baseline dueling algorithm performs better than the DRQ algorithm.

Thus, an algorithm that is built on dueling and is proposed to improve performance, in fact lowers the performance of dueling. Thus, our paper explicitly investigates and analyzes these methodological choices that lead to incorrect conclusions.

---

7. ”I’m not sure I agree that the experiments are “large scale extensive” as the results omit several notable modern algorithms for Atari 100k (e.g., BBF, EfficientZero, DreamerV3) and only examine a single setting.”

---

Please note that the algorithm proposed in BBF [1] is also SPR on top of Rainbow. Hence, this paper is also making the exact same implicit assumption that our paper describes, discusses and provides a deep concrete explicit analysis for.

Furthermore, we currently mention in the main body of our paper that these implicit assumptions are also made in EfficientZero as well, and note that the implicit assumptions and the methodological choices still persistently appear in a large body of even very recent work [3].

[1] Max Schwarzer, Johan Obando-Ceron, Rishabh Agarwal et al. Bigger, Better, Faster: Human-level Atari with human-level efficiency, ICML 2023.

[2] Mastering Atari Games with Limited Data, NeurIPS 2021.

[3] PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning, NeurIPS 2023.

Thank you for writing that the main claim of the paper is very smart and properly supported and our paper's claims are of great importance for the RL community.

---

1. ”For example, why define DRQ^ICLR instead of just citing a paper ?”

---

Because there is indeed a distinction between DRQ$^\text{ICLR}$ and DRQ$^{\text{NeurIPS}}$, DRQ$^{\text{ICLR}}$ refers to the original paper [1] and DRQ$^{\text{NeurIPS}}$ refers to the reproduced version of the DRQ algorithm by [2]. This is also explained in our paper in footnote 1 in Page 8.

[1] Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels, ICLR 2021.

[2] Deep reinforcement learning at the edge of the statistical precipice, NeurIPS 2021.

---

2. "Cite Arcade Learning Environment line 37.”

---

Currently, Arcade Learning Environment is already cited in Line 319 and 320. However, we can also cite in Line 37.

---

3. ”Is it really true that people make the assumption of monotonic data regime (lines 40-46)? Maybe you should add citations.“

---

Yes, it is true. This is explained and cited in the main body of the paper in the empirical analysis section. However, we can also further cite here too.

---

4. ”Can you confirm that your Human normalised scores are obtained by evaluating the learned policies (the Q-nets) outside of the training loops ?”

---

Yes.

---

5. ”What does canonical mean in your introduction?”

---

We use canonical in the exact same meaning with the prior work [1].

[1] Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Nature 2020.

Thank you very much for dedicating your time and preparing a well-thought out review, and thank you for referring to our paper as:

It is works like these that encourage other researchers in the community to often revisit fundamentals and without such works, often common misconceptions will have a long lasting effect, incorrectly misleading recent works, while issues existing in fundamental algorithmic principles.

---

1. ”I think the issue raised here are fundamental around deep Q algorithms, and not specifically to any distributional Q learning algorithms. Often the reference to distributional RL in the text seem confusing and presented as if the conclusions are only when comparing sample complexity regime for distributional Q algorithms. If I understand correctly, the paper presents the issues that are more generic to distributional RL - so it might be worth clearly stating that and not convoluting too much with distributional Q algorithms here. Please correct me if I’m wrong here.”

---

This is a great suggestion. Yes, the presented issue is indeed more general, we can definitely state this more clearly.

---

2. ”Experimental comparisons are often made between Dueling, IQN, C51 and others - referring to the comment in (2), if we do not present results specifically to distributional Q algorithms, would it be possible to also present basic Q algorithms (like DQN for example, or some variant of DQN) in the figures here? This would help put the paper in more context to addressing how the common misconceptions around performance and sample complexity exists widely - and that very basic deep Q algorithms may in fact still be good depending on the sample complexity regime being analysed.”

---

This is also another great suggestion. In Figure 1 we actually present a wider range of algorithms to demonstrate how the common misconceptions around performance and sample complexity exist widely.

---

3. ”I do not intend to add a question here that raises the experimental requirement significantly; but if it is within the scope of the work - maybe one or two experimental results also introducing policy based algorithms in this context might make the paper really interesting. For example, if you pick basic PPO or TRPO for these ALE benchmarks - can we see how those algorithms also compare to these existing ones, in different sample complexity regimes?”

---

Upon your suggestion we in fact recognized and wanted to highlight that the issues that our paper raises on the impact of methodological choices, indeed seem to extend to policy based methods. In particular, let us look closer at the results: the PPO median in the high data regime is 298%, and dueling median is 172%; however, we see that in the low data regime the orders seem to be reversed and the median for PPO is 2% and median for dueling is 23%.

You have some critical misconceptions in your review. Some stem from missing the results reported in our paper, some stem from understanding prior work and some stem from missing critical details.

---

1. ”I do not think it is accurate to say that algorithms developed for Atari100k (Data Efficient RL) are compared with those developed for Atari 200M (high-data regime) [2, 3, 4]. Please check the section "Case Study: The Atari 100k benchmark" in [5].”

---

You have a critical misconception here. The paper [5] is the DER algorithm, and Figure 1 in our paper reports results on the DER algorithm as well. The results reported in Figure 1 of our paper demonstrate that the baseline dueling algorithm outperforms the DER algorithm. Thus the implicit assumption is the fact that assuming that since Rainbow is the best performing algorithm in the high-data regime, specifically tuning Rainbow for the the low data regime, i.e. DER, must be the best performing algorithm in the low data regime as well. However, our results demonstrate that dueling outperforms both DER and DRQ.

Furthermore, note that the papers you cite [2,3,4] indeed also compare against DER, and indeed carry the implicit assumption that our paper discovers, describes and provides a deep concrete explicit analysis for.

[2] Max Schwarzer, Ankesh Anand, Rishab Goel et al. Data-Efficient Reinforcement Learning with Self-Predictive Representations, ICLR 2021.

[3] Max Schwarzer, Johan Obando-Ceron, Rishabh Agarwal et al. Bigger, Better, Faster: Human-level Atari with human-level efficiency, ICML 2023.

[4] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao. Mastering Atari Games with Limited Data, NeurIPS 2021.

[5] When to use parametric models in reinforcement learning?, NeurIPS 2019.

---

2. ”I believe the main takeaway of the paper is incorrect. It is clear that agents that perform well in the large data regime do not necessarily perform well compared to those developed for Atari100k. DER is essentially the same as the Rainbow agent, with different hyperparameters to perform well when data is scarce [5].”

---

The main takeaway of the paper is indeed correct. See Figure 1.

Note that DER [5] is just an abbreviation for Data Efficient Rainbow, and it is in fact the Rainbow algorithm that has hyperparameters tuned for the low data regime. The paper [5] shows that in fact the Rainbow algorithm is the best performing algorithm in the low-data regime, the hyperparameters just need to be tuned correctly. Our results show that the simple dueling algorithm that has been proposed years before in fact outperforms DER in the low-data regime.

---

3. ”I would argue that choosing the right hyperparameters for the algorithms that learn the state-action value distribution (the ones introduced in the paper) will significantly affect their performance on Atari100k.”

---

The hyperparameters are already tuned for the low-data regime.

---

4. ”Figure 2 gives the impression that Dueling is always effective across all the games, but this is not true, as shown in Figure 1 (supplementary PDF).”

---

As already has been reported in Table 1, indeed the dueling algorithm outperforms across the entire Arcade Learning Environment 100K benchmark in median and 20th percentile.

---

5. ”It is difficult to draw conclusions from Table 3. The sample-efficient algorithms are missing. What if you tweak the hyperparameters for the other algorithms (C51/QRDQN/IQN)?”

---

Hyperparameters of these algorithms are already tuned for the low data regime. The conclusion of Table 3 is clear. Table 1 reports results across the entire Arcade Learning Environment 100K benchmark. The results across the benchmark demonstrate that dueling performs better.

---

6. ”While the authors advocate for a paradigm shift in evaluating DRL algorithms, the paper does not discuss how its findings could be applied in practice. For instance, guidelines on designing experiments or adjusting evaluation metrics to avoid high-data biases would make the recommendations more actionable for researchers.”

---

Please see the strengths part of the review of Reviewer i3e2 for how the findings of our paper are relevant to practice.

---

7. ”A recent paper has highlighted the phenomenon of data regime issues, offering valuable insights and prompting a reformulation of the research question in the paper [1].”

---

This paper [1] is an empirical study on the hyper-parameter tuning of deep reinforcement learning. On the other hand, our paper is about methodological choices made in deep reinforcement learning based on implicit assumption of monotonicity of performance ranks across regimes.

Our paper and the paper you cite are about completely different concepts.

[1] Johan Obando-Ceron and João G.M. Araújo et al. On the consistency of hyper-parameter selection in value-based deep reinforcement learning, RLC 2024.

Thank you for replying to some of my questions. I am not convinced by the narrative of the paper. For me, it is unclear and lacks sufficient evidence to support its claims. I suggest the authors reframe the paper to improve clarity.

Questions:

A. From the paper: "One intriguing takeaway from the results provided in Table 1 and the Figure 5 is the fact that the simple base algorithm dueling performs 15% better than the DRQ (NeurIPS) implementation, and 11% less than the DRQ (ICLR) implementation."

1. Why is the performance of DRQ (NeurIPS) lower than DRQ (ICLR) [1]? It should be the opposite [1], not?
2. Table 5 does not say anything about the comparison with DRQ.

B. From the paper: "The instances of the implicit assumption that the performance profile of an algorithm in the high-data regime will translate to the low-data regime monotonically appear in almost all of the studies conducted in the low-data regime."

I disagree with this. I believe the RL community does not assume that agents performing well in the large-data regime are also optimal for the low-data regime. As I mentioned before, when evaluating performance on Atari100k, it is important not to make this assumption. Most of the baselines used for Atari100k differ from those used for 200M. If so, can you please provide more evidence?

C. From the paper: "...the results reported in Section 5 prove that the performance profile of an algorithm in the high-data regime does not monotonically transfer to the low-data regime."

Sure, [2] shows this and links those findings to hyperparameter tuning. However, this does not mean that the community assumes RL agents trained on 200M are sufficiently sample-efficient to perform well with scarce data.

D. From the paper: "Our extensive large-scale empirical analysis demonstrates that a major line of research conducted in the past five years resulted in suboptimal conclusions. In particular, Figure 5 shows that a simple baseline dueling algorithm from 2016, by a canonical methodological choice was never included in the comparison benchmark due to the implicit assumption that appears in all of the recent line of research that we have discussed in detail in Section 3."

I believe this is not an accurate conclusion. The Dueling result appears to be the same as in the DER [2] paper. The authors proposed some additional hyperparameters and made the Rainbow agent more sample-efficient. In the context of this paper, it is referred to as the Dueling agent. Indeed, agents like SPR and BBF all used Dueling. I think the authors should reframe the motivation of the paper, as some arguments lack sufficient support.

"Thus, we demonstrate that this baseline algorithm in fact performs much better than any recent algorithm that claimed to be better than the baselines, even including algorithms that are built on top of the dueling algorithm."

This is not right at all. You are missing a lot of recent work on sample efficiency that has used Atari100k for this.

E. "Please see the strengths part of the review of Reviewer i3e2 for how the findings of our paper are relevant to practice..."

I encourage the authors to add a section about guidelines on designing experiments or adjusting evaluation metrics to avoid high-data biases. This would be very valuable for the community.

To sum up, for this paper to be presented at a top-tier AI conference, it requires more clarity regarding its motivations and contributions. I feel that the problem definition and logical approach in this paper are insufficient. Based on the current responses from the authors and the state of the paper, I cannot improve my rating.

References:

1. Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc Bellemare. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
2. Johan Obando-Ceron and João G.M. Araújo et al. On the consistency of hyper-parameter selection in value-based deep reinforcement learning, RLC 2024.
3. Van Hasselt, Hado P., Matteo Hessel, and John Aslanides. "When to use parametric models in reinforcement learning?." Advances in Neural Information Processing Systems 32 (2019).

Thank you for stating that our paper is novel, both in terms of theory and experiments, and the theoretical results we provide on the data regimes are interesting, insightful, and non-vacuous, and furthermore our paper contains novel insights about several popular and well-cited algorithms for the Atari 100K benchmark with experiments indeed serving to demonstrate the points presented in our paper.

---

1. “Given that the paper's title is "A LARGE-SCALE ANALYSIS ON METHODOLOGICAL CHOICES IN DEEP REINFORCEMENT LEARNING", can you please list out all the methodological choices that were investigated?”

---

Our paper investigates a series of methodological choices made under the assumption that an algorithm's performance must be monotonic across the data regimes, i.e. if an algorithm performs the best in the high-data regime it must be the best performing algorithm in the low-data regime. We see several instances of this in the case of establishing Rainbow as a baseline algorithm and developing an extensive line of algorithms for the low data regime. Although it is an intuitive methodological choice to assume that the performance across regimes will be monotonic, our work demonstrates that it is indeed non-monotonic, and a simpler baseline algorithm, e.g. dueling, that has been proposed and published years prior to Rainbow in fact performs much better than Rainbow.

Furthermore, in an orthogonal axis of investigation our results demonstrate that some of the quite recent algorithms developed for the low-data regime that built on these existing prior baseline algorithms, i.e. dueling, did in fact not provide direct comparison to the algorithms that they were building on top of since the baseline algorithms were originally proposed for the high-data regime. Thus, when a direct comparison is made our results demonstrate that these recent low data regime algorithms in fact perform worse than the baseline algorithms that they were in fact built on.

Furthermore, on an independent axis our paper also reveals another methodological choice is how the Arcade Learning Environment 100K benchmark itself was established, in which we directly quote: “Specifically, for the final evaluation we selected games which achieved non-random results using our method or the Rainbow algorithm using 100K interactions.” Hence, another critical methodological choice is that the entire low-data regime dataset was established based on the assumption that our paper discovers and provides an extensive analysis for.