The Curse of Diversity in Ensemble-Based Exploration

Thank you very much for your positive comments! We are pleased to hear that there are no major concerns regarding our paper. Your feedback is greatly valued and has been encouraging for us.

In Weaknesses 4: Contrast this with BootDQN which switches ensemble members mid-episode. If this algorithm would improve individual performance, the reviewer would accept it as a fix of the identified problem.

We have tested this variant In Figure 17 of Appendix D.3.4 . Specificially, we switch ensemble members at each step instead of only at the start of an episode. As shown in the result it does not mitigate the curse of diversity. This is not surprising because it does not change the proportion of “self-generated” data for each agent in the replay buffer.

Response to “Minor comments”

The figure captions should contain more information about the shown algorithms (ensemble size, replay buffer size, L, ...) to make them more self-contained for cross-reading.

Thanks for your suggestion. We have included these in the captions in our revision.

Ensembles are primarily used because they are good out-of-distribution detectors. An experiment measuring this would have been very welcome.

Thank you for your suggestion regarding OOD detection. We agree it's an important aspect. However, we intentionally focused on the off-policy learning aspects of ensemble-based exploration for this paper, and incorporating OOD detection might go beyond our current scope. We look forward to exploring this in future research.

Response to “Questions”

If one accepts the idea that ensembles are primarily for exploration, then the diversity of responses is actually a feature, not a bug. Given that few wrong decisions per episode can significantly reduce the performance of an individual ensemble member, why do the authors actually consider the poor individual performance an issue, in particular when the aggregate performance is better than the baseline (e.g. on Mujoco)?

It is widely accepted that a diverse ensemble can potentially result in better exploration and a more robust aggregate policy. The point of our work is not to deny these advantages, but to point out that despite these benefits, diversity also has a negative side effect. The fact that its advantages (e.g., majority voting) sometimes outweigh its disadvantages (off-policy-ness) does not mean that the disadvantages do not exist or are not an issue. This has practical significance. For example, without realizing the negative effect of diversity, one might think that having more ensemble members is always better (if we ignore computational costs), which is not the case as we have shown in Figure 5. The cause of this non-monotonic behavior is obvious from the individual agents’ performance but not from the aggregate policy’s performance. Also, knowing the cause of the curse of diversity allows targeted solutions that may give rise to better algorithms. CERL is just a first step in this direction. Finally, as mentioned in the second paragraph of Section 3.1, there are some scenarios such as hyperparameter sweep with ensembles where we do care about the performance of the individual ensemble members.

How did the baseline (Double DQN and vanilla SAC) explore? If the baseline worked so well, doesn't this imply exploration is not very important in the tested environments? How would the analysis look like if it would concentrate on hard-exploration environments?

Double DQN uses epsilon-greedy and vanilla SAC just uses stochastic policy + entropy regularization. These are very simple baselines and the performance on the tested environment is far from saturated, so the results by no means imply exploration is not very important in the tested environments. Note that the suite of 55 Atari games we tested contains several games that are traditionally considered hard exploration problems (e.g., Montezuma’s Revenge. For a complete list see Table 1 in [3]). As shown in Figure 2 (top right) and Figure 10 of our paper, Bootstrapped DQN does not show clear advantages over Double DQN in most of these games. In fact, in the original Bootstrapped DQN paper, the author also finds that “On some challenging Atari games where deep exploration is conjectured to be important our results are not entirely successful, but still promising” and shows that the improvements could be highly unstable (e.g., on Montezuma’s revenge). This is not surprising as these hard exploration games often require special events that are very unlikely with primitive exploration techniques such as epsilon-greedy or ensemble.

We hope our response has addressed your concerns. Please let us know if you have any further questions!

References

[1] Ostrovski, Georg et al. “The Difficulty of Passive Learning in Deep Reinforcement Learning.” Neural Information Processing Systems (2021).

[2] Fujimoto, Scott et al. “Off-Policy Deep Reinforcement Learning without Exploration.” International Conference on Machine Learning (2018).

[3] Bellemare, Marc G. et al. “Unifying Count-Based Exploration and Intrinsic Motivation.” Neural Information Processing Systems (2016).

Thank you for the insightful comments and detailed feedback! These comments lead us to think more deeply about our results and are very helpful for enhancing the quality of our work. Please see our response below:

Response to “Weaknesses”

In Weaknesses 1: This is a problem in statements like "the two cases[, DQN(indiv.) and Double DQN,] have access to the same amount of data and have the same network capacity" (p.3), as the change in exploration means that the two algorithms are based on different data distribution and will therefore behave differently, irrespective of the on- and off-policy-ness of the data. This potential interaction between exploration (which part of the state-space is covered) and on-policy-ness (which member got to sample it) cannot be distinguished with the presented experiments, but the text reads as if the latter is the only obvious conclusion (see below).

In Weaknesses 2: …The fact that they do not show that there must be another effect at work here. This could be the exploration, but the experiments do not allow us to distinguish these potential effects

Thank you for raising the point regarding exploration/coverage. We have included a detailed discussion on this in the “Remark on data coverage” paragraph at the end of Section 3.2. The main points are summarized as follows:

1. State-action space coverage is purely a property of the data distribution, but off-policy-ness depends on both the data distribution and the policies. So indeed they are different.
2. Our p%-tandem experiment disentangles these two aspects: the active and passive agents are trained on the same data (thus same coverage), but experience different degrees of “off-policy-ness”.
3. As a result, our experiment indicates that “off-policy-ness” is detrimental and sufficient to cause the performance degradation we see. At the same time, we recognize that it does not imply whether wider state-action space coverage is beneficial (more likely and intuitive) or harmful.
4. On the other hand, as you mentioned, the fact that BootDQN (indiv.) performs better than the passive agent indicates that wider coverage is likely highly beneficial. Reasoning:
   1. Due to the presence of $N$ different policies, the individual agents in BootDQN likely suffer from more severe off-policy-ness than the passive agent.
   2. Therefore, if the wider coverage in BootDQN was not beneficial, due to more severe off-policy-ness BootDQN (indiv.) should have performed significantly worse than the passive agent.
5. However, unlike the comparison between the active and passive agents which is perfectly controlled, the comparison between BootDQN (indiv.) and the passive agent may have other (unexpected) confounders. Therefore we avoid making conclusive claims about the effect of state-action space coverage in our paper.

We have also added “even though they are trained on different data distributions and thus are expected to behave differently,” before the statement “the agents in these two cases have access to the same amount of data and have the same network capacity” in Section 3.1 to emphasize that the data distributions in these two cases are different.

In Weaknesses 1: the authors claim that "Surprisingly, simply aggregating the learned policies at test-time provides a huge performance boost in many environments" (p.3), and proceed to downplay the effect of exploration in favor of an explanation based on "aggregation"

To avoid misinterpretation, we have revised the second paragraph in Section 3.1 to emphasize that even though we show that majority voting plays a more important role than previously attributed, it does not indicate that exploration/state-action space coverage is not important or beneficial.

In Weaknesses 2: "These results confirm our hypothesis regarding the cause of the observed performance degradation" (p.5). This is dangerous abductive reasoning: experimental evidence that is consistent with your hypothesis does not make your hypothesis true (one can only falsify a hypothesis).

We agree that there could be other factors at play (e.g., state-action space coverage, as discussed), even though we do believe that our results provide strong evidence that off-policy-ness plays a major role in the observed performance degradation. We have changed our statement to “These results offer strong evidence of a connection between the off-policy learning challenges in ensemble-based exploration and the observed performance degradation” as a more accurate conclusion of our results.

Thank you for liking our presentation and providing detailed feedback and suggestions! Please see our response below

On high-level motivation

As has been demonstrated in [EDAC] and [REDQ], I acknowledge that using ensemble learning for the value function could lead to improved performance, as the value can be more accurate, with uncertainty. But what is the motivation for having multiple policies for ensemble

We emphasize that even though both (1) ensemble-based exploration methods such as Bootstrapped DQN and Ensemble SAC and (2) methods such as EDAC and REDQ use ensembles, they use ensembles in completely different ways and purposes. Bootstrapped DQN and Ensemble SAC use ensembles purely for better exploration. In contrast, methods such as EDAC and REDQ use ensembles purely for improving value estimation. The concrete differences are detailed below, which should clear up your confusion:

• The number of policies:
  • Both Bootstrapped DQN and Ensemble SAC maintain $N$ policies. Note that for Bootstrapped DQN the $N$ policies are implicitly defined by the $N$ value functions. Most importantly, all $N$ policies are used to collect samples. Concretely, for each episode, one policy is sampled for interaction. The motivation for having $N$ policies in Ensemble SAC is the same as that of having $N$ different value functions (and hence $N$ implicit policies) in Bootstrapped DQN: to explore the state space with a diverse set of policies (explicit or implicit) concurrently in a temporally coherent manner.
  • EDAC and REDQ are simply not designed for exploration. They only have one policy and it is the only one used to interact with the environment.
• How the value functions are used:
  • In Bootstrapped DQN and Ensemble SAC, each value function $Q_i$ evaluates one policies $\pi_i$. These value functions are independent (though they may share networks) in the sense that the TD regression target for each ensemble member is computed independently with their target network. Concretely, the target for $Q_i(s, a)$ is $r(s, a) + \gamma \bar{Q}_i(s’, a_i’)$, where $a_i’\sim\pi_i(a_i’|s’)$. They are not designed to produce more accurate value estimations than single-agent Double DQN or SAC. There is no explicit mechanism that encourages more accurate value estimations such as penalizing Q-value based on uncertainty (though the diverse data may help implicitly).
  • In EDAC and REDQ, all the $N$ value functions $Q_{1:N}$ evaluate the same single policy $\pi$. The $N$ value functions share a single TD regression target that is aggregated from all target networks. For examples, in EDAC, the target for $Q_i(s, a)$ -- regardless $i$ -- is $r(s, a) + \gamma\min_j \bar Q_j(s’, a’)$ (ignoring the entropy term), where $a' \sim\pi(a'|s')$. Note how the target values are aggregated with the $\min$ operator to counter overestimation.

In summary, the goal of having multiple policies in Ensemble SAC is to promote better exploration. This is the exactly same motivation as having multiple value functions (which implicitly define multiple policies) in Bootstrapped DQN. In contrast, EDAC and REDQ are not designed for better exploration; there is only one policy in them and it is the only policy interacting with the environment.

Thank you for appreciating our contributions and providing valuable feedback! Please see our response below:

Perhaps this is out of scope of the paper, but I do feel it's difficult to draw conclusions about deep RL more broadly without investigating distributional RL

This is a great suggestion and we have added experiments on four Atari games with QR-DQN. The results are presented in Figure 16 of Appendix D.3.3. The significant performance gap between QR-DQN and Bootstrapped QR-DQN (indiv.) shows that the curse of diversity is also present in distributional RL. Note that due to limited time, we are not able to finish the full 200M frames of training. We will include the full results when the experiments are finished.

CERL does seem to scale poorly with the ensemble size, since if the ensemble size is N, there are N x N Q-functions. In practice, this does not seem like a major problem.

In practice, CERL scales very well with ensemble size. As mentioned in the “Method” paragraph in Section 4, to implement CERL we only need to increase the output dimension of the last linear layer of the network (note this is the same architectural modification involved in distributional RL). So even though CERL requires N x N Q-functions as opposed to N Q-functions in a standard ensemble, the increase in the number of parameters and wall clock time is very small. For example, adding CERL to Bootstrapped DQN (L=0, N=10) increases the number of parameters by no more than 5% and the increase in wall clock time is barely noticeable. Even if we use N=20 which is already much larger than commonly used in the literature, applying CERL increases the number of parameters by no more than 11% and the increase in wall clock time is less than 5%.

For the bar plots in Figure 2, it might be helpful to plot the top plot minus the bottom plot to highlight the point.

Thanks for your suggestion! We have included this figure in Figure 12 of Appendix D.2.2. We do not include it in the main text to save space for other changes, and also because this plot mainly shows the benefits of majority voting which is not the primary focus of our work.

Question: Did you try different levels of passivity? Other than 10%?

We have also tried 0%, 5%, and 20% in four Atari games. The results are shown in Figure 19 of Appendix D.3.6. As expected, as the degree of passivity reduces (i.e., larger $p$), the active/passive agents’ performance gap also reduces.

Please let us know if you have any further questions or suggestions!