Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity

We thank the reviewer for their thoughtful review our work and for appreciating its merits. We have addressed the noted weaknesses below, as well as their question about parameterizations.

[W1] It is still necessary to hand-craft the high-level features, which needs expert knowledge (or at least quite high familiarity with downstream tasks) and can be heuristic.

We agree that this is a limitation of our work, which we discuss in Section 7 (Line 319). In our response to [W1] from Reviewer 7bxy, we have included discussion of an example work [1] where QD features are learned automatically; it is possible we can draw inspiration from works in this vein to further automatize DIVA.

[1] Ding, L., Zhang, J., Clune, J., Spector, L., & Lehman, J. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. In Forty-first International Conference on Machine Learning.

[W2] Evaluation only used one meta-RL method (VariBAD). It would be useful to see performance of other meta-RL methods such as RL^2 on the DIVA task distribution.

It is true that all evaluations are conducted using VariBAD the base meta-RL algorithm. However, we did perform some preliminary evaluations with RL2—which is the same as VariBAD, except (1) instead of using a VAE with a latent space, we directly pass the RNN hidden state to the policy, and (2) we backpropagate the policy loss through the RNN (VariBAD only applies the VAE loss)—however, we did not see a drastic difference in the relative performance—between DIVA and its baselines—and because of the RNN backpropagate via the policy loss, training is significantly slower (as originally noted in [2]).

Given these results, in absence of a performance differential, we decided on VariBAD, a state-of-the-art meta-RL algorithm. One reason we were drawn specifically to VariBAD is the harmony between the nature of task distributions, and VariBAD’s distributional nature. In DIVA’s case, task distributions are represented as a multivariate normal (some dimensions may be uniform) over a QD archive. We believe it may be possible for future works to exploit this connection; e.g. by using the location of a solution in the archive to ground the VariBAD latent space (a rough idea).

[2] Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., & Whiteson, S. (2019, September). VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. In International Conference on Learning Representations.

[Q1] Does DIVA depend on any parameterizations at all? It seems like it might be possible (and very useful) to try DIVA on “weakly” parameterized tasks such as those generated by prompting an LLM or, say Genie [1].

This is a great question, and certainly a direction worth exploring. DIVA in its current iteration can work with these weakly parameterized tasks, so long as (1) we have an environment generator that can accept this “weak” parameterization, and (2) we have access to some mutation method that can operate over this weak parameterization. For example, if the weak parameterization is language, we need some kind of language augmentation function (which prior literature has certainly explored). It may be even more effective to work with in the embedding space itself, since the augmentation function would be both simpler & smoother than language augmentation; in this case we would be essentially using QD to perform prefix/embedding tuning. We are excited to see these kinds of directions pursued in follow-up works.

We appreciate the reviewer’s feedback and attention to detail. We have made all of the minor changes suggested under “Questions”, which will be present in the camera-ready draft. We address both major concerns below.

(1) On the domains being “toy”

[W1] [...] domains [...] are all very simple [a] “toy” domains. In particular, these domains do not have all have [b] intrinsically complex parameterisations, and for two of them the authors had to, in effect, obfuscate the parameter spaces so the algorithm had a challenge to work against. [...] the authors’ results may [c] not be representative of more realistic open-ended domains where the parameter space complexity may manifest in a different way.

(1a) For academic institutions with limited computational resources, this set of domains presents challenge enough to usefully benchmark methods, while keeping training runs cheap enough that many experimental trials can be run—enabling us to achieve statistically significant results.

(1b) As discussed openly in Section 5, we recognize that the domains chosen require us to “obfuscate” the existing environment parameterization(s). We believe the reviewer may fundamentally misunderstand why adjusting the parameterization of these domains is necessary.

Most meta-RL/RL domains are released with a parameterization/generator that can produce levels of meaningful diversity. Especially for domains with complex dynamics, designing these are not so straightforward. Because DIVA-like approaches are not widely studied at present, most domain designers do carefully hand-craft task distributions so that researchers will use their domains.

As meta-RL approaches improve, and domains become more complex and open-ended, well-structured environment parameterizations will become increasingly expensive to implement by hand. Approaches like DIVA will enable learning on these domains, without requiring carefully hand-designing a structured parameterization and corresponding task generator. We believe that demonstrating the existence of DIVA-like methods that can handle ill-parameterized domains is a necessary step to inspire researchers to build more open-ended domains with unstructured parameterizations in the first place.

(1c) The main property that makes a parameter space “unstructured” is the low probability of producing meaningfully diverse levels. This is the guiding principle used to design the challenging parameterizations for our evaluation domains—which, as we have explained above in response to (1b), was a necessity, since most (if not all) current academic domains are released with convenient parameterizations. While we agree that “the parameter space complexity may manifest in a different way” for more complex open-ended domains, what will remain is the overarching principle of meaningfully diverse levels having a low probability of being generated. Without detailing some other specific property of open-ended domains we might have overlooked, this weakness is a “whataboutism” that can be applied in similar form to any set of finite empirical findings.

(2) On hand-selected features

[W2] […] The authors need to hand-select the features used by the algorithm, per domain. […] The impact of this technique is much more limited if feature sets need to be hand-tuned per domain, so without demonstrating that this is not the case, I think the authors do not demonstrate that this technique is likely to have wide impact.

First, only a few features specified (a subset of those detailed for each domain in the Appendix) were used in experiments, and the final designs for each were guided more by intuition (e.g. looking at the distributions and covariance matrices) than any "extensive" experimental tuning. We also demonstrate with DIVA+ (Line 259) that the misspecification of features/objectives can be made up for by performing a small number of online evaluations like UED works.

We believe that this criticism also overlooks the “extensive hand tuning” that is currently required to design parameterizations for complex domains. It is not that other approaches are able to sidestep the necessity of “hand-tun[ing]” feature sets per domain—it’s that this tuning is buried within the environment generation logic, which enabling a conveniently structured parameterization.

The assumption of having access to some features of useful diversity is much weaker assumption than the assumption of having pre-existing generators that can produce levels across these same features of diversity. Consider—how does one design a structured parameterization/generator without first determining what the resulting level features should look like?

Questions

[Q1] L159-161: [...] The downstream task distribution was first used to train the meta-learner’s hyperparameters, and then these hyperparameters were used to compare all of the approaches. But isn’t this essentially training on the test set? [...] I can appreciate that the authors might want to try to “factor” the behaviour of the meta-learner from the behaviour of DIVA, but it seems like a strong assumption that this is possible or useful.

We recognize the ambiguity our wording may have caused here. The meta-learner is in fact “tuned” on the structured parameterizations. As can be seen in Appendix C.2, only a few VariBAD hyperparameters are adjusted per domain, which mostly pertain to pragmatic considerations such as network structure, and RAM/storage considerations. Because the parameterizations we consider produce poor training levels, we validate that the meta-RL component works by running the agent on the structured generator. This is indeed to “’factor’ the behavior of the meta-learner from the behavior of DIVA” and was a pragmatic decision so that we could more confidently study the effects of our method over the baselines. Once the meta-learner configuration was determined for each domain, it was held fixed for each method evaluated.

More on (1a): GridNav is a standard navigation task which serves as a useful didactic domain (and serves this purpose in VariBAD [1] as well), the Racing domain is a control domain used to benchmark numerous UED works [2, 3], and symbolic Alchemy is a chemistry-inspired meta-RL domain that requires complex trial-and-error reasoning over numerous episodes, which nicely complements the navigation and control tasks.

More on (2): We might also add that this is the first work to consider the combination of QD and meta-RL in this manner. In quality diversity (QD) literature, features are typically assumed to be pre-determined [4]. This pre-specification of features is indeed a limitation, but one that can be lifted in a similar fashion to QD works that learn features automatically [5]. Based on the contributions of our work, we decided it was appropriately out of scope to additionally consider the problem of learning features automatically.

[1] Zintgraf, L., et al. (2019, September). VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. In International Conference on Learning Representations.

[2] Parker-Holder, J., et al. (2022, June). Evolving curricula with regret-based environment design. In International Conference on Machine Learning (pp. 17473-17498). PMLR.

[3] Jiang, M., et al. (2021). Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34, 1884-1897.

[4] Pugh, Justin K. et al. (2016) “Quality Diversity: A New Frontier for Evolutionary Computation.” Frontiers in Robotics and AI 3:40.

[5] Ding, L. et al. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. ICML.

[Q6] Line 188 claims that “DIVA, on the other hand, is able to capture this diversity.” However, doesn’t DIVA also decrease to the same level of percentage coverage as DR*? Why are there no confidence intervals for Figure 3a, like the others?

Figure 3a shows the ability of DR, DR*, and DIVA to “capture” diversity in generated levels with parameterizations of varying complexity. DR* reflects the upper bound of what a method like PLR can achieve with DR levels—it is the maximum diversity see by DR (i.e. the total number of unique levels seen over the entire generation process), whereas DR is just the final archive produced. DR has no memory mechanism to preserve diversity, whereas a method like PLR, with its priority buffer, can latch on to the diversity contained in DR* (but still may fail to preserve all discovered diversity).

The reviewer is correct that DIVA is shown to capture just as much as DR* at the most complex parameterization we tested. However, given the nature of the experimental design, this trailing off is destined to happen with a high enough complexity; it becomes statistically unlikely any algorithm to produce meaningful diversity. It was a purposeful decision to demonstrate that DIVA tails off in its ability to handle these highly challenging parameterizations. If we increased the number of update steps allowed for DIVA, or inserted a more sophisticated emitter algorithm (instead of MAP-Elites), DIVA would be able to benefit from this far more than DR* (the same trend would extend, but fall off just the same at a high enough complexity).

The takeaway from this plot is that QD is able to uncover and preserve diversity in the archive as the complexity increases better than DR (or our invented oracle DR*). This is also why we have not included confidence intervals—-the point is to show the trend; each point is just a single run of QD updates. Given the cleanness of the trend, we found it unnecessary to run more seeds for this plot, but we can do this for the camera-ready draft. Either way, we have clarified this point in the paper. Thanks for your feedback!

[Q2] If the original parameterizations are chosen randomly or are not handcrafted to what humans might care about, how would DIVA perform?

We believe this question likely reflects the same misunderstanding noted above; we refer the reviewer to our [Q1] response, and welcome any follow-up questions the reviewer may have on this point.

[Q3] To what extent and when do we consider an environment simulator to have structured or unstructured parameterizations?

We introduce the idea of structured/unstructured parameterizations in Figure 1, and elaborate specifically on the unstructured parameterizations DIVA can work with starting on Line 88. We define structured parameterizations as ones that enable random sampling to produce diverse levels of interest, whereas increasingly unstructured parameterizations produce these useful levels with diminishing probability. These terms are not absolute, but are instead relative, and represent two poles of a continuum. Open-ended domains, with more degrees of freedom and greater complexity in environment dynamics, must either be carefully parameterized and provided with a sophisticated generator that can navigate these complexities (i.e. a structured parameterization), or they can be more flexibility parameterized with a simpler generator (an unstructured parameterization), where meaningful diversity is possible, but not guaranteed with high probability. DIVA can make use these more unstructured parameterizations, and some knowledge of some knowledge of what levels should look like, in order to produce an abundance of training levels that are diverse in these ways. Importantly, this saves the prohibitive effort of having to carefully craft structured parameterizations / generators for complex open-ended domains.

[Q4] In the GridNav experiment, why do we want to prevent the generation of diverse goals along the y-axis?

We believe this may pertain to the same misunderstanding addressed in [Q1], so we refer the reviewer to that response, but we also address this question more extensively below.

The object of DIVA and the baselines are to produce meaningful diversity in the generated levels, and for the GridNav domain, this means producing goal diversity along both the x and y axis.

In our GridNav experimental setup, we inject complexity into the parameterization that makes it difficult to generate goal diversity along the y-axis. We design the parameterization in such a way that we can vary the complexity (unstructuredness) of the parameterization. As we increase the complexity, diversity along the y-axis becomes less and less likely. Our results in GridNav show that DIVA is best able to capture this diversity by (1) evolving levels with the expressed purpose of discovering and preserving levels of meaningful diversity (via the QD archive), and (2) avoiding agent evals like ACCEL/PLR, which are expensive, and doesn’t guarantee all diversity discovered is preserved.

[Q5] Is there a hypothesis on why “the domain randomization over increasingly complex genotypes diminishes diversity in terms of goal location” (lines 187-188)?

This is by design (described in Line 178) rather than some emergent property requiring hypothesis, and is likely relevant to the misunderstanding addressed in [Q4]. Increasingly complex genotypes diminish diversity in terms of goal location by our design of the genotype; and we design the general genotype scheme in such a way that we can conveniently vary the complexity—in this case, the number of genes that must coordinate to produce diversity along the y-axis. The effect of this design is that diversity along the y-axis becomes less and less likely (with random genotypes) as the complexity increases, because more genes need to coordinate.

We thank the reviewer for their thoughtful review of our work, and for engaging us with curiosity to better understand the paper. Below we address the two main weaknesses noted by the reviewer, and clarify certain aspects of the work in response to the reviewer’s questions.

[W1] [...] The presentation of the paper can be improved. For example, Figure 1 could include more descriptions about what is happening [...]

We appreciate this constructive feedback. We have updated Figure 1 (see Visual A in the 1-page supplemental), which now includes more descriptions as suggested by the reviewer.

Additionally, we have taken other efforts to improve the presentation quality of the rest of the paper. These changes include adding algorithmic pseudocode to the Appendix (Visual E in the 1-page PDF) for the sake of clarity, a number of minor updates in wording for the sake of exposition clarity, switching out pixelated plots for raw vectorized PDFs for better image resolution, and some other minor aesthetic changes along these lines.

We welcome any other concrete suggestions the reviewer may have for improving the presentation.

[W2] In line 128, how was the number 80% chosen? And why is it described as “roughly”?

We thank the reviewer for pointing out this ambiguity. By “roughly” we were referring to the nature of samples versus population: we set this threshold using the 80th percentile of the samples, which encapsulates roughly 80% of the population. We understand this wording is unclear, and have therefore clarified what we mean precisely in the manuscript.

[…] More justifications for the choice of these parameters should be included.

80% was chosen as an intuitive heuristic to balance the need for preserving the diversity contained within the DR samples used to initialize the archive (and the corresponding region that may be filled with useful mutations), and given a fixed archive size, maintaining as high a resolution as possible to enable the fastest propagation of solutions to and within the target region.

We have run experiments to demonstrate that any moderate setting of this value prevents the failure modes of either extreme—see Visual D in the 1-page PDF (which will be included in the camera-ready, along with relevant discussion). In Alchemy (left) we see that the failure mode preventing the generation of target solutions is when we set this too low—-i.e. where we take only a small part of the tail of the DR samples. In Racing (right) we see the opposite problem; when set too high, it has significantly reduced the number of target samples. In both cases, the any moderate settings of the hyperparameter seems to do the job; it’s really about preventing either of the extremes. We chose 80% to be on the conservative side, but from the experiments, it looks like a setting of around 50% might even be better (from the Racing results).

Additionally, we include robustness studies for other major hyperparameters in Visuals B, C, and F in the 1-page PDF (also for camera-ready). All attached figures correspond to Alchemy, but we can also provide similar ablations for Racing in the camera-ready draft. In Visual B we show robustness of final returns based on the mutation rate (Alchemy); in Visual C we see more QD updates is generally better, but our setting of 80k is not necessary to produce significant benefits over baselines; and in Visual F we see that while estimates for the normal parameters become less accurate with fewer sample features provided, DIVA is able to outperform baselines with as little as 5 downstream feature samples.

[W3] Typo in line 184, “the final y location is determine by”

Fixed, thanks!

Questions

[Q1] Why does DIVA want to evolve a population of minimally diverse solutions from the original parameterization? Does DIVA assume that the original parameterizations are what humans care about? If so, doesn’t that mean we still need to ensure that the original parameterizations are handcrafted enough to be close to what humans care about?

We believe there is a key misunderstanding here between the function of DIVA and the experimental setup. We choose parameterizations in our evaluation that do not provide the dials—so to speak—to tune the features that humans care about (like the dials on $E_S(\theta)$ in Figure 1—-updated version in 1-page PDF response, Visual A). And these challenging parameterizations are what produce the “minimally diverse” solutions with which the archive is initialized. DIVA is designed to work with parameterizations like these that are explicitly not designed to produce diversity that humans care about. DIVA overcomes these difficult parameterizations by using QD to populate an archive of solutions that are diverse along certain features—which are hand-designed in this work, but may be learned automatically in future work (e.g. [1], a citation we’ve added thanks to the suggestion of Reviewer 7bxy).

It is alternatively possible the reviewer might be asking if we need access to $E_S(\theta)$. DIVA does not need access to this generator/parameterization; all it needs is enough feature value samples from the downstream distribution to roughly estimate what our target distribution over the archive should be. It does not need the parameters that correspond to these scenarios. For example, if DIVA were tasked with generating house layouts for a house navigation task, it would need to know the target distribution for the number of rooms / the hallway width, etc. which could be determined from e.g. photos; but it would not need the parameters for constructing these sample houses in the simulator.

In short, it is the features that must be designed/specified to reflect what humans care about, not the parameterization. Most works assume the parameterization is well-behaved. We challenge this assumption, and replace it with an assumption we believe is more realistic for open-ended environment simulators.

[SAMPLR] Jiang, Minqi, et al. "Grounding aleatoric uncertainty for unsupervised environment design." Advances in Neural Information Processing Systems 35 (2022): 32868-32881.

SAMPLR attempts to correct the “curriculum-induced covariate shift” (CICS)—with respect to the downstream task distribution—that results from learning an adaptive curriculum. SAMPLR mitigates this bias over aleatoric parameters while preserving the benefits of utilizing a curriculum (versus sampling directly from the downstream task parameters).

Assessment: We agree with the reviewer’s assessment that this approach is less relevant for direct comparison to DIVA since SAMPLR requires access to the simulator itself (and the direct downstream sampling distribution, which we do not assume access to either). However, we believe SAMPLR’s relevance to the meta-RL setting (meta-RL potentially suffers even more from this bias) warrants mention in the related work section---an addition which will appear in the camera-ready draft.

[CLUTR] Azad, Abdus Salam, et al. "Clutr: Curriculum learning via unsupervised task representation learning." International Conference on Machine Learning. PMLR, 2023.

The authors of CLUTR argue that many UED methods are burdened by the difficulty of simultaneously learning the task manifold (implicitly through RL, for PAIRED-based methods) and the curriculum over these tasks. CLUTR disentangles the task representation learning from curriculum learning by first pretraining a task manifold with a VAE in an unsupervised manner, and then learning a curriculum over this (fixed) task manifold via maximizing regret.

Assessment: By training a VAE on levels sampled from the generator distribution, which lack greatly in diversity in DIVA’s setting, CLUTR would fail to learn a generator that produces phenotypically diverse levels. ACCEL, by performing mutations to existing levels, is able evolve its levels towards greater diversity over time, making it a stronger baseline for our setting. CLUTR may, however, be a useful future addition to a DIVA-like approach, e.g. by converting the archive to a learned manifold via a VAE. This could be potentially useful for storage considerations (i.e. if our archive is too large), or to extrapolate beyond levels that exist in the archive. Performing curriculum learning over this distilled archive would look something like our DIVA+ algorithm, which combines the benefits of DIVA (diversity) with UED (curriculum) approaches, but would have these storage/extrapolatory benefits. Due to its relevance to future work, we have cited and added discussion of this work to the manuscript, which will appear in the camera-ready draft.

[DRED] Garcin, Samuel, et al. "DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design." Forty-first International Conference on Machine Learning. 2024.

The authors introduce data-regularized environment design (DRED), which combines adaptive sampling (like UED) with a level generator that approximates p(x). DRED assumes access not directly to p(x), but to a set of level parameters X with which a VAE can be trained. Once the VAE is trained, PLR is used to produce scores for sampling (other details omitted here for simplicity).

Assessment: The biggest difference between DRED and DIVA’s settings is that DIVA does not assume access to the underlying level parameters, but rather only the feature values---and these alone are used to define the archive. The QD search is then tasked with producing samples that match the feature distribution (in our work, for both simplicity and sample efficiency, making assumes of either independent Gaussian/Uniform distributions). DIVA’s archive is initialized with random parameters from the parameter space; it never sees the parameters corresponding to the feature values. This difference makes DRED suffer from the same issue as CLUTR when faced with the setting that DIVA operates in: if the parameter space cannot produce diverse levels for the VAE training, then the resulting model will produce levels just as lacking in diversity. Resampling a la PLR will have a similar effect as our PLR baseline. Thus, as was mentioned of CLUTR above, ACCEL remains a stronger baseline than DRED for this reason. That being said, DRED’s close resemblance to our setting and possible applicability to future work (like CLUTR), we have cited and added discussion to our manuscript, which will appear in the camera-ready draft. We would like to note, however, that DRED’s absense in our original discussion was due to its very recent publication (uploaded to ArXiv in February 2024; presented at ICML in July 2024).

[W4] In Figure 5d, “unique genotypes[“] is a quite bad metric for diversity, as completely random levels would score quite well even though most random levels are quite qualitatively to each other.

We agree with the reviewer that unique genotypes is indeed a bad metric for diversity. We include this plot not to highlight diversity, but to make the same observation the reviewer notes (see Line 207)—-that despite DR (for example) producing more unique genotypes than DIVA, DIVA is able to produce greater meaningful diversity, which leads to better performance. We also note that in Line 208 we mistakenly referenced “Figure 5” (in general) instead of “Figure 5d”, and we have made this correction.

[Q2] In the abstract it is not clear to me what "well-behaved simulator parameterisations" means, what "unscalable flexibility" refers to, or what "ill parameterised simulators" means.

We agree with the reviewer that the abstract should be self-contained, and that terminology should either be widely understood or defined in the abstract itself. We will modify these phrases in the camera-ready draft to be clear and unambiguous. We thank the reviewer for bringing our attention to these instances.

[Q3] It would be good to have a citation on line 46 for how one could learn the axes of QD.

Great suggestion; we have added one such example [1] to our manuscript to be present in the camera-ready draft.

[1] Ding, L., Zhang, J., Clune, J., Spector, L., & Lehman, J. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. In Forty-first International Conference on Machine Learning.

[Q4] It's not immediately clear what "genotype" means, and it seems to be used in different ways on line 82 and 88. In the first case it may be more conventional to call it the "level generator" and in the second case it could be more conventional to call it the "level parameters".

The reviewer is correct that the first use of genotype in Line 82 is misleading; we will update to “level parameterization” as this is what we mean. In Line 88 we will use “level parameters” in as the reviewer suggests for clarity, but we will still introduce and use “genotype” as a shorthand/alternative for “level parameters”, in order to respect the connection to evolutionary approaches.

We thank the reviewer for taking the time to write such a thorough review, and for not only appreciating the merits of this work, but for identifying points of weakness to strengthen our manuscript. We have responded to each of the weaknesses listed and questions posed below, and look forward to further discussion.

[W1] The comparison between DIVA and PLR/ACCEL is a bit of an apples-to-oranges comparison. [...] the fairest comparison would be against CLUTER or DRED.

[Q1] How would this method compare to CLUTR or DRED?

We have added each of the suggested approaches (SAMPLR, CLUTR, and DRED) to the related works section, which will appear in our camera-ready draft. Our assessment of each of the individual works—see conclusions below, and full assessments attached as "official comments"—is that none are as appropriate a baseline choice as ACCEL, despite some similarities (especially DRED) to our problem setting.

SAMPLR We agree with the reviewer’s assessment that SAMPLR is less relevant for direct comparison to DIVA since SAMPLR requires access to the simulator itself (and the direct downstream sampling distribution, which we do not assume access to either). However, we believe SAMPLR’s relevance to the meta-RL setting (meta-RL potentially suffers even more from this bias) warrants mention in the related work section---an addition which will appear in the camera-ready draft.

CLUTR By training a VAE on levels sampled from the generator distribution, which lack greatly in diversity in DIVA’s setting, CLUTR would fail to learn a generator that produces phenotypically diverse levels. ACCEL, by performing mutations to existing levels, is able evolve its levels towards greater diversity over time, making it a stronger baseline for our setting. Due to its relevance to future work, we have cited and added discussion of this work to the manuscript, which will appear in the camera-ready draft.

DRED The biggest difference between DRED and DIVA’s settings is that DIVA does not assume access to the underlying level parameters, but rather only the feature values---and these alone are used to define the archive. This difference makes DRED suffer from the same issue as CLUTR when faced with the setting that DIVA operates in: if the parameter space cannot produce diverse levels for the VAE training, then the resulting model will produce levels just as lacking in diversity. Resampling a la PLR will have a similar effect as our PLR baseline. Thus, as was mentioned of CLUTR above, ACCEL remains a stronger baseline than DRED for this reason. That being said, DRED’s close resemblance to our setting and possible applicability to future work (like CLUTR), we have cited and added discussion to our manuscript, which will appear in the camera-ready draft.

[W2] It occurs to me that DIVA + UED [...] could be used as an algorithm for decisions under ignorance, and would possibly be quite a good algorithm. It may be worth running [...] on the traditional maze, F1, and bipedal walker environments and transfer tasks to check if it consistently outperforms existing methods.

[...]

The limitation of online agents evaluations is a real bottleneck for the UED approaches which DIVA avoids, but it is a bit of an odd comparison as DIVA generates the model once offline, and thus isn't aiming to be adaptive towards current agent performance. It seems like DIVA looses the benefits of adaptivity by completely removing online agent evaluations. I would be interested if a DIVA-like approach with limited online evaluations could keep the best of both.

We would like to note that the DIVA+ algorithm (which demonstrating learning benefits when the archive is “misspecified”) is indeed a “DIVA-like approach with limited online evaluations”, so the reviewer is correct in identifying the promise of such a combination. And we agree that this kind of approach warrants more exploration; we include these preliminary results simply to showcase the promise of an approach of this nature as a potential future direction, and leave it to future work to both (1) provide a more principled integration of UED/QD (our DIVA+ algorithm is just one such combination), and to (2) produce more extensive results with such an approach on relevant domains, such as the ones the reviewer has listed.

[W3] It seems like the assumption about the probability of generating a useful level from the simulator underlying DIVA is similar to the assumption necessary for ACCEL or PLR. Both of these approaches should work if there is a ~0.01 % chance of generating a useful level. The boot-up time would just be a bit slower to get the initial buffer of levels.

While the reviewer is correct in highlighting that this general assumption is shared between DIVA and ACCEL (and PLR), we demonstrate that DIVA is able to work with generators that produce smaller percentages of useful levels. In practice, the boot-up time for ACCEL (and PLR) is more than a bit slower. Because ACCEL must simulate the agent on each environment, and DIVA only requires rendering the first timestep, then ACCEL is $|\tau| \times H$ times slower if we wish to perform just as many evolutionary updates as DIVA (in practice rendering the first timestep to compute features may be slower depending on the environment, but the more general point still holds). Because DIVA can produce far more evolutionary updates in the same amount of time by avoiding simulating the agent on every level (and DIVA+ avoids this too by only simulating levels that constitute the final archive, after all the initial QD updates), DIVA can work more effectively with environments where generating a useful level is even rarer. This is a point we tried to get across in Section 3 (Problem setting), but for the camera ready we will also include the reasoning we’ve provided above—we thank the reviewer for prompting us to consider this point more closely in our discussion!

In response to the authors' detailed responses to my and other reviews, I have two comments:

On the question of evaluation domains and their parametrisation. The authors note that the reason they deliberately re-parametrise their domains is because most existing evaluation domains are built with meaningful parametrisations by design. I think this point was already understood. My original comment was not to question why this was done, but rather to highlight that this introduces a critical weakness to the paper's argument: that DIVA is likely to be a useful technique for domains where a meaningful parametrisation is naturally not available.

I don't think this criticism is "whataboutism" as the authors' poetically put it. It is well known - perhaps an "open secret" - that while genetic techniques are in principle very general, their efficacy in real-world scenarios often depends critically on the structure of the genotype-phenotype mapping. Indeed, if this map has no exploitable structure then genetic techniques generally reduce to random search. My fear is that by introducing their own simple "scrambling" of the genotype-phenotype mapping, that they have inadvertently introduced structure in this map which results in compelling performance from their algorithm. But it is certainly not obvious whether the required exploitable structures exist in more complex open-ended environments.

To put it succinctly, the authors' main claim is that "Our empirical results demonstrate DIVA's unique ability to leverage ill-parameterized simulators to ..." and I'm not sure this is well supported, as the environments the authors have used may well not be "ill-parametrized" in a relevant way, with respect to more interesting open-ended environments.

On specifying features. The authors' comment that while they appreciate hand-crafting diversity measuring features is a limitation, it is significantly less limiting than having to build a diverse environment generator. This comment is well taken. However, I think my objection is more that the authors make quite a big claim, that "These findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains, and to produce more robust and adaptable agents" and I don't think that claim is well supported, given that they have introduced a new scalability bottleneck. This is less a comment on the research, and more a comment on how it is pitched. I would recommend softening the claim a little so that it is more consistent with what the authors have achieved.

Overall, the authors comments don't significantly change my opinion of the paper's suitability for NeurIPS.

[Edited for formatting only.]