Adversarial Environment Design via Regret-Guided Diffusion Models

We appreciate Reviewer a2dv for the valuable feedback and review. Below is our response to the reviewer's comments and questions.

Weak 1: About the x axis for the plots

We agree that using total steps is disadvantageous for replay methods. However, we want to point out that if we use student updates, it becomes disadvantageous for methods that do not use a replay. We deliberated on this issue when organizing the experimental results, and decided to use total steps based on two reasons; (1) it is common in reinforcement learing (RL) research to evaluate the performance of the online RL algorithm after the fixed number of environmental steps. (2) replay methods prior to ACCEL also used total steps for the x axis [1, 2].

Despite our decision, we realize that there could be a controversy about this issue. Hence, we measure the performance of our method trained for only half of environmental steps used in the original experiment to accommodate the perspective that using the number of policy updates as a metric is more reasonable. When using half of environmental steps, average score of our method in bipedal walker is 127.4 +/- 16.0, and average success rate in maze environment is 0.72 +/- 0.04. We believe the performance is still competitive compared to baselines, and will add this performace to the appendix C.1 and C.2.

Weak 2: About the baselines

We agree that it would be beneficial to compare our method against CLUTR [3] and PAIRED with high entropy and protagonist-antagonist behavioral cloning [4]. However, we want to point out that [3] was proposed to solve combinatorial explosion issue and that the bipedal walker domain is not a target domain. Furthermore, [4] eventually utilizes replay in the bipedal walker, so it is not adequate methods to represent learning-based methods. Based on these points, we did not include them in the baselines, but since experiments in the maze environment is possible, we will conduct additional experiments on these baselines and add the result to the Appendix C.1.

Weak 3: About including the pre-training steps into the x axis

We want to point out that there is no interaction with environments during the pre-training phase since we randomly sample parameters without access to environments and train the diffusion model using the sampled parameters. So we disagree about the reviewer's concern that we should take account the steps from pre-training phase in the x axis. However, similar to the reviewer's point, we agree that the time spent on pre-training should be considered for a fair comparison. To resolve this issue, we will include the performance measured after the number of environmental steps that is adjusted to reflect the time spent on pre-training. Since pre-training took much less time than agent training (9 hours vs 56 hours in the maze environment, and 7 hours vs 92 hours in the BipedalWalker environment), we expect there will be no drastic performance degradation.

Weak 4: Examples of generated environments in the main body

We will resize Figure 7 and Figure 12 and add each to Section 5.1 and Section 5.2.

Weak 5: An additional citation

We appreciate the reviewer for pointing out the remarkable research [5] related to UED. We will add it to our citations.

Question 1: About the meta data plots of ACCEL

The ACCEL paper [6] describes two implementation methods. The difference between the two methods lies in the domain from which the initial environment parameters are sampled. One method samples from the full parameter range, while the other restricts sampling to a range that ensures simple environments are generated. Starting with a simple environment and gradually evolving it is a simple yet powerful idea. However, when using ACCEL as a baseline, it might not be fair because it already incorporates prior knowledge of which parameters create simple environments. Thus, we chose to use the ACCEL implementation that samples parameters from the full parameter range as the baseline. As a result, complex environments are generated at the beginning of training, leading to differences from the original paper's plot.

References

[1] Jiang et al. "Prioritized Level Replay." International Conference on Machine Learning. 2021.

[2] Jiang et al. "Replay-Guided Adversarial Environment Design." Advances in Neural Information Processing Systems. 2021.

[3] Azad et al. "CLUTR: Curriculum learning via unsupervised task representation learning." International Conference on Machine Learning. 2023.

[4] Mediratta et al. "Stabilizing Unsupervised Environment Design with a Learned Adversary." Conference on Lifelong Learning Agents. 2023.

[5] Bauer et al. "Human Timescale Adaptation in an Open-Ended Task Space." International Conference on Machine Learning. 2023.

[6] Parker-Holder et al. "Evolving Curricula with Regret-Based Environment Design." International Conference on Machine Learning. 2022.

Thanks for the response. We misunderstood the reviewer's concern as a suggestion to replace PAIRED with [4]. We now understand the reviewer's point and will include the results of [4] in the main body.

We appreciate Reviewer 1NF7 for the valuable feedback and review. Below is our response to the reviewer's comments and questions.

Weak 1: Minor typo "challenging"

We agree that the word "challenging" in line 253 is redundant, so we will exclude it.

Weak 2: About complexity metrics: shortest path length and number of blocks

We used shortest path length and the number of blocks following prior works [1, 2, 3]. While the number of blocks may not always correlate with an environment's complexity, there is a tendency for environments to become more maze-like and challenging as the number of blocks increases when the goal and start positions are fixed. Thus, considering both the number of blocks and the shortest path length can give a rough estimate of the environment's complexity. Additionally, we agree that a metric considering the lengths of all paths can better explain the complexity of an environment. However, since there are numerous paths to the goal within the time limit, simply averaging these paths does not meaningfully explain complexity. Hence, a carefully designed metric considering the lengths of possible paths is needed, and we believe this is an area that requires further exploration by UED researchers.

Weak 3: About the expression “successfully generates adversarial environments while preserving diversity”

It summarizes the complexity and diversity of the environments generated during the training process, which are shown in Figures 2(c) and 2(d). We would like to point out that it is not about the performance.

Weak 4: What each experiment is trying to test

To help readers better understand the purpose of each experiment, we will improve the explanation in Line 266-272 as follows. "The primary goal of our experiments is to evaluate the generalization capability of the trained agent. To this end, we measured the zero-shot transfer performance in challenging, human-designed environments. Additionally, to understand where the differences in performance originate, we conducted quantitative and qualitative analyses of the curriculum of the generated environments by tracking complexity metrics and generating t-SNE plots."

Weak 5: A third evaluation domain and applicability

Regarding the reviewer's concern about applicability, we would like to point out that most tasks used in current UED research either have a form similar to Minigrid or utilize a continuous parameter space. Hence, sampling environment parameters using a diffusion model is feasible. Therefore, we agree that experiments in third domain will strengthen our work, but not necessary to demonstrate our method's applicability. For deeper discussion on the workable parameterization, please refer to the response to the question 1.

Question 1: About the assumptions on the structure of the environment parameterization

The assumed environment parameterization is one that can be learned by the diffusion model. If the parameter space is continuous, methods such as DDPM [4] can be applied. Even when the parameter space is discrete, methods such as D3PM [5] can be applied. Hence, we believe that our algorithm can cover most tasks that can be handled by existing UED methods, which model the environment generation process as an MDP [1] or randomly sample environment parameters [2, 3].

Even for environment parameters requiring constraints, such as ensuring that a path always exists from start to goal in a maze environment, we can utilize guided diffusion and rejection sampling to meet the constraints. Furthermore, when the constraint is complex, the diffusion model can be an effective solution, as discussed in [6]. Therefore, it can be seen that our method has a potential to handle both continuous and discrete parameters with constraints. However, if the environment itself is challenging to parameterize, it would be difficult to apply ours and other UED algorithms.

We hope this response sufficiently addresses the reviewer's question. If there is an example of a parameterization where challenges in applying our approach is expected, we would be happy to discuss it further.

Question 2: About RGB channels used to describe the maze environment

The representation that we used is a continuized binary, where open spaces are represented by 0 and other elements by 1. And there are two reasons why we use three channels. First, it allows us to represent obstacles, a start point, and a goal point in different channels using binary values, which we believe is a natural representation. Second, it is easy to visulize. It is technically feasible to use a single channel and represent each element with a different value. We expect this would make training even easier than with our representation, as the reduced dimension could lead to faster learning. We hope we have understood the reviewer's question correctly and our respones addresses it sufficiently.

Question 3: About t-SNE embeddings

We generated the t-SNE embeddings using all environment parameters, and each plot displays only the points corresponding to the respective methods. By doing so, we were able to compare the diversity of environments generated by each baseline.

References

[1] Dennis et al. "Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design." Advances in Neural Information Processing Systems. 2020.

[2] Jiang et al. "Replay-Guided Adversarial Environment Design." Advances in Neural Information Processing Systems. 2021.

[3] Parker-Holder et al. "Evolving Curricula with Regret-Based Environment Design." International Conference on Machine Learning. 2022.

[4] Ho et al. "Denoising Diffusion Probabilistic Models." Advances in Neural Information Processing Systems. 2020.

[5] Austin et al. "Structured Denoising Diffusion Models in Discrete State-Spaces." Advances in Neural Information Processing Systems. 2021.

[6] Yang et al. "Compositional Diffusion-Based Continuous Constraint Solvers." Conference on Robot Learning. 2023.

Thanks for the response. We are glad that our answers cleared up the reviewer's concerns.

We appreciate Reviewer TVkr for the valuable feedback and review. Below is our response to the reviewer's comments and questions.

Weak1: ACCEL results in the bipedal walker domain

There are two main differences between the original ACCEL paper and our experiments. First, the domain of environment parameters is different. As shown in Table 6 of the original ACCEL paper [1], ACCEL was implemented in two ways, ACCEL+ and ACCEL++ (we replace dagger of the original paper with "+" for readability in the review format). ACCEL++, which shows better performance, sampled environment parameters from an easy parameter range where the difficulty level is inherently low (Table 1 below or Table 9 of the ACCEL paper). In contrast, ACCEL+ and other baselines generated environments from the full parameter range (Table 2 below). While ACCEL++ shows powerful performance, we decided to use ACCEL+ as a baseline since it can be seen that ACCEL++ uses prior knowledge of which parameters create simple environments. It is one of the reasons why the average score of ACCEL is lower than that of the original paper.

The second difference is that the ACCEL paper recorded performance after the fixed number of policy updates, whereas we recorded performance after the fixed number of environmental steps, following other UED papers [2, 3] and traditional deep reinforcement learning research. Since ACCEL requires episodes without policy updates after mutating replayed environments, the number of environmental steps per policy update is higher than other baselines. Therefore, when using environmental steps as the metric, the performance of ACCEL degrades than that reported in the original paper.

We will include an explanation of this difference in the paper to ensure that readers are aware of these differences. Addtionally, we will add the performance of our method trained for only half of environmental steps used in the original experiment to accommodate the perspective that using the number of policy updates as a metric is more reasonable. In bipedal walker domain, average score of our method after half of environmental steps is 127.4 +/- 16.0, and we believe it is still competitive performance compared to baselines.

Table 1: Easy parameter range in the bipedal walker domain

Table 2: Full parameter range in the bipedal walker domain

Weak 2: Prior works and citations

We appreciate the reviewer for highlighting prior works ([4], [5]) related to our approach. We will add these to related works and reference them to strengthen our claim regarding the addtion of the entropy term and the use of a learned critic. We will also cite POET [6] and GPN [7] to help readers better understand previous research on the general concept of environment design.

Weak 3: No minimax regret UED

We will add an explanation to the related works to clarify that there are UED methods with non-minimax regret objectives.

Weak 4: Periodically fine-tuning the diffusion model

We agree that periodically fine-tuning the diffusion model using newly generated levels could potentially result in a more powerful generator. However, we are not sure that fine-tuning the diffusion model using levels generated by the diffusion model itself would genuinely enhance the generator's capabilities. One possible approach to address this issue is to use another UED algorithm in parallel to generate levels that our diffusion model cannot create. We believe it could be an interesting future work.

Weak 5: Clarity

Additional plots: We will include per-transfer bar plots and bootstrapped CI plots below Table 5 and 6 to improve the readability of the experimental results.

Figure 4 and Figure 9: We will include Figure 4 and its explanation in the beginning of Section 5.1. We will also include Figure 9 after Section 5.2 and add its explanation, which is described in Appendix A.5, after Section 4.4.

ADD acronym: The ADD acronym stands for "A"dversarial environment "D"esign via regret-guided "D"iffusion models. We welcome any suggestions for a better acronym.

Question 1: Replacing equation 12 with a PAIRED-style loss

We have not yet attempted to replace Equation 12 with a PAIRED-style loss, which predicts the difference between the antagonist and protagonist. However, we believe it would be a worthwhile experiment and could potentially yield competitive performance.

References

[1] Parker-Holder et al. "Evolving Curricula with Regret-Based Environment Design." International Conference on Machine Learning. 2022.

[2] Jiang et al. "Replay-Guided Adversarial Environment Design." Advances in Neural Information Processing Systems. 2021.

[3] Garcin et al. "DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design." International Conference on Machine Learning. 2024.

[4] Mediratta et al. "Stabilizing Unsupervised Environment Design with a Learned Adversary." Conference on Lifelong Learning Agents. 2023.

[5] Bhatt et al. "Deep Surrogate Assisted Generation of Environments." Advances in Neural Information Processing Systems. 2022.

[6] Wang et al. "Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions." arXiv:1901.01753. 2019.

[7] Bontrager et al. "Learning to Generate Levels from Nothing." IEEE Conference on Games. 2021.

We appreciate Reviewer rCfY for the valuable feedback and review. Below is our response to the reviewer's comments and questions.

Weak 1: Is the diffusion process critical for the success? Can you train PAIRED with a differentiable regret?

The proposed algorithm critically relies on both the diffusion process and differentiable regret. Without a differentiable regret, using only the pre-trained diffusion process would theoretically result in generating random environments, failing to create a meaningful curriculum. This outcome is evident in Figures 2 and 3, where the performance of ADD without guidance is shown. Additionally, since differentiable regret is an approximation of the previously used regret estimation method, it is possible to train PAIRED [1] using this approach, but significant performance improvement is not expected.

Weak 2: is the entropy term critical? Can you train the diffusion process without it and get comparable results?

The entropy term is also critical. Adding the entropy term ensures that the distribution from which environment parameters are sampled becomes the softmax distribution of regret, as shown in Equation 8. Without the entropy term, it would be challenging to implement the sampling of environment parameters that maximize regret using the diffusion process, to the best of our knowledge. However, by setting \omega to a exteremely high value, we can effectively simulate the absence of the entropy term. We will include an ablation study to explore the performance changes with varying \omega values in the appendix. We report partial results in the following table.

Table 1: Performance in accordance to \omega in partially observable navigation task. Blanks stand for experiments that are not finished yet.

Additionally, as reviewer TVkr pointed out, there is a prior work [2] that claims adding high entropy bonus when training neural generator of PAIRED yields better performance. This supports our claim that the entropy term plays a critical role.

Weak 3: Can you explain the reason why you trained the critic model with binned returns rather than using actual returns in a regression objective?

The reason for training the critic model with binned returns, rather than using actual returns in a regression objective, is to obtain a differentiable regret estimate. We need to estimate the maximum and average returns, and to accurately reflect the stochasticity of the environment and policy, we trained a network to predict the distribution of returns, similar to distributional RL. We used binned returns following one of the foundational distributional RL studies, C51 [3]. However, it is also possible to use methods like Implicit Quantile Networks (IQN, [4]), which learn the distribution with actual returns as the output, and are known to yield better performance.

Weak 4: How critical is the pre-training for the scoring function? How does the performance change with less or more number of environment samples?

Pre-training the scoring function is an essential step. Through pre-training, the diffusion process becomes capable of sampling a wide range of environment parameters. This diversity enables the diffusion process to generate meaningful curricula when guided by differentiable regret. As the number of environment samples used in pre-training increases, we expect the diffusion process's ability to generate diverse environments to improve, thereby enhancing overall performance. We are conducting experiments with a smaller number of environment samples , and we will add the results to the appendix.

References

[1] Dennis et al. "Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design." Advances in Neural Information Processing Systems. 2020.

[2] Mediratta et al. "Stabilizing Unsupervised Environment Design with a Learned Adversary." Conference on Lifelong Learning Agents. 2023.

[3] Bellemare et al. "A Distributional Perspective on Reinforcement Learning." International Conference on Machine Learning. 2017.

[4] Dabney et al. "Implicit Quantile Networks for Distributional Reinforcement Learning." International Conference on Machine Learning. 2018.

Dear Reviewer rCfY,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

We are writing this comment to help reviewers understand some of reviewer rCfY's concerns and our response, as well as to share the complete results of additional experiments. First, in response to the concern about the importance of the entropy term (Weak 2 in the rebuttal), we explained that the entropy term is crucial because it shapes the target distribution in a way that allows environment parameters to be sampled by the diffusion model. Since the influence of the entropy term diminishes as \omega in Equation 7 increases, we mentioned that we were conducting an ablation study on different \omega values and would include the results. The completed experimental results are shown in Table 1 below. From the results, we observed that as \omega becomes large, performance decreases, which highlights the importance of the entropy term.

Table 1: Performance in accordance to \omega in partially observable navigation task. Blanks stand for experiments that are not finished yet.

Next, to address the concern about the number of samples used during the pre-training phase (Weak 4 in the rebuttal), we trained the diffusion model using 1 million samples, which is 100 times fewer than in the original experiment, and measured the performance of the proposed algorithm. The result is a mean success rate of 0.76 +/- 0.07 in the partially observable navigation task. This is about 11% lower than the result reported in the main text (but still outperforms the baselines), which supports our claim that a larger number of samples used in pre-training would lead to better performance. Additionally, we would like to point out that since we are dealing with an unsupervised setting and the samples used in pre-training are generated through random sampling, there is no need to worry about data scarcity.

We hope that this additional comment helps the reviewers better understand our rebuttal.