Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling

We thank the reviewer for valuable time and feedback,

• W1. The results.

  A1. It is crucial to note that SHED demonstrates more consistent and superior performance in the more complex BipedalWalker and Maze, as depicted in Figure 3,4. This suggests that SHED offers better generalizability and robustness across different environments. Moreover, the less variability in performance with SHED underscores its stability and reliability, which are essential qualities in real-world applications where consistent performance is critical.

• W2. The method is limited to the types of environments.

  A2. In the maze domain, learning block placement is challenging due to the sequential decision process involved (long horizon and large action space). Random placing blocks (ACCEL) may be viable, as it could result in mazes without solutions—especially under constraints of limited resources. To overcome these challenges, we introduced the use of LLM to manage the maze generation process on a more granular level. This approach not only mitigates the complexities associated with direct block placement but also introduces a degree of variability in the environment generation process. This innovative approach, which is unprecedented in the field of UED research, not only ensures the generation of solvable mazes but also extends the control in environment design. This environment design method has received positive feedback from reviewer EFDP.

• W3. Do all methods use an LLM?

  A3. Yes, in the maze domain, all methods generate mazes using ChatGPT, where we provide several key factors—rather than just single difficulty numbers—that define the teacher’s action space (environment parameters) for maze generation (see Appendix D). We can adjust the 'temperature' value in ChatGPT to enhance the diversity of the environments produced. This capability highlights the flexibility and robustness of our LLM-based method, offering an improvement over ACCEL that directly places blocks, which might generate unsolvable mazes and cannot capture the nuanced intentions for the desired generated mazes.

• Q1. similar to [1]

  A1. Thank you for pointing out this paper, which shares some similar motivations with our work. However, there are some key differences to note:

1. While [1] also adopts a diffusion model to generate trajectories, it lacks specific technical details on how these trajectories are generated. In contrast, our paper thoroughly describes the generation process.
2. Additionally, [1] focuses on upsampling student trajectories, whereas our proposed hierarchical MDP framework emphasizes the challenge of collecting the teacher's experience. Our method specifically upsamples the teacher's experience to improve training efficiency for the teacher.

• Q2. intuition of $\eta=0$?

  A2. As indicated in lines 270-274, when $\eta=0$, the teacher might obtain higher rewards by sacrificing student performance in one subset of evaluation environments to enhance it in another. This contradicts our objective of developing a student agent with general capabilities across all environments.

We have included experiments in the appendix where $\eta$ is set to higher values, as shown in Fig. 8. We observed that the test performance deteriorates in certain environments.

• Q3. How variable results are when using different environment parameters.

  A3. In Appendix E, we provided details on how the number of evaluation environments influences the results. We demonstrate that increasing the size of the evaluation environment set leads to improved transfer performances.

The variability in results is minimal when using different (randomly generated) environment parameters. This stability is due to our explicit discretization of the environment parameters, which ensures that the evaluation environments are sufficiently diverse and thus effectively reflect the student's general capabilities.

However, when using just a few DR levels without explicitly discretizing the environment parameters for evaluation, the performance can be enhanced if the parameters are reasonable and do not heavily feature an OOD environment, as shown on the right in Fig. 3 in the attached PDF. Conversely, if similar evaluation environments predominantly include complex and unreasonable parameters (OOD), the results are notably poorer, as shown on the left in Fig. 3 in the PDF. Therefore, introducing diversity in the evaluation environments is essential to ensure stability in the overall performance.

• Q4. high entropy version of PAIRED introduced by [2].

  A4. We didn't use the entropy version of PAIRED, we will include this in the related work.

• Q5. How easy it is to generate larger environments?

  A5. In our framework, scaling up to generate larger environments with more extensive parameter vectors is feasible. Our framework is designed to be modular, allowing for a straightforward extension of the parameter space. However, the complexity and computational demands will increase with the size of the environment parameters.

• Q6. Incorporate the multi-step generation of PAIRED?

  A6. It is possible to integrate the multi-step generation approach of PAIRED into our framework; however, as mentioned in W2, learning to generate a maze step-by-step is inherently complex. In PAIRED, this process required hundreds of thousands of updates to learn effectively, which is feasible under settings with limited resources.

• Q7. Sample the action in the diffusion process from the teacher's policy?

  A7. Yes, sampling actions from the teacher's policy during the diffusion process is feasible.

• Q8. results

  A8. Please refer to A2 to Reviewer AEHi

• L1. limitation

  A1. While we have outlined some limitations in the appendix, we will revise and expand this to provide a more comprehensive analysis of potential weaknesses and areas for future improvement.

Thank you for your interest in our work and your prompt response. We have done additional experiments to address your concerns.

Q1. SHED performs about the same as ACCEL on lunar lander and bipedalwalker. Shed does slightly better in maze, but that is using a nonstandard way of generating mazes.

A1. We conducted experiments in the LunarLander environment to exclude very challenging test environments when testing zero-shot transfer performance. This is because all approaches perform poorly in these challenging environments, which would narrow the performance gap among them. Because we cannot directly show detailed results, we roughly report the results: After removing the challenging environments and completing 1,000 PPO updates, the average performance of SHED, h-MDP, and ACCEL improved significantly on LunarLander. However, ACCEL remains only slightly weaker than SHED. Here are the results: SHED: -6.019, h-MDP: -12.92, ACCEL: -8.706, Random: -30.65, PAIRED: -126.2.

It is important to note that the training curves of ACCEL and Random show significant variance and fluctuations, while the learning curve of SHED is much smoother. This indicates that SHED can consistently provide suitable environments, enhancing robustness and stability in training.

Q2. Sampling actions from the teacher's policy during the diffusion process is feasible. Why did you not do this?

A2. First, we provide detailed steps for sampling actions from the teacher's policy during the diffusion process in Appendix B.2 to upsample the teacher's experiences. However, we chose to randomly sample actions to generate synthetic trajectories because it is more straightforward, which is the method we used in our work. Second, using a diffusion model directly as a policy to obtain the teacher's actions would require further modifications to train the diffusion model. This approach is explored in works by Wang et al. (citation number 20) and Janner, Michael, et al. [1]. However, incorporating this into our work would significantly increase its complexity, and therefore, we leave this direction for future work. [1] Janner, Michael, et al. "Planning with diffusion for flexible behavior synthesis.". International Conference on Machine Learning. PMLR, 2022.

Q3. Could you at least run evaluation on the same levels that ACCEL used? What is the computational demand of running Bipedal Walker for 30K PPO updates? (or at least something like 5k-10k, 50 doesn't seem enough to learn much)

A3. As reported in the paper, all models were trained on a single NVIDIA GeForce RTX 3090 GPU and 16 CPUs. Training ACCEL for 30K PPO updates can take approximately 5 days, which is quite long. There seems to be a misunderstanding: we trained on 50 environments, where each environment ran for 4 epochs, and each epoch included 5 PPO minibatches, resulting in a total of 20 PPO updates per environment. Across all 50 environments, there are 1,000 PPO updates. Also, We included some of the levels that ACCEL used, such as SmallCorridor and FourRooms. We will include more levels in the final version.

Q4. what details are lacking?

A4. Their work is generally well-explained. One difference between their work and ours is that they lack a more detailed technical illustration of how actions and next states are generated using the diffusion model.

Q5. The SynthER method is a way to upsample data for RL agents, and your teacher is still an RL agent. So I would still argue that it is very similar to what you are doing?

A5. SynthER utilizes a diffusion model to generate synthetic experiences from a limited dataset, which is similar to our approach. This work shares similar motivations with ours. However, the key difference is that SynthER focuses on directly synthesizing the experience dataset for the student agent and can be used for online/offline training. In contrast, our approach involves simultaneously upsampling the upper-level teacher's experiences to further assist in training the upper-level teacher agent efficiently, addressing drawbacks in our proposed hierarchical framework. Additionally, directly synthesizing student trajectories to help train the student agents would be unfair in our comparison, as it might induce extra training in terms of the number of PPO updates compared to other approaches. Overall, while we acknowledge that their work and ours share similar motivations and technologies, the application details are different, as we use synthetic trajectories in the upper-level teacher's experience within our hierarchical framework. We will include their work in related work and provide a discussion.

Q6. Could you run a baseline comparing how well normal ACCEL does that places the blocks directly?

A6. Yes, we conducted an experiment using the maze generation method in ACCEL, which involves placing blocks directly to generate the maze. The maze generation process is as follows:

1. We randomly sample the maze size.
2. We randomly place the start and end positions.
3. We randomly sample the number of blocks.
4. We randomly place the blocks, ensuring that the blocks, start position, and end position are not the same. Here are the rough results:

• Randomly sample maze width and height from {8, 9, 10, 11, 12, 13, 14, 15}, randomly sample the number of blocks from [20, 40]: Results: ACCEL: -9.8

• Fix maze size at 13*13, randomly sample the number of blocks from [20, 60]: Results: ACCEL: -12.7

• Fix the maze size at 13x13, and fix the number of blocks at 60: Results: ACCEL: -15.3

For comparison, here are the ACCEL performances under our maze generation method: ACCEL: -4.98

Additionally, we added a filter function to exclude mazes without a feasible solution until generate a feasible maze, and the results are:

• Randomly sample maze width and height from {8, 9, 10, 11, 12, 13, 14, 15}, randomly sample the number of blocks from [20, 40]: Results: ACCEL: -5.3

• Fix maze size at 13x13, randomly sample the number of blocks from [20, 60]: Results: ACCEL: -8.5

• Fix maze size at 13x13, fix the number of blocks at 60: Results: ACCEL: -7.8

Q7. please rectify all of the minor points in your revised manuscript.

A7. Thank you for pointing out the minor issues. We will revise the paper to address these points accordingly.

We thank the reviewer for valuable time and feedback, and we kindly request the reviewer to consider our clarifications.

• W1. The proposed method introduces significant complexity, particularly in the implementation of the hierarchical MDP and the generative modeling components. This might limit the accessibility and reproducibility of the approach.

  A1. We would like to clarify that the two major components introduced in our work, the hierarchical framework and the synthetic trajectories, can be decoupled from each other and can be used independently. Additionally, the algorithm itself is not overly complex; our hierarchical framework can be viewed as a variant of hierarchical reinforcement learning (HRL), The abstractions in SHED are twofold:

1. Use performance vector $p(\pi)$ to approximate the embedding of the student policy and also serve as the state for the teacher agent, informing the current student agent’s capability.
2. The teacher’s agent is the environment parameters. The teacher’s action $a^u$ is an abstract representation of the next generated environment that is used to train the student agent, allowing the teacher to guide the student’s learning process by setting tailored training environments.

• W2. While the empirical results are promising, the evaluation is limited to a few specific domains. It would be beneficial to see broader applicability across more diverse and complex environments.

  A2. We acknowledge that this might be perceived as a shortcoming of our study, given our reliance on parameterized environments. However, we would like to emphasize that our work introduces a novel application of GPT-generated maze environments, a concept that has not been previously explored in UED literature. This innovation enables us to extend traditionally non-parameterized environments to be modeled and controlled by large language models or large multi-modal language models, broadening the potential for future research and applications. Our approach opens up opportunities for further exploration of diverse and complex environments in future studies.

• W3. Figure 4 is not properly formatted (no values on the axes).

  A3. We have modified the figure. Please see the Figure 1 in the attached PDF document.

• Q1. Why were diffusion models chosen over other generative models (e.g., GANs, VAEs)? Have other models been considered or tested?

  A1. We chose diffusion models due to their recent success in generating high-quality synthetic data, and we have demonstrated their performance in generating good synthetic trajectories in the Appendix C. While we acknowledge the potential of other generative models, such as GANs and VAEs, we did not test them as our focus was not on comparing the effectiveness of different models for generating synthetic trajectories. Given the strong empirical results achieved by diffusion models, we prioritized exploring their application within our framework.

• Q2. How well does the proposed method scale to real-world applications with significantly higher complexity and variability? Have any tests been conducted in such settings?

  A2. Our current study focuses on controlled environments to establish foundational principles, and we plan to test them in real-world applications in future work. It should be noted that we are training across environments, so there is a natural stochasticity/complexity that arises in transitions and rewards. That is to say in Minigrid, an agent in one environment may see a different transition/reward than in another environment and it has to still provide a good action that works across both environments.

• L1. The limitations are discussed in Appendix F.1 but I think the authors should discuss the limitations in the main paper.

  A1. We appreciate your suggestion about discussing the limitations. Initially, we moved this section to the appendix due to page constraints. However, we understand the importance of highlighting these limitations in the main paper, and we will ensure they are included in the revised version.

We thank the reviewer for valuable time and feedback, and we kindly request the reviewer to consider our clarifications.

• S1. The idea of combining this with Genie is incredibly exciting.

  A1. Genie is the first generative interactive environment trained in an unsupervised manner using unlabelled Internet videos. Genie can generate action-controllable virtual worlds described through text, synthetic images, photographs, and sketches. We propose using a diffusion model to upsample the teacher's experience, which is intuitively compatible, given the diffusion model's proven success in image generation. In our approach, the diffusion model predicts the next student policy representation, similar to how Genie forecasts the next visual frame. Thus, our framework (using a generative model) can be leveraged to learn a simple label that assists Genie in generating more coherent and continuous frames.

• W1. There appear to be two confounding features of the method, the new objective for PAIRED and then the synthetic data. Why do they make sense to combine in this way? Maybe we should just do ACCEL with synthetic data for instance?

  A1. Our primary motivation is to train a student agent with general capabilities more efficiently under resource constraints, specifically with a limited number of environments generated and a limited training horizon compared to the open-ended training in previous UED papers. This is why we employ an RL-based teacher like PAIRED, which generates suitable environments more effectively compared to random generation methods like ACCEL. However, PAIRED uses a regret-based metric—a scalar reward that quantifies the difference between the performances of an expert agent and the current agent. Such metrics can produce difficult environments that are challenging for an RL agent to learn from because regret represents the best-case (upper bound) learning potential, not the actual learning potential of an environment. Consequently, it is hard to fully capture the true general capabilities of student agents. To address this limitation, we propose a hierarchical-MDP framework that better aligns with our objectives and uses the performance vector to approximate the student policy representation.

However, the RL-based teacher approach can be costly in terms of collecting the experience required for training, which conflicts with our motivation to reduce interactions between environments and agents due to limited resources. To mitigate this issue, we propose using a generative model to upsample experiences, thus aiding the training of the RL teacher without incurring significant costly interactions between agents and environments.

While it is theoretically possible to combine ACCEL with synthetic data, this approach would not align with other methods. In all our settings, we ensure that student agents maintain the same level of interaction with environments and have the same number of updates. If we were to use synthetic data with ACCEL, it would create an unfair advantage by providing extra training opportunities for the student agent, and ensuring the same number of interactions would make it difficult to demonstrate the superiority of using synthetic data with ACCEL, which is why we did not consider combining ACCEL with synthetic data.

We want to highlight that synthetic data can be decoupled from all UED methods, and our hierarchical framework also provides insights into the Hierarchical RL field.

• W2. Can we get some more rigorous analysis?

  A2. Yes. We have provided the aggregate results after min-max normalization (with range=[-21, 1] in the Partially observable Maze domain in Figure 2 in the attached PDF document. Notably, our method SHED dominates all the benchmarks in both the IQM and optimality gap.

• W3. What happens if ACCEL has access to ChatGPT as an editor and then uses replay?

  A3. Thank you for your insightful suggestions. In our Maze experiment, we have already integrated ChatGPT as an editor for ACCEL and utilized replay. Our findings show that ACCEL with ChatGPT and replay demonstrates higher variance in performance, highlighting the instability of random teachers in identifying suitable environments for training student agents.

• W4. The related work is very light.

  A4. Thank you for highlighting this; we will enhance the related work section to more thoroughly discuss the relevant works in our revised version.

• Q1. The authors say "The hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay". Why did you not do this? Then it would be much clearer to see if it is state of the art.

  A1. Our primary motivation is to train a student agent with general capabilities more efficiently under resource constraints, specifically with a limited number of generated environments and a limited training horizon. Integrating prioritized level replay is less appealing in our framework because we aim for our RL teacher to directly generate tailored environments for the current student agent. This approach focuses on dynamic adaptation rather than replay, ensuring that each environment is specifically suited to the student's evolving capabilities.

Thank the reviewer for the feedback. We understand your concerns and appreciate the opportunity to consider our original motivations.

Our primary motivation, as stated in the abstract and introduction, is to train a student agent with general capabilities more efficiently under resource constraints, specifically with a limited number of generated environments and a limited training horizon. Previous UED papers rely on open-ended training, requiring thousands of generated environments and billions of environment interactions. However, this approach is unrealistic in real-world scenarios due to the limited resources available to construct environments.

To address this challenge, we focus on restricting the number of generated environments (50 environments in our settings) and interaction steps (millions of interactions) to train the student agent to achieve general capabilities. This necessitates designing a teacher agent that can generate suitable environments tailored to the current student's capability level, as opposed to previous methods that relied on random environment generation. Our hierarchical framework addresses this by having the teacher take an approximation of the student policy as the observed state and output environment parameters to generate tailored environments.

However, training the teacher agent is challenging due to the costly collection of the teacher's experience, which requires extensive environment interactions. This conflicts with our original motivation. To mitigate this issue, we propose using a diffusion model to upsample the real collected teacher's experience, reducing the costly interactions needed to gather the teacher's experience.

Additionally, our hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay. Our proposed module is designed to be decoupled from other techniques. For example, if we have a larger budget for generated environments, we can implement a level buffer to store environments with high learning potential and revisit them in future training.

We hope this explanation clarifies our motivation.

We thank the reviewer for valuable time and feedback, and we kindly request the reviewer to consider our clarifications.

• Q1. Fingerprint Policy Optimization (Paul et al, 2019) also models the learning process of the student agent. It would be interesting to explain more about how this paper's idea is related and contributes to this line of thought.

  A1. Thank you for pointing out the FPO work, which is very interesting.

FPO utilizes Bayesian optimization to actively select environment variable distributions, thereby enhancing the efficiency of the policy gradient method. It models the environment's effect on policy updates and optimizes for one-step policy improvement by balancing bias and variance. FPO introduces two low-dimensional policy fingerprints:

State Fingerprint: This fingerprint represents the stationary distribution over states induced by the policy. It involves fitting an anisotropic Gaussian to the set of states visited in the trajectories sampled during the estimation of the policy's performance. The size of this fingerprint is equal to the dimensionality of the state space. Action Fingerprint: This fingerprint represents the marginal distribution over actions induced by the policy. It is approximated as a Gaussian distribution derived from the actions taken by the policy across various states. The size of this fingerprint corresponds to the dimensionality of the action space.

Both fingerprints serve as low-dimensional proxies for the policy, enabling the Gaussian Process used in Bayesian optimization to efficiently model the relationship between the policy parameters, environment variables, and expected returns. These fingerprints allow FPO to condition its optimization on a compact yet informative representation of the policy, facilitating robust policy learning in environments with significant rare events (SREs).

In contrast, our approach introduces an RL teacher designed to learn the student's learning process and directly generate suitable actions and environments. While FPO uses state fingerprints to represent policy states, we employ a performance vector in the evaluation environments. This approach provides a more accurate representation of the student's general capabilities. Another advantage of using RL is its consideration of long-term rewards, which can significantly enhance the student's general capabilities. Our method aims to optimize the learning trajectory of the student agent by generating suitable environments dynamically to its evolving learning needs.

• Q2. It is questionable whether all baselines were fully trained with the right settings. Also, the performance of the proposed algorithm and those of the baselines are statistically too similar to see whether SHED improves over the baselines in Lunar Lander and BipedalWalker benchmarks.

  A2. The reviewer's point is easy to explain. First, our motivation lies in training a student agent with general capabilities more efficiently under resource constraints. Previous algorithms, such as ACCEL, focus on randomly generated environments for open agent training. For example, they train the agent with about 30k PPO updates (1b step), while in our setting, due to the limited generated environment and training horizon, for example, we use only about 1k PPO updates (50 environments) to train the student agent. Therefore, due to resource constraints, there are fewer interactions between the agent and the environment. In this case, the student agent is not the strongest and does not achieve the best performance. Note that we have conducted an ablation study in the appendix, increasing the budget of resource constraints, i.e., more training environments and training horizons. For example, in the left figure of Figure 7, our proposed algorithm SHED can efficiently train the student agent to achieve better general capability in a shorter time horizon, but as the training horizon increases, the performances of ACCEL and SHED converge and tend to be same, which shows that without considering resource constraints, our algorithm can also match the state-of-the-art (SOTA). Under resource constraints, our proposed environment generation process can better generate suitable environments for the current student policy, thereby enhancing its general capability, especially when there is a constraint on the number of generated environments and training horizon.

Second, Figure 3 reflects the average performance over a set of evaluation environments, rather than a single vanilla environment. We consider improving the general capabilities of student agents rather than focusing on a single environment. Please note that our test environments are randomly generated and they are usually much more challenging compared to the vanilla bipedalwalker environment, thus overall performance is much smaller than 300.

• Q3. Figure 4 has no scale on timestep and reward.

  A3. We have modified the figure. Please see the Figure 1 in the attached pdf document.

Thank you for your prompt response.

Q1. Would it be OK to rephase the sentence as following? : ACCEL was trained with 1 billion timesteps to interact with the environment and PPO policy was updated 30,000 times. SHED interacted with the environment with 50 environments and had 1,000 PPO updates to the policy.

A1. Yes, that is true. We will revise the sentence accordingly to improve clarity.

Q2. If the above is true, how many steps did SHED spent interacting with the environment in total for the training?

A2. SHED interacted with 50 environments. Each environment ran for 4 epochs, and each epoch included 5 PPO minibatches, resulting in a total of 20 PPO updates per environment. Across all 50 environments, this amounted to 1,000 PPO updates. In the LunarLander environment, there were around 1,000,000 total environment steps, while the BipedalWalker environment and Maze environment involved around 10,000,000 and 400,000 total environment steps across all environments, respectively.

Q3. Why is SHED's interaction measured in number of environments not in steps.

A3. ACCEL was trained on thousands of environments. We would like to emphasize the motivation of our work is to train a student agent with general capabilities more efficiently under resource constraints, i.e., with a limited number of generated environments. Please note that the number of generated environments is crucial for improving the agent's general capabilities as we only "shallowly" train the agent on each generated environment. Therefore, we use the number of environments as a metric, which provides a clearer understanding of the training conditions. Additionally, we report results in terms of the number of PPO updates to ensure a fair comparison across all approaches with the same number of updates, consistent with other works like PAIRED and ACCEL.

Q4. When you say 50 environments, are we referring to 50 different obstacle courses in BipedalWalker?

A4. This refers to 50 different instances of the environment that the student agent will be trained in to acquire the general capabilities.

Q5. What are the cases that there would be concerns about how many obstacle courses you load for training, instead of how many timesteps the agents spent interacting with the world?

A5. In general, these two terms are correlated. In this work, we focus on training a student agent with general capabilities more efficiently under resource constraints. The number, diversity, and complexity of the environments are crucial for improving the agent's general capabilities as we only "shallowly" train the agent on each environment. In such cases, the number of environments would be a concern.

Q6. Why does figure 3 report only 1000 training steps when the baselines were trained with a scale of a billion steps?

A6. There is a misunderstanding, the 1000 training steps refer to the number of PPO updates, not the environment steps. All training algorithms for UED (PAIRED, PLR, ACCEL, and many others) report results in terms of the number of PPO updates. We also do the same in this paper.