Implicit Curriculum in Procgen Made Explicit

We highly appreciate the feedback from the reviewer-2ABJ and will address each point in detail below.

---

W1. Issues Regarding to Literature Comparison

We appreciate the highlighting of Minigrid and Minihack as notable works similar to our C-Procgen, and regret the oversight of not mentioning them in our initial submission. Additionally, we acknowledge the omission of 'Unsupervised Environment Design' in our submission, despite its relevance and prominence in the field.

1. We agree that Minigrid and Minihack are widely used in curriculum learning due to their customizable environments. While Procgen offers a broader variety of challenges, it lacks the customizable features essential for detailed control, which we have incorporated into C-Procgen.

2. After reviewing 'Unsupervised Environment Design', we recognized its overlap with C-Procgen, especially in using configurable parameters for training environments. We will incorporate terms like 'underspecified environment' and 'free parameters' to clarify and deepen our discussion!

---

W2. Typos

We appreciate the detailed feedback on the typographical errors in our paper. We have thoroughly reviewed the manuscript and corrected these issues to provide a revision. Thanks to the reviewer for highlighting these points.

---

W3. Lack of Clarity and Definitions

We appreciate feedback from the reviewer on clarity. We will define key terms like 'context parameters' earlier in the text and refine our descriptions of the learning process to address these concerns more directly. And, we would love to align our terminologies with those used in 'Unsupervised Environment Design' in our revision.

---

W4 & Q5. Lack of Metric Explanation

We apologize for the oversight and will address this in our revision. The score improvement which is defined as $\frac{score_{\text{PLR}} - score_{\text{PPO}}}{score_{\text{PPO}}}$. Here, $score_{\text{PLR}}$ and $score_{\text{PPO}}$ represent the scores achieved by the PLR and PPO algorithms respectively.

---

W5 & Q3. Loss Specification

We apologize for not specifying this clearly in the manuscript. The 'loss' referred to in the paper is the PPO Loss, which is composed in a weighted sum of three terms: $l_{policy}$, $l_{value}$, and $l_{entropy}$.

---

W6. Obviousness of Training Progression Findings

As the reviewer pointed out, it might seem unsurprising that agents collect more samples from harder contexts as they learn to prolong survival, especially when episodes end upon the agent's 'death.' However, this is not the only interpretation.

• For example, in a maze scenario, an agent might accumulate many steps in challenging contexts not because it is effectively learning to navigate but because it cannot quickly exit the maze, potentially ending the episode with zero reward after expending many steps.
• While initially this may not seem surprising, on reflection, the implication that an agent spends significant steps in unrewarding contexts is noteworthy.

---

W7.

Reviewer: It is unsurprising that more challenging contexts results in higher loss values.

We observed that the contexts resulting in higher loss values dynamically change throughout the learning process, rather than being consistently fixed on more challenging contexts. Interestingly, this phenomenon appears to be nearly universal across all environments tested.

---

W8.

Thanks for the suggestion! We have included learning curves with more detailed metrics information in the global response PDF attached. Upon comparison, we found that the performance of PPO on C-Procgen is largely consistent with its performance on Procgen.

---

Q1:

Reviwer: On line 142, how does a continuous value bomb_prob produce only 6 unique contexts?

Although bomb_prob can be a continuous value, in the game logic, it is restricted to three discrete values: 0, 0.25, and 0.5. The six unique contexts are determined by these discrete values and the number of sections (num_section). Specifically, the contexts setting of num_section and bomb_prob are as follows: [1, 0], [2, 0.25], [3, 0.25], [3, 0.5], [4, 0.5], [5, 0.5],

---

Q2:

Reviwer: On line 149, if gap_range and bomb_prob can be individually configured, how are they correlated?

In the game logic, as gap_range increases, bomb_prob also increases. They are correlated in the following three combinations: when gap_range is [0], bomb_prob is 0; when gap_range is [0, 1], bomb_prob is 0.25; and when gap_range is [0, 1, 2], bomb_prob is 0.5.

---

Q4:

Reviwer: Is it still possible to seed a C-Procgen environment with the original procedural content generation?

Yes, it is indeed possible to replicate the original Procgen environments in our C-Procgen framework.

---

Q6:

Reviwer: Have the authors investigated whether previous work has identified higher loss values in challenging states or environments?

We are aware of methods such as PLR and PER (Prioritized Experience Replay) where higher loss samples are leveraged to accelerate training. However, we may not have encountered specific studies explicitly identifying higher loss values in challenging states or environments. We would appreciate any references the reviewer could provide to enrich our understanding and literature review.

---

Limitation

Thanks to reviewer for highlighting this limitation regarding the generalizability of our findings beyond PPO and their applicability to environments other than Procgen.We agree that exploring the effectiveness of our observations with value-based algorithms and in other environments would be beneficial. However, as our paper focuses on observations and discoveries within the C-Procgen environment, this is our primary area of emphasis.

---

Once again, we thank the reviewer-2ABJ for his/her detailed feedback. We will address the mentioned limitations, clarities, and omissions in our citations thoroughly in our revision.

Dear Reviewer 2ABJ,

We hope you are well. We tried to consider and address the points you raised in your review, including literature comparison, clarity, and the evaluation metrics. If there’s anything further you’d like us to clarify or discuss, we would greatly appreciate your feedback.

Thank you again for your time and thoughtful review!

We value the detailed feedback from the reviewer-7pFG and address each point in detail below.

---

W1. Insights on Curriculum Design & Limitation

Thanks for the reviewer-7pFG's feedback! While at first glance the implications for curriculum development might not be immediately apparent, our study does provide several insights that can be leveraged to enhance curriculum methods:

1. Dynamic Adjustment of Context Sample Distribution: As detailed in Section 3.1, by analyzing the learning dynamics across different contexts, we observed variations in the 'Loss Per Sample' —- an indicator of how efficiently an agent assimilates signals from its environment. These variations suggest that agents learn different amounts of information from different contexts at different stages of training. However, the distribution of context samples mostly focuses on the contexts generating lower loss values, which may not be the most informative for learning. This insight supports the idea of dynamically adjusting the context sample distribution based on the loss signal distribution to make learning more efficient.
2. Influence of Context Gaps on Curriculum Learning: As discussed in Section 4, the gaps between contexts significantly impact the effectiveness of the curriculum. Section 4.1 shows that missing certain contexts can hinder an agent's learning progress, while Section 4.2 demonstrates that adding new contexts can facilitate faster transfer learning to new situations. It implies that a well-designed curriculum should consider the distribution of context samples and the gaps between contexts to optimize learning efficiency.

These insights suggest that tailored curriculum designs should not only focus on adjusting to the agent's learning needs but also optimize the distribution and progression of learning contexts to improve overall training effectiveness. This is precisely the purpose for which our C-Procgen was proposed: to design a system that meets these needs.

---

W2.

Reviewer: The analysis, and I think conclusions, are limited to procgen, and possibly similar games. For instance, if we have a domain where all episodes have an equal length, then the different samples per context results would not hold.

Thanks to the reviewer-7pFG for his/her insightful comments. Indeed, our experiments are based exclusively on C-Procgen, as the development and analysis of this new platform we introduced constitute the primary objective of our work.

Additionally, it is true that in domains where all episodes have equal length, the distribution of samples per context would naturally be uniform. However, this does not conflict with our findings: the distribution of loss signals across different contexts does not align with the distribution of samples, which contributes to inefficiencies in curriculum learning. This misalignment is particularly noteworthy because the distribution of loss signals is dynamic, not fixed or uniformly distributed.

---

W3.

Reviewer: The LPE/LSE seemed somewhat tacked on, can this metric be used in some way by a method/designer?

As a post hoc evaluation metric, LPE (Loss Production Efficiency) cannot be directly used by designers or during method development.

However, LPE can be utilized to evaluate the efficiency of loss signal production of a whole learning process, providing insights into how effectively an agent learns from its environment.

In practice, rather than using LPE directly to adjust curriculum strategies, methods can leverage monitoring of the Loss per Sample to dynamically adjust the distribution of contexts within the curriculum.

---

W4. Clarity of Definition & Q1

Thanks for the reviewer-7pFG's valuable feedback! We will enhance the clarity in our revision regarding the definitions and calculations pertinent to our methodology.

Regarding to score improvement, it is utilized as a metric to quantify the performance enhancement that the PLR (Prioritized Level Replay) method brings over the standard PPO. The calculation of score improvement is defined as follows: $\frac{score_{\text{PLR}} - score_{\text{PPO}}}{score_{\text{PPO}}}$. Here, $score_{\text{PLR}}$ and $score_{\text{PPO}}$ represent the scores achieved by the PLR and PPO algorithms respectively, after training for 25M steps under the same configuration.

Additionally, the 'loss' referred to in the paper is the PPO Loss, because all our experiments are based on the PPO. The PPO Loss is in a weighted sum of three terms: $l_{policy}$, $l_{value}$, and $l_{entropy}$, and it is this weighted sum that we record.

---

W5.

Reviewer: There are many figures, for all of the games, and this makes getting an overview of the results somewhat challenging.

Thanks to the reviewer-7pFG for his/her feedback regarding the number of figures in our submission. We acknowledge that the abundance of figures for each game might complicate obtaining a clear overview of the results. To address this, we will attempt to use broader metrics such as Loss Production Efficiency to summarize key phenomena more succinctly in our revision. Given the significant diversity in game design and context spaces within Procgen, we initially included numerous figures to substantiate our points comprehensively.

---

Q2.

Reviewer: Line 212-216, should it be LPE? If not, what does LSE mean?

We acknowledge this and appreciate the reviewer-7pFG for pointing it out. The LSE at line 212-216 should indeed be LPE, which means the Loss Production Efficiency introduced at [line 201].

---

Q3 & Q4.

Thanks to the reviewer-7pFG for catching these typos. We will correct them in our revision!

---

Q5. Clarity of Figure5

We appreciate the reviewer-7pFG's suggestion to improve the clarity of Figure 5. The primary purpose of using detailed figures was to visually highlight which contexts within the environments were masked, a detail that may be crucial for nuanced analysis across different games.

---

We sincerely thank the reviewer-7pFG for his/her/their prompt response! We believe our work primarily makes contributions to the field in two aspects:

1. We have introduced C-Procgen, which incorporates functionalities that enrich the community’s research tools, enabling more dynamic and versatile experiments in reinforcement learning.
2. Leveraging C-Procgen, we have identified an implicit curriculum in learning dynamics — a concept previously unexplored — that could inspire future studies in curriculum learning.

---

C-Procgen

• Curriculum and Transfer Learning: C-Procgen aims to enhances curriculum learning by allowing for dynamic adjustments to the learning context, helping to gauge and improve agent adaptability in varied scenarios.
• Context-aware Learning: C-Procgen facilitates context-aware reinforcement learning, where agents adjust their strategies based on the current context.
• Diversity and Edge Cases: C-Procgen contributes to the diversity of training environments and supports the creation of edge cases. These are vital for assessing agent resilience in non-standard conditions.
• Environment Design: C-Procgen enables precise manipulation of game mechanics and environment attributes to study their influence on agent behavior and to experiment with new game dynamics.

---

Implicit Curriculum

With the help of C-Procgen, we have conducted extensive experiments to analyze learning dynamics within Procgen games. Our findings reveal the presence of an implicit curriculum, where despite a uniform distribution of game contexts, the learning progression of agents is notably non-uniform. Interestingly, agents tend to learn games in a specific order, favoring those that are neither too easy nor too challenging. Furthermore, the dynamics we observed, including the loss signals, the distribution of samples, and the performance of agents across different contexts, all exhibit notable interrelationships.

This insight diverges from previous works[1][2] that primarily focus on explicitly crafting curricula to guide agents from simpler to more complex tasks. Instead, our observations suggest that the learning process itself naturally conforms to a curriculum structure — a phenomenon that has not been discussed previously. This novel perspective enhances our understanding of how agents adapt and learn, opening a new venue for future research.

---

We appreciate the reviewer's constructive and insightful suggestions. We are happy to address any further questions and continue the discussion.

[1] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.

[2] Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., & Levine, S. (2020). Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33, 13049-13061.

We truly appreciate the constructive feedback from the reviewer-h4d8 and will respond to the points raised as follows.

---

W1. Termination Conditions

We appreciate the insightful feedback from reviewer-h4d8. While termination conditions are indeed an important aspect to consider, they do not detract from the observations reported in our study.

1. As noted, certain environments in our study indeed terminate before reaching the maximum number of steps per episode. This premature termination can shift the sample distribution across different contexts from what might be expected under uniform distribution conditions. However, as highlighted, this does not alter our finding that the sample distribution does not align with the distribution of loss signals per sample in each context, which may lead to inefficiencies in curriculum learning.

2. Additionally, we concur with reviewer-h4d8 that incorporating a deeper analysis of termination conditions would enrich the information provided in Section 3! In our revision, we would love to also expand our discussion to include more details on the reward functions and dynamics of these environments, thereby providing a more comprehensive evaluation of how the implicit curriculum progresses.

---

W2. Interpretation of Implicit Curriculum

Thanks to reviewer-h4d8 for the thorough review and the insightful queries raised. It appears that our phrasing may have led to a misunderstanding regarding the interpretation of the 'implicit curriculum' and the intent of Section 3.2.

1. Firstly, regarding the interpretation of the 'implicit curriculum', we believe that the combination of different game contexts generated by various level inherently forms an implicit curriculum. This does not necessarily conflict with early termination of episodes. As reviewer-h4d8 noted, the agent does not deliberately form a curriculum from the environmental contexts; however, for the agents, these varying contexts of difficulty inherently serve as a curriculum, guiding their learning progression.

2. Additionally, I would like to clarify a point regarding the agent's focus during training. Our discussion in the manuscript seems not assert that 'the agent's focus shifts during training without a curriculum'. Instead, we examine how the distribution of loss signals and samples across contexts influences the learning process. We introduce the concept of Loss Production Efficiency to assess this dynamic. This metric helps us understand the effectiveness of a curriculum and the agent's interaction with it.

We appreciate the opportunity to clarify these points and will ensure that our revision reflects these clarifications to avoid any potential misinterpretations.

---

W3. Evaluation Metrics

Thank you for your observation regarding Section 3.2, which indeed focuses on utilizing the Prioritized Level Replay (PLR) to illustrate the mismatch between loss and sample distribution in the C-Procgen environments.

We are grateful to the reviewer for bringing to our attention the studies by Klink et al. (2022) and Huang et al. (2022). Upon a thorough review of these works, we agree that they offer valuable insights and present methodologies that are excellent and relevant to our field of study. These studies provide advanced approaches to curriculum reinforcement learning, which could enrich our understanding and application of similar concepts in future research.

However, due to the significant variability across games in Procgen, replicating similar presentations to demonstrate the learning dynamics of agents under a curriculum poses challenges. Each game's unique characteristics and context spaces necessitate a tailored approach when applying curriculum learning strategies, which our current focus on PLR aims to address in context-specific manners.

---

Q1. Consistency of Metrics

In our Figures 2, 3, and 5, the range of the score metric is consistent across different training stages and settings. For other metrics, they are normalized on each heatmap for clarity and visibility. Comparing these metrics, especially 'loss per sample', at different times can be challenging due to their significant scale variability during training. We recognize the importance of this clarification and will address this explicitly in the versions to ensure the data's context and limitations are well understood.

Dear Reviewers and Area Chairs,

We sincerely thank the reviewers and area chairs for the time and effort they put into reviewing our submission. The reviewers' thoughtful feedback and suggestions have been especially valuable in improving our work.

We deeply appreciate the reviewers' insights and the guidance provided. Thank you once again to everyone for their support!