Heterogeneous Adversarial Play in Interactive Environments

Dear Reviewer fmid,
Thank you for your thoughtful feedback. We try to answer your questions as follows:

HAP appears to underperform on easy tasks, as shown in Table 1, limiting its generalizability.

We don't see this as an weakness. It is indeed expected that HAP may underperform or show no significant advantage on easy tasks, as HAP does not modify the underlying RL backbone or introduce new model capacity; rather, it focuses on adaptive curriculum sequencing to facilitate learning, particularly in more complex or diverse task environments. For simple tasks where standard RL algorithms already learn efficiently without advanced curriculum scaffolding, HAP does not necessarily provide a theoretical guarantee of improvement, but as shown the difference is not that significant.

Missing reference in Line 256 for the TSCL baseline.

[1] Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2019). Teacher–student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9), 3732-3740.

The paper lacks a detailed analysis of learning trajectories, despite claiming (Lines 80–84) that HAP enables phased skill reinforcement and context-aware difficulty scaling. It would be insightful to compare how other curriculum learning baselines adapt task selection over time in Figure 2(c).

Thank you for this suggestion. We have actually included HAP's task selection trajectory in Figure 2(c), which demonstrates our method's phased skill reinforcement and difficulty scaling. We chose not to include all baseline methods' task selection trajectories in the same figure, as displaying multiple overlapping curves would create visual chaos and make it difficult to discern the key patterns. You can always run baselines and log their task selection with different baselines. We consider adding this part to the Supp in revision.

Lack a detailed description of the optimization process of teacher and student models.

We kindly refer you to the Supp Sec B for more details.

Have you conducted ablation on some key components, such as cold-start, entropy regularization, and task selection lower bound?

Yes. We conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:

Easy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.

What is the teacher’s update frequency vs student steps? Did you try asynchronous updates?

As described in Supp Sec C, the update frequency of the teacher is task-dependent and tuned for each experimental setting. We have also experimented with asynchronous updates, but observed no significant improvement in student learning performance. Additionally, updating the teacher typically requires a full evaluation of the student, which can introduce a time bottleneck, especially in more complex environments. As a result, we choose update intervals that balance adaptation speed with computational efficiency. We will further clarify these implementation details and their impact in the revised manuscript.

How sensitive is the teacher’s task-selection capability to the data budgets?

In all of our experiments, the teacher utilizes student reward history, as it is important not only for assessing the current state but also for understanding learning trajectories and longer-term trends in student performance. We report the result without using student history below:

These results show that incorporating a certain amount of history is beneficial; however, using too much history may not accurately reflect the student's current learning status and can lead to decreased performance.

Why does HAP perform poorly on the Minigrid task?

No. HAP outperforms the other baselines on the more challenging tasks (middle: 0.46 vs. 0.34; hard: 0.20 vs. 0.18). HAP only performs slightly worse on the easy tasks (0.92 vs. the best score of 0.98), which, as discussed above, can be explained by its focus on more challenging tasks.

Thank you for your insightful review. We sincerely welcome your feedback.

I noticed that your input has not yet been recorded in the ongoing discussion. As the Review-authors' discussion deadline is approaching, I would like to kindly remind you to review the author’s response and contribute your comments to the discussion in a timely manner.

Dear Reviewer wSJi,
Thank you for your thoughtful feedback. We try to answer your questions as follows:

While real-world tasks often lack clear difficulty labels, the absence of a manually ordered (easy-to-hard) curriculum baseline limits the evaluation of HAP, as it omits a fundamental reference point for measuring the benefits of automated curriculum learning.

Thank you for this important suggestion. In response, we have conducted additional experiments that include a manually ordered (easy-to-hard) curriculum baseline. Instead of training tasks one-by-one, we trained them group-by-group and, finally, trained on all tasks together to help mitigate the forgetting problem (although this issue remains significant). The results are presented in the revised manuscript and are summarized below:

We didn't include this baseline in the original submission because manually designed curricula introduce their own potential for bias and inequality. For example, the effectiveness of a manual curriculum strongly depends on subjective choices in task ordering, switching policies, and definitions of difficulty, which can vary significantly between designers. We will add these discussions in the revised manuscript.

the lack of clear descriptions for how curriculum learning baselines (e.g., TSCL, EXP3) are configured undermines the transparency of the experimental setup.
The application of HAP to supervised learning tasks is only briefly mentioned, with insufficient methodological detail, leaving it unclear how the adversarial setup is adapted beyond the reinforcement learning context.

For curriculum learning baselines, they are mainly based on their official github repos. For the supervised setting, it is aligned with the standard supervised curriculum learning paradigm, in that it adaptively selects which samples to present to the learner based on their evolving performance. We formulate HAP task selection around label-wise performance: each label or class is considered a distinct sub-task. The teacher observes the student model’s evolving accuracy or loss per label and uses this information to decide which label(s) should be emphasized in subsequent training steps. The implementation of these baselines are based on the benchmark CurBench [1], following the default and standard hyperparameters. We will add further clarification to the baseline implementaitons and hyperparameters in revision.

[1] Zhou, Yuwei, et al. "CurBench: curriculum learning benchmark." ICML.

it remains uncertain whether HAP’s effectiveness is consistent regardless of the underlying learner, raising questions about its framework-agnostic generality.

We have described the RL backbone and model architectures used in HAP for each experiment in Supp Section C. In all three environment settings, HAP demonstrates improved performance compared to baselines using the same RL backbone. The effectiveness of HAP is definitly linked to the choice of learner, as different learning algorithms possess varying capabilities and may benefit differently from curriculum adaptation. Our paper presents results showing that HAP consistently outperforms standard baselines when applied with both A2C and PPO learners. We will include additional experimental results with other baselines and learning algorithms in our revised manuscript.

Why was there no baseline that follows a manually ordered easy-to-hard curriculum? While real-world task difficulty may not be explicit, such a baseline would serve as a useful reference point for evaluating HAP’s effectiveness.

With a set of diverse tasks, intuitively, manually ordering all tasks from easy to hard becomes impractical and unlikely to reflect an optimal or even reasonable progression. The complexity and variety of the tasks mean that an imposed manual ordering is unlikely to generalize well or yield competitive performance. That said, based on reviewer feedback, we nevertheless implemented a simplified version of this baseline by grouping tasks in rough difficulty levels and training in group-wise order. See the discussion and results above.

How does HAP scale with increasing task space size or task granularity?

Scaling adversarial frameworks to very large task spaces does present significant challenges. In our work, we specifically address this issue by introducing a Cold Start mechanism and an additional entropy loss term, both of which help constrain the teacher's task selection trajectory and encourage balanced exploration.
Empirically, we observe that our model (as described in Supp Section C) can effectively control the teacher’s task proposals when the candidate task set is around 10 or fewer. This can be further adjusted by tuning the hyperparameter λ in the entropy loss, which regulates the diversity of proposed tasks. However, we note that the optimal setup is influenced by both task complexity and the learning capacity of the student model.

You mention entropy regularization in the teacher’s loss to mitigate underfitting. Is there any ablation study to quantify its impact on training stability and performance?

We conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:

Easy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.

In Figure 2(b), all task difficulties appear to improve almost simultaneously. Could you explain why this occurs?

The result is in fact an expected outcome given the simplicity of the tasks in this benchmark. As shown in our results, all models eventually converge, indicating that no task poses significant lasting difficulty for the learning agents. In this context, the primary differentiator is data/sample efficiency—that is, how quickly each method enables the learner to master the tasks. With HAP as the curriculum method, the teacher transitions to harder tasks once the easier ones have been sufficiently learned, accelerating progress.
The appearance of concurrent improvement across tasks in Figure 2(b) is also influenced by the sparsity of our evaluation checkpoints for HAP, which may make progress across tasks appear more simultaneous at the sampled points. Figure 2(c) is a more granular view of the learning dynamics where step-wise performance trends are shown in greater detail.

Could you provide further interpretation of the teacher’s task selection behavior and the student’s learning trajectory across Minigrid, CRAFT, and Crafter?

Due to the much larger number of tasks and the significantly longer training sequences in these environments, presenting the full breakdown at a similar level of granularity becomes impractical and potentially confusing. In the revision we will consider including simplified or aggregated illustrations that highlight key aspects of the curriculum dynamics and learning progress for these more complex environments.

Is HAP applicable only to pre-defined, static sets of tasks? How might it be extended to open-ended environments?

We have not yet directly applied HAP to truly open-ended environments, and we acknowledge that significant changes or expansion in task distribution could introduce new challenges, especially for the teacher module—particularly as the number or diversity of possible tasks grows and novel tasks emerge dynamically. A possible solution for such settings is to equip the teacher with mechanisms for continual exploration and adaptation, such as leveraging generative models to propose new tasks or using meta-learning strategies to rapidly adapt the teacher’s proposal policy as the task landscape shifts.

You describe a simplified “Simple Probability Teacher” using student history to guide task selection. Have you conducted any ablation to show whether including history is necessary or beneficial?

In all of our experiments, the teacher utilizes student reward history, as it is important not only for assessing the current state but also for understanding learning trajectories and longer-term trends in student performance. We report the result without using student history below:

These results show that incorporating a certain amount of history is beneficial; however, using too much history may not accurately reflect the student's current learning status and can lead to decreased performance.

Thank you for your insightful review. We sincerely welcome your feedback.

Dear Reviewer FFm9,
Thank you for your thoughtful feedback. We try to answer your questions as follows:

The zero-sum game formulation assumes perfect adversarial dynamics between teacher and student. However, in practice, overly aggressive task generation (e.g., tasks with near-zero solvability) could destabilize training. The paper does not analyze conditions for Nash equilibrium or discuss failure modes (e.g., teacher overfitting to generate unsolvable tasks).
HAP relies on implicit difficulty estimation through reward signals, but lacks a formal metric to define "challenging yet learnable" tasks. This contrasts with prior work (e.g., Graves et al., 2017) that uses explicit difficulty measures. Without this, the teacher’s curriculum may oscillate between trivial and impossible tasks.

We acknowledge that we do not provide a formal analysis of conditions for Nash equilibrium in this particular teacher-student setting, as the equilibrium structure can indeed be complex and may depend on the properties of the environment, the expressiveness of the teacher and student models, and the specifics of the feedback and update mechanisms.

From the formula, the zero-sum objective drives the teacher to find tasks where the student is most likely to fail but can still learn. Ideally, the zero-sum formulation creates a dynamic equilibrium where the teacher finds tasks that maximally challenge the student's current capabilities. For the simple probability teacher, there is indeed no such grantee. We do encounter some cases like you mentioned when training collapses due to pathological teacher task selection, so we further introduce improvements like entropy regularization and cold-start policies to help avoid these cases. See the table below for the ablation results for these modules. Like in conventional RL, there is always exploration and exploitation and the tasks considered happen to be solvable by the agents. But we acknowledge that when scaling to more complex environments, design issues need revisiting.

Could the authors discuss HAP’s potential bottlenecks (e.g., teacher network architecture) when applied to high-dimensional state spaces (e.g., pixel-based inputs)?

From our point of view, applying HAP to high-dimensional state spaces introduces several potential bottlenecks but particularly regarding the teacher network's efficiency. In high-dimensional environments, the teacher must process and reason about complex student observations and performance metrics. A more expressive teacher architecture (e.g., with convolutional or attention-based modules) is often required to capture relevant task and state information, which in turn increases computational cost and may slow down both training and adaptation. The teacher’s signal for what constitutes a good task for the student may become weaker or noisier in high-dimensional settings, requiring additional regularization or sophisticated feedback mechanisms to maintain stable and meaningful curriculum generation. Conventional probability-based methods or teachers relying on handcrafted loss designs may no longer suffice in these scenarios. Therefore, it may become necessary to leverage meta-learning techniques or other advanced approaches to train a more general and capable teacher.

How does HAP handle tasks with non-linear difficulty progression or combinatorial dependencies? An ablation study on CRAFT’s dependency graph could clarify this.

HAP is designed to adapt to a learner’s evolving abilities by dynamically assigning tasks near the student’s competence frontier, rather than simply following a static or presupposed linear difficulty chain. This means that, in environments like CRAFT where tasks are structured with combinatorial dependencies, the teacher agent can give tutorials following some basic dependencies. We will add a graph in revision showing how HAP scheduling tasks for the student in CRAFT.

The entropy coefficient and warm-up duration are critical but empirically set. A sensitivity analysis would strengthen robustness claims.

We conducted comprehensive ablation studies to evaluate the impact of different components, running each configuration 10 times with different random seeds. The results on the Minigrid environment are shown below:

Easy tasks are consistently learnable across all configurations. Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.

While HAP excels in grid-world and Minecraft-like environments, its performance on Crafter’s open-world tasks (general score 0.723 vs. human 0.85) suggests challenges in scaling to real-world complexity. The paper does not analyze computational costs or memory demands for larger tasks.

Thank you for raising the important point. The overall computational cost of training in our framework is primarily determined by the student policy’s update frequency, batch size, and episode length, as well as the size of the neural networks. Importantly, for each teacher update, the student must complete $N_{evalbatch}$ evaluation batches, which further adds to the computational cost. The total cost per interval of teacher update can thus be summarized as:

where $N_{batch}$ is the training batch size, $N_{steps}$ is the episode length, $N_{evalbatch}$ is the number of evaluation batches per teacher update, $N_{evalsteps}$ is the number of steps per evaluation batch, $P_{student}$ is the number of parameters in the student policy, and $P_{teacher}$ is the teacher’s parameter count. Taking the minigrid experiment as an example, the teacher policy, implemented as a lightweight MLP and updated every 100 episodes, introduces minimal overhead compared to student training and rollout collection. For memory usage, key contributors include the student and teacher network parameters, as well as storage for trajectory data and the teacher’s history buffer:

where d is the storage size per step (observation + action + reward per task, etc.).

HAP assumes tasks can be ordered by difficulty (e.g., Minigrid’s "simple" to "extremely hard"). Its efficacy in environments with non-hierarchical or interdependent tasks (e.g., multi-modal RL) remains untested.

No, HAP has no assumption about task dependency or difficulty (line 131-136). The difficulty shown in tables was for illustrative purposes. To demonstrate that, we further test our model and compare the result with other CL baselines in a continuous motion_control task parametric-continuous-stump-tracks-v0. We kindly refer you to our response to reviewer EgTN.

The lack of theoretical guarantee of the proposed methods.

We have to acknowledge that deriving comprehensive theoretical guarantees for adaptive adversarial curricula in complex domains remains challenging. We try to admit a foundational analysis based on properties of stochastic approximation and policy gradient methods. Recall our setting:

$\min_{\phi} \max_{\theta}\ J(\theta, \phi) = \mathbb{E}\left[ R(\tau; T) \right]$

Provided that:

• The teacher and student policies are updated with sufficiently small learning rates (ensuring stability under stochastic approximation),
• $J(\theta, \phi)$ is differentiable and gradients are unbiased estimators,
• The parameter space is compact or regularized,

then the update scheme converges to a local minimax point or egalitarian equilibrium of $J$ where

At this point, neither the teacher nor the student can locally improve their objective by small, unilateral changes, reflecting a stable adversarial curriculum. The student learns maximally under the hardest (yet solvable) curriculum the teacher can assemble, given both agents' functional capacities, and The teacher cannot further reduce the student's success without stepping outside the feasible task distribution.

However, this guarantee pertains to local stationary points, not necessarily global minimax solutions (which is standard in deep adversarial learning). The overall quality and pedagogical value of the equilibrium depend on the capacity of the networks and the granularity of the task space.

Thank you for your insightful review. We sincerely welcome your feedback.

I noticed that your input has not yet been recorded in the ongoing discussion. As the Review-authors' discussion deadline is approaching, I would like to kindly remind you to review the author’s response and contribute your comments to the discussion in a timely manner. Mandatory Acknowledgement without any comments is not suggested this year.

Dear Reviewer EgTN,
Thank you for your thoughtful feedback. We try to answer your questions as follows:

Characterizing the teacher-student interaction as a zero-sum game seems like an odd choice to me, .... The common intuition is that the teacher is supposed to pick tasks with a goldilock difficulty ... There wasn’t sufficient justification for why HAP frames the interaction as zero sum, ... why the training would not collapse to the teacher picking an adversarial curriculum.

We respectfully disagree with your intuition. Adopting an goldilock standard does not affect the zero-sum / adversarial nature of our setup. Letting $\bar{r} _ {\text{stu}} = \frac{1}{n} \sum _ {i=1}^N r _ {\text{stu}}^{i}$, the teacher's reward can be simplified as:

Yet, the student reward can also be modified such that $r _ {\text{stu}} = 0 (\text{if } \bar{r} _ {\text{stu}} = 0 \text{ or } \bar{r} _ {\text{stu}} \leq 1 - \epsilon)$, resulting in the zero-sum setup as well. The motivation is that if the model always gets the solution right, it is not worth ``learning'' anymore.

We believe the core disagreement stems from different philosophical perspectives. Your critique appears to adopt a cooperative learning framework, where the teacher helps the student learn by finding the optimal "Goldilocks zone" of difficulty. However, from our perspective, we intentionally design HAP from an adversarial angle with a dynamic equilibrium where the teacher continuously raises the bar as the student improves, rather than seeking a static "optimal difficulty". The teacher's objective is not only providing data for the student to learn, but also to generate increasingly challenging problems that push the boundaries of what the student can solve, such that the generated problems themselves are hard and valuable, rather than assisting the students to learn only. The student, in turn, aims to master these progressively difficult tasks. This creates a bootstrapping dynamic where the teacher explores a larger space of challenging problems and the student continuously adapts to solve harder tasks. The ultimate goal is not cooperative learning assistance, but rather the emergence of a problem proposer capable of generating highly challenging tasks, and a solver that can evaluate and overcome these difficult problems.

We acknowledge that this pure adversarial setup introduces training difficulties. Ideally, the zero-sum formulation creates a dynamic equilibrium where the teacher finds tasks that maximally challenge the student's current capabilities. But for teachers like simple probability teacher, there is indeed no such guarantee. We do encounter cases when training collapses due to pathological teacher task selection, so we further introduce entropy regularization and cold-start policies to help avoid these cases. See the table for the ablation results for these modules. These training difficulties are natural in this adversarial setup. Like in self-play of AlphaZero, it's also possible that one agent leverages a weakness spot of its opponent and somehow stuck learning. However, there is exploration and exploitation and MCTS search tree to help making progress even such a weakness spot was found. But these design relate to the techical methods to mitigate training difficulties. Philosophically, they should be adversarial.

(For the table: Entropy regularization proves crucial for hard task performance. Without it, we observe significant performance degradation on harder tasks. Cold start primarily affects training efficiency rather than final performance, influencing convergence speed during the initial training phase. Lower bounds are essential for training stability. Without them, models tend to get stuck on hard tasks and suffer from catastrophic forgetting of simpler tasks.)

The related work section missed out on some of the most relevant frameworks where the teacher and student engage in zero-sum or asymmetric games.

Thank you for bringing these important works to our attention. All of these studies will be incorporated into our revised Related Work section.

All three are grid-based environments with similar visual representations and discrete action spaces. There is no evaluation on continuous control tasks like robotics, physics simulations. All are visual grid worlds - no language-only or multimodal tasks.

Regarding the scope and diversity of our experiments, we further evaluated HAP on the continuous control task parametric-continuous-stump-tracks-v0, based on the ACL benchmark TeachMyAgent [1]. TeachMyAgent leverages procedurally generated environments to assess the performance of teacher algorithms. To accommodate HAP’s implementation, we reduced the continuous teacher action space to 10 sampled tasks. Due to time constraints, our experiments were limited to the "fish" embodiment and employed "ppo" as the student algorithm. The results are as follows:

HAP achieves competitive performance and beats ACL methods like GoalGAN and Self-Paced in continuous control tasks with robotics and physics simulations (Setter-Solver has extra pre-knowledge for the solver so the result is significantly better). Tasks with continuous space actually have no difference in the algrithim structure compared with discrete space.

[1] Romac, Clément, et al. "Teachmyagent: a benchmark for automatic curriculum learning in deep rl." ICML.

For "no language-only or multimodal tasks," we are not entirely certain if this refers to tasks where only language is used (without images) or to more complex settings involving multiple modalities (e.g., visual+language). For completeness, we note that HAP has also been evaluated on both CV and NLP tasks under the supervised learning setting (see Section 6.1). We will further clarify this point in the revision.

Insufficient human evaluation. Although I appreciate it that the authors did human experiments to validate the effectiveness of HAP at creating curriculum for humans, the sample size is very limited and the choice of task is debatable. It’s unclear what aspect of human learning is captured by teaching Prolific workers Minigrid, ...

Thank you for your thoughtful feedback. Our human experiment is an initial step toward evaluating HAP's applicability for real-world curriculum design. Our goal in this study was to provide a controlled, proof-of-concept demonstration that HAP-generated curricula exhibit pedagogical properties beneficial to human learners—such as revisiting foundational challenges during plateaus and adapting task difficulty to learner progress. Regarding sample size, we recognize the limitations of our current study. We note, however, that sample sizes of 20–30 participants are common in early-stage psychological and educational research, especially when studies are time-consuming or require careful supervision. The results were statistically significant with p < 0.0001 (compared with w/o tutorial), providing meaningful evidence for our claims. Nonetheless, we acknowledge that larger and more diverse cohorts are important for drawing generalizable conclusions.

What exactly is the significance of heterogeneity in heterogeneous adversarial play? And how does it differ from "asymmetric"?

“Heterogeneity,” especially as commonly defined in multi-agent and federated learning literature (e.g., [1]), emphasizes agents with fundamentally different roles, capabilities, and action/observation spaces. In our framework, the teacher and student are not merely engaged in the same task from different perspectives; rather, they operate in genuinely distinct domains—one generates problems and the other solves them. In contrast, “asymmetry,” as in asymmetric self-play [2], often involves agents with related (sometimes swapped) roles but still share comparable skill sets, experiences, or objectives—for instance, a single agent alternately acting as problem setter and solver in similar tasks. Our design adopts a genuinely adversarial perspective rather than cooperation in self-play: the teacher’s objective is not only explicitly to help the student learn but to continually propose more challenging tasks, pushing the boundaries of the student’s capabilities. Meanwhile, the student’s objective is to learn to solve tasks of ever-increasing difficulty. By using the term “heterogeneous,” we foreground the structural and functional differences between our teacher and student, which are much more pronounced than in standard asymmetric setups.

[1] Zhong, Yifan, et al. "Heterogeneous-agent reinforcement learning." JMLR.
[2] Sukhbaatar, S., et al. (2018). Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. ICLR.

Thank you for your insightful review. We sincerely welcome your feedback.