From Machine to Human Learning: Towards Warm-Starting Teacher Algorithms with Reinforcement Learning Agents

We appreciate the reviewer's engagement and would like to clarify our work's scope and contributions, which differ substantially from the cited literature.

Our Focus: RL-to-Human Transfer, Not RL-to-RL

Our work addresses a distinct problem: using RL agents to bootstrap teacher algorithms for human learners, not RL agent training. The reviewer's summary ("RL approach to train RL agents") mischaracterizes our contribution. We propose solving the cold-start problem in human education by using RL-generated synthetic data to warm-start teacher algorithms, then deploying these teachers with human subjects.

Why Existing RL Curriculum Literature Doesn't Apply

The cited works (Silva & Costa 2019, Narvekar et al. 2020, Matiisen et al. 2019, Sukhbaatar et al. 2018) focus on RL-to-RL transfer with fundamentally different constraints:

• [Scale]: These methods require millions of timesteps across small number of tasks; our humans interact with only 10-17 episodes across 3000 timesteps
• [Feedback]: RL curricula rely on value functions and dense rewards unavailable in human settings
• [Applicability]: Self-play mechanisms (Sukhbaatar 2018) and evolutionary approaches (ACCEL) are impractical for human learners due to time and feasibility constraints.

Our Baselines Are Appropriate

We compare against standard human learning baselines (random, handcrafted, no-training controls) consistent with human subjects literature (Bassen et al. 2020, Doroudi et al. 2019: tables 3,4,5,6,7). In particular, owing to the interleaved practice effect (Taylor K, Rohrer D, 2010, Ebbinghaus 1948), random curricula are by no-means a trivial curricula for humans (Zhou et al, 2017, Shen et al 2018a). Indeed, most of the early prior works failed to show significant effects over a random curriculum (Doroudi et al 2019). Our contribution lies in demonstrating that RL-initialized curricula can match expert-designed sequences while requiring no domain expertise.

Novel Contribution

To our knowledge, we are the first to bridge RL synthetic data generation with human curriculum design, validated through controlled human subjects studies (n=471 across two domains).

We welcome incorporating the suggested references to better contextualize our work within the broader curriculum learning landscape.

---

Zhou, G., Wang, J., Lynch, C.F., Chi, M. (2017). Towards closing the loop: Bridging machine-induced pedagogical policies to learning theories. In Proceedings of the 10th international conference on educational data mining (pp. 112–119). International educational data mining society.

Shen, S., Ausin, M.S., Mostafavi, B., Chi, M. (2018a). Improving learning & reducing time: a constrained action-based reinforcement learning approach. In Proceedings of the 2018 conference on user modeling adaptation and personalization: ACM

Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice. Applied Cognitive Psychology, 24(6), 837–848. https://doi.org/10.1002/acp.1598

Ebbinghaus, H. (1948). Concerning memory, 1885. In W. Dennis (Ed.), Readings in the history of psychology (pp. 304–313). Appleton-Century-Crofts. https://doi.org/10.1037/11304-034

We thank the reviewer for his early reply, and would like to clear up misconceptions. In this work, we primarily focused on using RL agents to generate relevant data for our teacher algorithm. Our teacher algorithms are NOT trained by RL, but merely leverages data from RL agents to train on. For example, PERM-H collected (θ,r) tuples from observing an RL student training to achieve an Item-Response Theory model, which is trained by a variational objective.

In our work, we adopted the UPOMDP (Section 3) to draw links to RL and Human training to frame our discussion and drive similarities, but we do not use RL-like mechanisms to assume how humans train. We do not use reward functions during our human training to make any assumptions about human performance.

As we have asserted in the paper and in our rebuttal, all the prior work that focuses on curriculum learning for RL does not apply well to humans, especially on the inordinate amount of data required to train these algorithms in. While bearing in mind that we have 1 hour with our human subjects, aligning these algorithms to human-time scale trivializes these prior work:

• ACCEL (Parker-Holder et al, 2022) uses value estimation functions, which are not available from humans, and evolutionary methods to discover new environments. Constraining the approach to 10 environments (which we use in our Jumper experiment) is akin to random initialization with an almost static curriculum.

• TSCL (Matiisen et al 2017; proposed by reviewer) uses sampling methods that are based on observing performance on each task K times. In our case, K=1 for our tasks, and the algorithms will not be able to converge, leading to a curriculum that is akin to random sampling, which we have also included. Notably, TSCL algorithms (Figure 5) do not beat the manual curriculum on the 2D addition task, which we have already included in our paper.

• Sukhbaatar et al (2018; proposed by reviewer) proposes self-play "show-and-reverse" as a curriculum which is a concept that is infeasible for humans. The best approximate implementation of Sukhbaatar et al's work for humans would be to get human students choose their own task, but one would easily argue that they are not equivalent given the sophistication behind the reward function expected of Alice and Bob.

While we could theoretically adapt these algorithms for human learning contexts, the examples above demonstrate that doing so would require such substantial modifications that the resulting approaches would no longer meaningfully represent the original methods. Any comparative evaluation would therefore be inherently unfair, as we would be comparing our approach against fundamentally altered versions of the baseline methods rather than their intended implementations.

We therefore firmly reject the reviewer's assertion that these works "could be easily added to the experimental evaluation". Such a characterization fails to acknowledge the significant constraints and assumptions involved in translating RL-based curriculum learning approaches to human subject studies.

We ask the reviewer to recognize that our work specifically addresses human learning within the practical constraints of limited time, data, and cost. These constraints necessitated the development of new methods rather than the adaptation of existing approaches that were designed for entirely different contexts. On our part, we have provided relevant prior works in the domain of human learning, our area of focus for this paper, and provided discussion in this particular context.

We thank the reviewer for their in-depth review and feedback. We are glad that the reviewer found our work an important challenge and the potential for impact. We apologise for the formatting errors and will ensure proper formatting in the next iteration. Here are our discussion on the long-term prospects of our work:

1. On the validity of convergence-based difficulty metrics (cθ) and occupancy measures for task similarity, citing algorithm-dependence issues and asking about limitations for automated curriculum design. Where will they break down, and what does this imply about the potential of automated curriculum design for generating data for automated teaching?

We acknowledge these metric limitations while emphasizing their practical effectiveness for curriculum design.

[Convergence-based difficulty]: While algorithm-dependent, we mitigate biases through averaged statistics across multiple PPO runs and careful task verification. Our designed tasks allow validation against known complexity features (Figure 16), showing strong correlation between our metrics and intended task difficulty.

[Occupancy measures for similarity]: These capture behaviorally relevant patterns essential for knowledge transfer. Tasks requiring similar state-action sequences (e.g., treating asthma patients with specific medications) share transferable knowledge. Our human validation (r=.490 correlation) demonstrates meaningful alignment, sufficient for effective curriculum generation.

[Breakdown conditions]: Our choice of metrics would likely break down as we progress towards more abstract domains, where curriculum demands a more nuanced understanding of the learning progress. For example, using a single dimension for task difficulty will likely break down when the task involves a complex combination of different abilities to execute (e.g. language abilities and logical reasoning in high school curriculum), preventing a more targeted approach for curriculum design.

[Implications for automated curriculum design]: Rather than fundamental limitations, these represent current design challenges. The key insight is that moderate fidelity suffices for warm-starting teacher algorithms and perfect metrics aren't required. This aligns with our stand that while RL agents provide useful cold-start data, the ultimate goal should be a human-centric design with more human data. Future improvements should focus on environment design that better captures human-centric factors (cognitive load, attention) while maintaining the practical advantages of automated data generation. Our Emergency Response environment is an example of how sophisticated medical procedural knowledge can be designed into an environment that still produces learning gains.

The broader potential remains strong: automated curriculum design provides scalable foundations that human data can subsequently refine.

2. Motivation for PERM-H

Our changes to PERM is motivated by prior work (Tsividis et al, 2017) that humans learn at a rate higher than artificial agents. Through pilot studies, we arrived at this formulation due to our observations that adjusting the rate of difficulty increase alleviates boredom and frustrations in our human subjects.

3. Incorporating Individual Differences and Mindful RL-Human Gap Management

Current foundation supports individual adaptation with clear extension pathways

[Adaptive baseline]: Our algorithms are inherently adaptive, adjusting to individual performance patterns without requiring prior demographic modeling. This addresses cold-start scenarios where student characteristics are initially unknown, allowing the system to discover individual differences through interaction.

[Future individualization]: Extensions could model multi-dimensional learner profiles (learning pace, preferred modalities, attention spans, prerequisite knowledge) by expanding our similarity metrics to capture these factors. RL agents could be trained with varied "learning personalities" to generate diverse curricula templates that accommodate different learning styles.

Two perspectives to managing RL-human learning differences

[Bridge the gaps through pedagogically-informed design]: The first perspective suggests that one should manage these gaps in order to improve the outcomes for humans. Taking a top-down perspective from learning sciences, we can design environments and teacher models that better capture human-centric learning factors. For example, incorporating cognitive load theory through task complexity constraints, or implementing spacing effects by restricting how frequently similar tasks appear. In this work, we have taken extra care to designing the Emergency Response game, where we managed to translate medical knowledge to an environment which both RL and Humans can demonstrate learning.

[Leverage differences as educational advantages]: The second perspective embraces these differences as potentially beneficial. Successes in complex domains like Dota2 (OpenAI Five) and Go (AlphaGo's move 37) demonstrate that RL learning processes can unlock creative approaches that humans might overlook. Research could explore when these differences benefit human learning and under what conditions such phenomena occur.

To be clear, we adopted the first perspective in this work, where we took pedagogical theories to inform our teacher algorithm and investigated the gaps in the context of our selected metrics. We show here that while the metrics are not perfectly aligned, the educational outcomes are prevalent in our students.

Practical implementation strategy

The key insight is starting with universal learning principles (difficulty progression, knowledge continuity) that work across individuals, then progressively personalizing as more data becomes available. This provides immediate practical value while building foundations for sophisticated individualization, ensuring positive outcomes through validated pedagogical principles rather than relying solely on RL-derived patterns.

4. How does pedagogy fit into this research agenda more broadly?

Our work advances computational pedagogy by solving a fundamental bootstrapping problem: effective teacher algorithms require extensive human learning data, yet collecting such data is prohibitively expensive.

[Pedagogical contribution]: We demonstrate that RL agents can serve as pedagogical proxies, capturing sufficient learning dynamics to warm-start sophisticated teacher algorithms. This opens previously inaccessible pedagogical approaches in domains that are new, such as task difficulty measurements in the environments we have designed.

[Broader research agenda]: This enables two complementary research directions. First, enriching RL-generated data with pedagogical nuances. Ho et al.'s "showing vs doing" distinction exemplifies this; RL agents naturally exhibit "doing" behaviors (exploitation without considering educational value) that differ from demonstrations with educational intent. Teacher algorithms aware of this distinction can leverage both trajectory types appropriately.

Second, scaling sophisticated pedagogical models that were previously data-prohibitive. Our framework accommodates diverse pedagogical theories (Zone of Proximal Development, Spiral Curriculum) and can support more complex approaches incorporating spacing effect (Ebbinghaus 1948) and scaffolding via expertise-reversal effect (Kalyuga et, and individual differences as they develop.

[Key insight]: Rather than replacing pedagogical expertise, our approach democratizes access to data-hungry pedagogical models. This creates opportunities for educational researchers to test and deploy sophisticated theories without massive upfront data collection, accelerating the translation of pedagogical research into practical educational technologies.

The ultimate vision is pedagogically-grounded AI that starts with universal learning principles and progressively personalizes through human interaction.

---

Ebbinghaus, H. (1948). Concerning memory, 1885. In W. Dennis (Ed.), Readings in the history of psychology (pp. 304–313). Appleton-Century-Crofts. https://doi.org/10.1037/11304-034

Kalyuga, S., Ayres, P., Chandler, P., Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016). https://doi.org/10.1038/nature16961

OpenAI (2019). Dota 2 with Large Scale Deep Reinforcement Learning. 10.48550/arXiv.1912.06680.

We thank the reviewer for the positive review and for recognizing the importance of our work. We are encouraged that the reviewer saw our evaluation as solid and well-detailed.

1. How could the RL approach tackle the diversity in learners, in terms of backgrounds, learning capabilities, disabilities, etc? How can these differences be modeled?

We appreciate this important question about accommodating diverse learners, which is indeed a core challenge in personalized education.

In the current work, our proposed algorithms adapt to individual differences through performance-based inference

[Ability-focused design]: Both PERM-H and SimMAC continuously adjust to individual students by inferring their current ability levels from performance feedback and selecting appropriate next challenges. This responsiveness naturally accommodates different learning rates, prior knowledge, and capabilities without requiring explicit demographic modeling.

Our human studies demonstrate this flexibility across participants with varying gaming experience and medical backgrounds (Sections 6.1, A.4). Importantly, our approach improved learning outcomes regardless of participants' initial prior knowledge levels, suggesting robustness to individual differences in our tested domains.

[Current framework handles basic diversity, with clear pathways for complex cases]

Without a doubt, there is room for greater levels of personalization (for more nuanced considerations like cognitive differences, learning disabilities) for which we propose extensions to our current work below.

[Enhanced teacher models]: Teacher models could be further extended to more sophisticated models. Prior work in Doroudi et al's (2019) survey showed that models that were backed by pedagogy-motivated models performed better than data-driven approaches. As an example, Item Response Theory, which underlies PERM-H, offers extensions for modeling multiple ability dimensions simultaneously (e.g., language proficiency and logical reasoning).

[Population-specific fine-tuning]: Following transfer learning principles from machine learning, our RL-pretrained teacher models can be fine-tuned on specific populations to accommodate specialized needs while retaining the general learning patterns.

Our two-stage approach naturally supports this progression: RL data provides general learning patterns, then human data from specific populations can refine the models for targeted accommodation. This maintains the cold-start benefits while enabling specialized adaptation as needed.

2. What would be the main difficulty to employ these approaches in more abstract domains?

We anticipate two challenges in deploying our framework in more abstract domains:

[Environment design complexity]: Abstract domains like philosophy, mathematics, or critical thinking require natural language interactions and sophisticated reasoning situations to train. We would have to utilize LLMs to augment our trainers to generate appropriate situations at the right time.

[Learning process modeling]: Our approach relies on observing RL agents' progression patterns to capture learning dynamics. In abstract domains, this may not be sufficient, as learning can also involve certain "intrinsic" understanding beyond "extrinsic" knowledge/skills.

[LLMs as learning proxies?]: For abstract domains like mathematics or logic, large language models may serve as more appropriate proxies for human learners than traditional RL agents. RL, unlike pre-trained LLMs, grows while exploring the environment, presenting a good simulation of learning. Recent work in cognitive modeling (Binz et al 2024) through language models (foundation models for human decision making) shows promise.

[Transferable principles from our current work]

Our paper's fundamental insight is that artificial agents can capture useful learning patterns for curriculum design and that moderate alignment suffices for effective teaching. This lays the foundation for research into artificial agents generating data for teacher algorithms for humans. The technical implementation would require domain-specific innovations, but our two-stage framework provides a general methodology for bootstrapping personalized learning systems across diverse educational contexts.

3. Is there room for a third algorithm that deals with the limitations of PERM_H and SimMAC? How would it be shaped?

Yes, we think that there is a possibility of building more sophisticated models that consider the science of learning principles. Then, the challenge would be to see if there are methods or clever design of the exploration stage (be it on the environment design front, or the MDP formulation front) that can be used to address the data needs of these models. As research into leveraging RL as data-generators increases, we envision more models that are nuanced in their understanding of the RL-human gap, and bridge them through design.

---

[1] Binz, Marcel, et al. "Centaur: a foundation model of human cognition." arXiv preprint arXiv:2410.20268 (2024).

We thank the reviewer for their time in reviewing, and welcome the questions.

1. Why can we model students as a standard RL learning agent? What if there is a significant difference between the student learning process and ML Learning process? What if the human student learning process differs from RL Learning? How to justify that these two processes are similar?

We appreciate this fundamental question about the validity of using RL agents as proxies for human learners.

Our claim is that we do not need perfect equivalence between RL agents and human learners. We need enough similarity to generate effective curricula.

This is because effective curriculum design requires capturing learning progression patterns and task difficulty relationships, not perfective cognitive modeling of human learners. We explicitly acknowledge differences between RL and human learning while showing that moderate alignment is sufficient for warm-start data generation.

[Empirical evidence demonstrates sufficient alignment across the following key dimensions]

Task similarity patterns: Section 6.3 shows moderate positive correlation (r=.490, p<.001) between RL and human inter-task similarity measures across 287 task pairs in the Emergency Response domain.

Difficulty ranking consistency: Both RL agents and humans consistently agreed on task difficulty rankings in the Jumper environment, with RL agents reaching comparable difficulty levels to human-designed curricula (Figure 3).

Learning progression patterns: RL agents exhibit similar progression dynamics: starting from low performance and gradually improving through structured practice, mirroring fundamental human learning characteristics.

[Human studies validate the practical effectiveness despite imperfect modeling]

Our RL-initialized curricula (PERM-H, SimMAC) matched expert-designed curricula and significantly outperformed random baselines in human studies (Figures 1-2). This demonstrates that the captured learning patterns translate to effective human learning outcomes.

[Our framework is designed to handle modeling limitations]

The two-stage approach explicitly addresses this concern: RL data provides initial bootstrapping, then human interaction data supplements and eventually replaces RL data for better alignment. This design acknowledges that RL is an approximation while leveraging its benefits for cold-start mitigation.

The strong empirical outcomes validate that moderate alignment is sufficient for addressing the cold-start problem in personalized education systems.

---

2. Why is having exploration stage/synthetic data helpful? Could authors provide some justification of that?

The exploration stage solves a critical practical and ethical problem

[Cost barrier:] Traditional teacher algorithms require extensive human learning data before becoming effective. Bassen et al. (2020) demonstrated this challenge, requiring 900 man-hours for their RL teacher to converge (lines 80-87), creating prohibitive deployment costs.

[Ethical concerns:] Early learners in cold-start systems receive poor instruction while the algorithm learns, raising fairness concerns about subjecting students to ineffective teaching during the learning phase.

RL agents generate learning patterns that transfer effectively to humans

Scalable pattern generation: RL agents can generate extensive training trajectories across diverse difficulty levels and task combinations without human cost, capturing fundamental learning dynamics like difficulty progression and knowledge transfer relationships.

Validated transfer: Our empirical results demonstrate this transfer works in practice. Both PERM-H and SimMAC, trained solely on RL-generated data, significantly outperformed random curricula and matched expert-designed sequences in human studies (Figures 1-2).

Concrete evidence shows synthetic data captures relevant learning dynamics

Difficulty alignment: RL agents reached difficulty levels comparable to humans in the Jumper environment (Figure 3), suggesting they capture meaningful challenge progression patterns.

Task relationship modeling: In Emergency Response, RL-derived task similarity patterns showed meaningful correlation with human patterns (r=.490, p<.001), indicating successful capture of knowledge transfer relationships.

[The exploration stage provides competent initialization without human cost]

The exploration stage essentially "pre-trains" our teacher algorithms on synthetic learning trajectories, eliminating the cold-start problem where early learners receive poor instruction. This provides a competent starting point that can be refined with human data as it becomes available, combining scalability with adaptability.