From Child's Play to AI: Insights into Automated Causal Curriculum Learning

Regarding additional baselines

We thank the reviewer for their suggestion on baselines to run. Please see the general comments for the baselines we are currently running and will report back on.

We agree that the switch threshold is an important hyperparameter. We are also interested in seeing the results and will report back what we learn.

To clarify the manual curriculum, there are 16 parallel tasks so that the agent can collect data more efficiently by interacting in parallel across 16 tasks. When we increment the difficulty by 1/16, this means that, of the 16 parallel tasks, one of the tasks has the difficulty incremented (and the others do not change). So, the progression would be 1) 16 1-lanes, 2) 15 1-lanes and 1 2-lane, 3) 14 1-lanes and 2 2-lanes, etc. We did originally experiment with switching all 16 tasks by one difficulty, but we found that this posed a sizable distribution shift to the agent that caused training divergence at much easier difficulties than if the difficulty was instead incremented by 1/16. However, we did notice as in Section 5.3 that training divergence does eventually occur. In Section 5.4, the level progress used in the auxiliary reward is calculated at the termination of an episode and then added to the extrinsic reward. For cases where we present average episode level progress (e.g., Figure 6b), this is calculated by taking the mean level progress over all finished episodes between policy updates.

We thank the reviewer for the suggestion for trying an off-policy algorithm that uses a replay buffer. In the baselines that we just ran, we found that stochastically sampling previously solved levels indeed mitigates the catastrophic forgetting issue, however, progress made within the curriculum / towards solving the goal level is less than what we found in Section 5.4.

We thank you for your detailed feedback and suggestions. Your suggestions on the writing and experiments helped us to improve our paper, we hope to address your questions and concerns here.

Regarding the terminology of ‘level progress’.

We very much appreciate your suggestion to better define level progress and other key terms for our paper. We have revised the terminology from ‘level progress’ to ‘level competence’, to reflect that this is not a derivative, but rather the current level of success the agent/ human achieves on the level. We have also written up more precisely defined definitions (see General Comments above).

Level Competence: An indicator of success in an episode of game play on a specific level, reaching 100% when the specific level is successfully solved in that episode. (Note: This is the term formerly referred to as 'level progress.')

Global Competence: A measure of success in an episode of game play with respect to the target level, reaching 100% if the target level is successfully solved in that episode.

Level Advancement: Difference between current level competence and initial level competence within a particular level in the curriculum; it is a measure of how much competence increases or decreases in the same level

Global Advancement: Difference between current global competence and initial global competence across levels in the curriculum; it is a measure of how much competence increases or decreases with respect to achieving the target level Auxiliary reward: A scaled level competence

Regarding the alternative hypotheses that could explain the data.

We very much appreciate the suggestion to make our hypothesis more explicit, and we appreciate the reviewer for diligently differentiating between the hypotheses. The reviewer is correct in their understanding of what our proposed hypothesis is: Children have a causal understanding of the world and maximize their causal learning for a given goal (i.e., solving the target level). We appreciate the reviewer’s suggestion “to include levels from games different from the target game level.” We are currently collecting data for a follow-up study that addresses this question. We have a version of this study where we add an irrelevant “non-causal” axis, e.g., changing the color shades, and easy levels from other games. We will mention this follow-up study in the work. As an additional response to the reviewer’s question about examples of when children almost never select the irrelevant game, it can also be seen with our addition of the extra difficult level 4 (with the target being level 3) in the game design. Solving level 4 can be argued as irrelevant because it takes more unnecessary effort and does not help children achieve the extrinsic goal and win a sticker. We find that the majority of children never select the irrelevant, overly-difficult level 4 for building their curriculum to solve the target level.

We also ran additional exploratory analyses to address your third and fourth alternative hypotheses: On intermediate difficulty bias: because the easier levels are direct subsets of harder levels in the games Leaper and Climber, we can calculate competence on easier levels as a percentage of competence in the target level to get global competence for each episode. We can then also determine which level children should objectively choose next by comparing their global competence with the percentage of the target level that each easier level represents. Here, we found that half of the time children were making objective choices, but another half of the time children chose harder levels than what they should objectively choose, suggesting that they have an intermediate to high difficulty bias that is likely also driven by the desire to causally solve the difficult target level On bias towards learning progress: we ran further analyses and found that children do not have a bias towards learning progress or advancement: our child participants neither used global advancement nor level advancement (as defined in our definitions in the General Comments) to guide their choice of next level to play, suggesting that they are not driven by how much progress they have made, but more by how well they are currently performing to move towards the goal (as demonstrated by our statistics on local competence guiding next level change) We will revise our paper to clearly state our hypothesis and include statistics of these additional findings.

Regarding the broken citation.

We fixed the broken Vygotsky citation, thank you for letting us know.

We thank the reviewer for their time to follow up with additional comments for interpreting the child data.

We are interested in both kinds of causal understanding (rules and physics of the game + practicing on easier levels causes improvement on target levels) and have ongoing experiments probing both ideas.

We also informally looked at children’s open responses and found that they often referred to both the causal structure of the rules of the game and the causal structure of how solving an easier level could then enable them to solve a harder level of the same game.

In our games, the causally relevant axis to help you achieve ultimate success at the goal level is difficulty. A child or an agent learns the game as a causal curriculum by choosing how to move along the difficulty axis. We consider changing the color shades to be a non-causal axis because there is no causal relationship between color shade and winning the game. Color does not have any significant causal power or insight in the game for achieving the target level.

In our experiment setup, each child is subject to only a maximum of 10 rounds of automated curriculum learning; in every round they have the choice to select any of the game levels. Many children even solved the target level and terminated in less than 10 rounds (the average total number of rounds played is 7 across participants, with the minimum being 3). Thus, we do not have enough statistical power at the moment to adopt existing learning progress approaches (e.g., comparison between last 3 rounds and previous 3 rounds within a level) in our measure of progress or advancement. To circumvent this, we calculated level competence as just the competence in the active episode, and level advancement as the difference between the most recent episode and the initial episode of the same level.

While we only have 22 children and are underpowered to do model fitting and model predictions, we have built linear mixed effects models to compare various hypotheses, and found that our data can be best explained by the idea that “children have a causal understanding of the world and use their level competence to guide their causal curriculum learning for a given goal (i.e., solving the target level)” (this model has the lowest AIC and BIC). After further examination of the data, intermediate difficulty bias and level advancement (not computed as the classic learning progress due to limited data per individual participant) so far did not explain much of child behavior - a notable portion of children seemed to remain at the same level or move to higher levels even when they had very low level competence or negative level advancement because they were motivated to achieve the extrinsic goal of solving the target level, but we have yet to collect more data to confirm this idea.

Once we have collected enough data, we will definitely try adopting your recommendations on fitting different models with some portion of child data and using them to predict other children!

We thank the reviewer for their time to follow up with additional comments for the RL experiments.

Regarding the auxiliary reward, at the end of each episode, the agent receives a reward that is the sum of 1) an extrinsic reward (which, in this case, only occurs if the level is solved) and 2) the current advancement of the agent within the level. The agent only receives these rewards at the end of an episode, not at each step. We agree that the consequence of providing such an auxiliary reward is that it “densifies” the rewards.

Although it is not directly curriculum learning, we see that it impacts curriculum learning results. Having such a reward helps prevent forgetting which arises due to the distribution shift induced by switching levels. It may not be the only way to prevent forgetting, but having the auxiliary reward appears promising to both prevent forgetting and advance the curriculum. In the baseline experiment we recently conducted using a stochastic curriculum with sampling previously solved levels, we see that forgetting was also avoided, but the agent made less progress within the curriculum than with auxiliary rewards.

The relationship between the auxiliary reward in the RL experiments and the human experiments is that we observed that the children’s competence within a level improved without necessarily solving the level. The analogy for the RL agent is that if there was an additional reward besides the sparse extrinsic reward that could help with learning, the agent could potentially learn more effectively than with extrinsic rewards alone.

At the present time, our RL experiments were conducted using a hand-specified curriculum. In future work, we are interested in experimenting with different level selection policies. However, our results do show that, at least for this task with sparse rewards, that other algorithmic capabilities besides level selection (e.g., capabilities for preventing forgetting that arises when switching levels) may be needed for effective curriculum learning.

We very much appreciate the reviewer’s time and feedback.

Regarding the terminology surrounding “level progress”

We would like to thank the reviewer for their suggestion to better define level progress and other key terms for our paper. We have introduced a definition table, which contains clearer explanations. These definitions are presented below. Level Competence (previously referred to as ‘level progress’): an indicator of success in an episode of playing a level, reaching 100% when the specific level is successfully solved in that episode.

Regarding authorization to do human subject experiments

We thank the reviewer for checking on authorization to do human subject experiments. We have IRB approval to conduct human subject experiments and will include a sentence in our experimental design that the experiments have been approved by our university’s IRB.

Regarding inaccessibility and generalizability of level progress

We would appreciate it if the reviewer could clarify their concerns regarding the inaccessibility and generalizability of level progress. We agree that, in our current formulation, the use of level progress as an auxiliary reward does have the effect of providing a denser supervision signal. To the question of how we plan to access level progress, we believe the work of Bruce 2023 “Learning About Progress From Experts” (as mentioned by reviewer Rafo) could be extended to our domain.

Regarding the number of random seeds

As for the RL experiments, each of the experiments in Section 5.3 and 5.4 are run over six trials each, where each trial has a unique random seed. Figure 4 and Figure 6 are from one representative trial within the six trials. For Section 5.3, we consistently saw catastrophic forgetting across random seeds. The hyperparameters used were those from Jiang 2021 “Prioritized Level Replay”.

Thank you for your response. However, I am still confused about some important aspects.

1. It is clear that 100% of a level progress means the completion of the level. But how do you define the progress formally when it is sth like 8%, 67%, 93%? Does there exist a unified notation of such progress for all the environments? At least in the suite of Atari games, I personally find such a level progress hard to define.

2. How did you calculate the level progress in your current experiments? Is it manually assigned?

3. If the experiments were conducted in multiple random trials, I am expecting to see the confidence intervals (e.g., a shaded area around the curve) instead of a single curve. And for fig.4 and fig.6, I would suggest the authors present the full results instead of hand picking a "representative trial", which seems not to be a good practice. You may refer to Agarwal et al. [1] for a recommend way of reporting results in RL.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

We very much appreciate the reviewer’s time and feedback.

Regarding clarifications on the experimental results and environments.

We appreciate the reviewer's suggestion regarding clarifications on our experimental results and the experimental environments. Could we ask the reviewer to specify additional details or highlight specific areas of concern for us to address. We are enthusiastic about enhancing the clarity of our explanations and offering a more comprehensive understanding of our experimental setup.

Regarding additional baselines.

We thank the reviewer for their suggestion on baselines to run. To clarify the experiments in Sections 5.3 and 5.4, the agent advances to the next level of the curriculum when the average reward obtained exceeds 9. The auxiliary reward mentioned in Section 5.4 is inspired by human curriculum learning, but the advancement of level is still just done when the average reward obtained exceeds 9. We are interested in running a baseline where the curriculum also changes based on lambda.

Regarding the number of random seeds for our RL experiments.

For the experiments in Sections 5.3 and 5.4, six trials were used for each experiment, where each trial has a unique random seed. The results shown in Figure 4 and Figure 6 are for one representative trial within the six trials.

Regarding related work by Bruce 2023.

We also thank the reviewer for bringing Bruce 2023 to our attention, as this work seems highly relevant. Indeed, the concept of task progress is not novel to our work. However, we note a difference between our setting and Bruce 2023 is that Bruce 2023 requires expert trajectories to learn their progress model. In our case, we do not have access to expert trajectories, so we can envision that our approach could introduce novelty by relaxing this assumption and thereby extend their method.

We very much appreciate the reviewer’s time and feedback.

Regarding the number of children in the study.

We are collecting more data on children at the moment. There are developmental cognitive science papers that have published experiments with a sample size of about 20-30 each (e.g., see Experiment 1-3 in [3], see Experiment 1 in [4]). (Children are much harder to recruit than adults, but they are important for understanding the bases of curriculum learning.)

But more importantly, even though 22 children looks like a small sample size, each child does not only contribute to one data point. We are interested in examining children’s steps (or rounds of the game played) throughout their curriculum; within each step, we have details about which level they select, how well they perform on the specific step, etc. We currently have data for 159 steps that children made in their curricula and even when we factor in individual differences in our regression model, we are still able to find a statistically robust pattern between their level competence and their choice of next level.

[1] Kushnir, T., Gopnik, A., Chernyak, N., Seiver, E., & Wellman, H. M. (2015). Developing intuitions about free will between ages four and six. Cognition, 138, 79-101. [2] Sobel, D. M., & Corriveau, K. H. (2010). Children monitor individuals’ expertise for word learning. Child development, 81(2), 669-679.

Regarding additional baseline experiments.

We appreciate the reviewer for pointing this out. thank the reviewer for their suggestion on baselines to run. We have indeed conducted additional experiments, and we report the results here. First, we have conducted a baseline where environments are chosen randomly from 1-5 water lanes, without the curriculum. This is done for six trials/random seeds with a limit of 10M frames. We find that the agent makes very little learning progress over 10M frames. The return for the goal level (5 water lanes) is generally zero over the entire 10M frames. Moreover, the agent does not obtain much training return, either. We hypothesize that this is a difficult exploration problem for the agent without a curriculum, and although the agent may eventually make progress, the timescale would far exceed the 10M frame budget we used for our experiments.

Second, we conducted a baseline against Section 5.3 (curriculum without auxiliary reward) where, 30% of the time, previously solved levels are chosen based on where the agent currently is in the curriculum. For example, if the current curriculum progress is 3.5 water lanes, then levels would be chosen from [1, 2] water lanes 30% of the time. This is done for 6 trials/random seeds with a limit of 10M frames. We do find that this mitigates catastrophic forgetting, as training divergence did not happen in any of the trials. However, the agent makes less progress towards the curriculum than in Section 5.4 with the auxiliary reward. In Section 5.4, all six trials got to at least 4 water lanes, whereas in this baseline, only one trial got to 4 water lanes. The average number of water lanes across all trials was 2.958. Additionally, in this baseline, none of the trials were able to solve the goal level (unlike in Section 5.4, where 5 of the trials could solve the goal level). We find that, although stochastically selecting previously solved levels prevents catastrophic forgetting, it may not be an optimal approach for advancing the curriculum. We hypothesize that this occurs because 30% of the training budget is being spent to prevent agent forgetting, rather than learning how to solve the latest level in the curriculum. Therefore, it would be valuable to have an agent learn and utilize the auxiliary reward used in Section 5.4, as this both prevents catastrophic forgetting and advances the curriculum.

Third, we conducted a baseline that was similar to the second one, except instead of sampling previously solved levels, all possible levels (from 1 to 5 water lanes) were sampled. Six trials/random seeds were used with a limit of 10M frames. This baseline generally does poorly. In 4 trials, the training divergence still occurred. In only one trial was the training still active; this trial reached 3.0 water lanes. We hypothesize that this baseline performs worse than sampling previously solved levels, because sampling more difficult levels generally leads to more difficult levels that may not be solvable. This yields sparser returns and induces the same training divergence behavior seen in Section 5.3.

We hope this has addressed the reviewer’s concern, we would be happy to discuss/ conduct any additional experiments if the reviewer has additional experiments/ concerns in mind.

We appreciate the reviewer for their positive and detailed feedback. We attempt to address the questions raised by the reviewer and would be more than happy to address any additional questions that the reviewer may have.

Regarding questions on the user study of children. We agree that it is important to have a well controlled and understandable setup for children. In our experiment setup, we included an extra difficult level beyond the target level to validate that children were not just making random choices: we found that so far only 14% of children selected to play this extra difficult level, suggesting that most children understood what their goal was and did not attempt what was beyond that goal. Furthermore, we observe statistically significant evidence that children use their game level competence to guide their decision of which level to play next, again reflecting that they understood their task and were not making random responses. With regards to multiple choice questions, there are developmental studies studying children of similar age ranges that use multiple choice questions [1],[2]. We will include relevant citations in our paper. More importantly, we found that most children were able to respond correctly to the multiple choice question pertaining to the true causal rule of the game. [1] Kominsky, J. F., Gerstenberg, T., Pelz, M., Sheskin, M., Singmann, H., Schulz, L., & Keil, F. C. (2021). The trajectory of counterfactual simulation in development. Developmental Psychology, 57(2), 253. [2] Rafetseder, E., & Perner, J. (2018, April). Belief and counterfactuality: A teleological theory of belief attribution. Zeitschrift für Psychologie, 226 (2), 110-121. doi: 10.1027/ 2151-2604/a000327

Regarding additional RL experiments. We appreciate the reviewer for pointing this out. We are running experiments on other games in Procgen. We will try our best to complete the experiments within the time constraints, and we will report the results as soon as they are done."