EduGym: An Environment Suite for Reinforcement Learning Education

To answer your questions:

• Yes we definitely open-source the code, and invite everyone to contribute. The repository is already on Github, although we kept it private for now because of the double-blind review.

• We will add novel methods in the future, and have been discussing an “EduGym 2”. Topics to include could be multi-agent, multi-objective, goal-conditioned, etc., as we discuss in the last paragraph of the Discussion. Your suggestions are also very valid!

We hope this answers your questions. We appreciate your summary under strenghts, which indeed captures our exact intentions. We are curious to hear whether the additional feedback affects your judgement.

You indicate our work is outside the scope of ICRL, which we disagree with. In recent years, many papers have appeared at main machine learning conferences that present either an environment suite, benchmark and/or educational effort. Examples include [1,2,3,4,5]. Our work actually falls in the same category. We agree these papers don’t present a novel algorithm, but we disagree they don’t contribute to the research community (they usually get cited a lot). The research community also needs educational efforts for new students entering the field, as does it need benchmarks and small test environments that isolate a particular challenge (which did not exist yet as a collection). Since this type of work has appeared at ICLR, ICML and NeurIPS over recent years [1,2,3,4,5], we do not think it is fair to judge our paper as outside the scope of ICLR.

You also ask about the difference with OpenAI SpinningUp and the code of Sutton and Barto. Spinning Up is very different: it is educational (and very useful), but it implements complex specific deep RL algorithms, such as PPO and DDPG. This is very useful for more advanced students, but it does not isolate individual conceptual aspects of RL, such as partial observability, on/off-policy, sparse exploration, etc. We write about this in the Related Work section, in the paragraph called “Algorithmic frameworks”.

The code of Sutton and Bart comes much closer, and the CliffWalking example to illustrate the difference between on/off-policy was a major inspiration. However, there are two main differences: 1) Sutton and Barto mostly focus on credit assignment, and do not include specific conceptual examples for most of the challenges we did include (state dimensionality, partial observability, language, continuous actions) etc., and 2) most importantly, the code of Sutton and Barto does not come with interactive notebooks, which prevents the mixing of text and code, and makes it way less accessible to students.

Finally, you aks about best practices on deep learning. We do include a notebook about deep learning, which is the notebook on state dimensionality. Also, deep learning appears in several other notebooks, such as partial observability and continuous state/actions. However, we do indeed not discuss all practical aspects of tuning deep learning, but this is also clearly beyond the scope of our work. We intend to teach reinforcement learning fundamentals: exploration, credit assignment, partial observability, etc. Our main focus lies on small conceptual examples, while for complex deep RL many codebases actually do exist (such as Spinning Up).

We are curious to hear your view on the above topics.

1. Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., ... & Van Hasselt, H. (2019, September). Behaviour Suite for Reinforcement Learning. In International Conference on Learning Representations.
2. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., ... & Stoica, I. (2018, July). RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning (pp. 3053-3062). PMLR.
3. Pleines, M., Pallasch, M., Zimmer, F., & Preuss, M. (2022, September). Memory Gym: Partially Observable Challenges to Memory-Based Agents. In The Eleventh International Conference on Learning Representations.
4. Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., ... & Ravi, P. (2021). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 15032-15043.
5. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (pp. 1094-1100). PMLR.

You first of all ask about the learning theory below EduGym. This is mostly “Learning by Doing”, about which there is much literature [1,2,3]. The main idea is that we need to connect theory (equations, text) to practical interactive examples (code, notebooks) in a single document. In addition, we attempt to break the problem down in underlying conceptual topics. We have not included this in the paper since it seemed too far off for a machine learning conference, but of course see your point as well, and could include it.

You also ask whether EduGym will be 1) extended and 2) kept up-to-date. Regarding the former, we definitely will extend Edugym, as we indicate in the last paragraph of the Discussion, where we for example indicate we think of a future EduGym extension with multi-agent, multi-objective, goal-conditioned RL, etc. Regarding the latter, we will also maintain EduGym, but additions will probably mostly focus on novel notebooks. The reason is that EduGym teaches the RL basics, such as the distinction between on- and off-policy updates. These conceptual topics are so fundamental to RL, that their basic explanation will probably stay quite stable. Of course, with novel research completely novel topics may develop, which we will then try to incorporate into EduGym.

As a weakness you mention the lack of value iteration and policy gradients. They are actually included: value iteration appears in the notebook on model-based RL, while the policy gradient appears in the notebook on continuous state/actions.

You finally mention that the innovation is limited, which we disagree with. It’s true that EduGym does not contribute a novel algorithm, but we believe the RL community as a whole has immensely profited in recent years from environment suites, benchmarks and educational efforts. These papers often get quite some traction (citation-wise), which indicates they have their merit for the community. We think EduGym also falls in this category of papers, and although there is no novel algorithm, we do not think this implies the innovation is limited.

You otherwise seem quite positive about the idea of EduGym, and are curious to hear your thoughts on our comments.

1. Schank, R. C., Berman, T. R., & Macpherson, K. A. (2013). Learning by doing. In Instructional-design theories and models (pp. 161-181). Routledge.
2. Thompson, P. (2010). Learning by doing. Handbook of the Economics of Innovation, 1, 429-476.
3. Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. Psychological review, 86(2), 124.

We appreciate your positive feedback, but are surprised you then reject it based on a mismatch of track. We do not think this is totally fair. Many similar papers to ours have appeared in the main track of important machine learning conferences in recent years, such as [1,2,3,4,5]. These papers all either present an environment suite, a benchmark set and/or an educational component, which our paper actually combines. (Note that our main focus is not benchmarking, but actually the environment suite and educational component.)

We think such papers have a clear merit to the research community, even though they do not directly contribute a novel algorithm. The referenced algorithms are for example well cited, which shows the research community also values such efforts. Our paper specifically targets students entering the field, which should have a clear benefit, and it can also be used as an environment suite which disentangles challenges of RL. As such, we believe it could be very beneficial to the RL research community, and fits the ICLR main track as several other papers from recent years.

We are eager to discuss this topic with you, since you otherwise seem very positive about our effort!

1. Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., ... & Van Hasselt, H. (2019, September). Behaviour Suite for Reinforcement Learning. In International Conference on Learning Representations.
2. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., ... & Stoica, I. (2018, July). RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning (pp. 3053-3062). PMLR.
3. Pleines, M., Pallasch, M., Zimmer, F., & Preuss, M. (2022, September). Memory Gym: Partially Observable Challenges to Memory-Based Agents. In The Eleventh International Conference on Learning Representations.
4. Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., ... & Ravi, P. (2021). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 15032-15043.
5. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (pp. 1094-1100). PMLR.

You indicate our paper has an educational rather than technical contribution, which does not suit ICLR. We believe this criticism is not fair, when compared to the RL papers that have been published at major ML conferences in recent years. Our work contains an environment suite, of which several have appeared in big ML conferences in recent years [2,3], and can also be leveraged as a benchmark suite, which has also previously appeared at ICLR [1]. The same applies to structured codebases at the intersection of research and education [4]. All these papers got cited quite a lot, and have their impact on the community by making novel research feasible. Our additional educational focus should only add to this, since entering the research field can be challenging. In short, we believe our paper has merit to the research community, even though it does not present a new algorithm.

Your second critique is that our work is not properly empirically validated. Of course, empirical validation is important, but we actually did include this evaluation. We asked a range of bachelor, master and PhD students with experience in RL whether they valued EduGym, which most of them strongly agreed with. These students already know a bit of RL, and can therefore judge whether our resources help them compared to what they used so far. Note that nearly all other teaching resources (books, public codebases) available are just published, and never evaluated with A/B testing. We therefore believe this criticism is also not completely fair, since we 1) included a useful practical evaluation and 2) already go beyond the practical evaluation of other educational resources.

We are curious to hear your opinion on the above topics.

1. Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., ... & Van Hasselt, H. (2019, September). Behaviour Suite for Reinforcement Learning. In International Conference on Learning Representations.
2. Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., ... & Ravi, P. (2021). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 15032-15043.
3. Pleines, M., Pallasch, M., Zimmer, F., & Preuss, M. (2022, September). Memory Gym: Partially Observable Challenges to Memory-Based Agents. In The Eleventh International Conference on Learning Representations.
4. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., ... & Stoica, I. (2018, July). RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning (pp. 3053-3062). PMLR.