OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments

Dear Reviewer,

we thank you for your valuable feedback. We really appreciate all the time you have already spent to help us improve our paper. The points raised will help us to improve. We appreciate your indication that OCAtari greatly impacts the community that develops the object-centric RL algorithms. Let us now shortly address the opportunities.

Opportunity for improvements:

1. Object-centric states of OC-Atari might not be the most optimal. OC-Atari can allow to separately train the object detection from the policy search, as we explain in the introduction, and does not prevent to train end-to-end agents. For our tested games, the representation extracted by REM allow matching deep agents’ performances. We cannot exclude that other object representations might be better suited for some tasks, we think that our open source code allows RL practitioners to convert or augment our object centric representations.
2. Why are OCAtari agents learning slower than deep ones?. We asked ourselves the same question. This is very likely due to the fact that we have done any hyperparameter tuning. We used the same hyperparameters as the original (deep) PPO agents, but the backpropagated gradients are much lower. Changing them on Boxing already leads to agents learning faster. Our goal is here to show that learning is feasible, we leave optimization of the learning speed for future work. We added this comment in the experiment section.

Raised Questions:
(1) For some games, the object detection method seems to be far from perfect.

We are constantly improving performance and adding new games. However, there are elements in some games that are captured by REM, but not by our VEM method, or vice versa, but do not participate in the gameplay (e.g. detecting particle objects). For example, we added to the appendix DemonAttack, in which the enemies spawn with a particle effect. This noise is not captured by REM, but is by VEM.

We have noticed that some games in our table were still missing in the appendix, like the mentioned DemonAttack, ChopperCommand or IceHockey. We have now added details of all the games that are supported by REM in our appendix. We also added a small section about common mistakes which occur in games like DemonAttack, also in the appendix.

(2) If the environment gives you a list of objects, how does it handle "object permanence" over different frames?

In contrast to a purely visual approach where object permanence must be treated separately, REM detects blinking and overlapped objects. This can be used to train methods that are concerned with object permanence, such as [1]. We improved our discussion on this in the new manuscript version.

[1] Liu, I. J., Ren, Z., Yeh, R. A., & Schwing, A. G. (2021, September). Semantic tracklets: An object-centric representation for visual multi-agent reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 5603-5610). IEEE.

Dear Reviewer,

we thank you for the valuable feedback. We really appreciate all the time you have already spent to help us improve our paper. The points raised will help us to improve. We are happy that you think that OCAtari greatly impacts the community that develops the object-centric RL algorithms. Let us now shortly address the opportunities.

Opportunities for Improvement:

1. Why selecting Atari games ? Thank you for this. We have answered this concern in the general comment. Shortly: \

• ALE is the most used benchmark for existing RL algorithm, \
• ALE’s games diversity covers many learning aspects and challenges \
• The lack of human reasoning capabilities of ALE trained agents has yet to be addressed. \

2. Incorporating Aitchison et al. (2023) in OCAtari. Thank you very much for this amazing resource. We will concentrate our next efforts to cover the 3 remaining games that are recommended (and will of course cite this work). We are nearly done with BattleZone already. We also believe that many other games (such as Skiing for difficult credit assignment, and e.g. Montezuma’s Revenge or Pitfall for sparse reward) provide challenges that are worth exploring as well. We added this comment on page 8. \

3. Red circles in Figure 4. Good point. These circles represent are supposed to show different cases where the REM and VEM differ, as the objects are present in the environment but blinking (in MsPacman), exploded (SpaceInvaders) or partially occluded (FishingDerby). We added this to the caption of the figure (and removed the one from Skiing). \

4. Why are OCAtari agents learning slower than deep ones? We asked ourselves the same question. This is very likely due to the fact that we have done any hyperparameter tuning. We used the same hyperparameters as the original (deep) PPO agents, but the backpropagated gradients are much lower. Changing them on Boxing already leads to agents learning faster. Our goal is here to show that learning is feasible, we leave optimization of the learning speed for future work. We added this comment in the manuscript.

Dear Reviewer,

we thank you for the valuable feedback. The points raised have help us to greatly improve the manuscript. Thank you for your positive comments on the use of our library to train object-centric RL agents. Let us address the different points that you have raised.

Opportunities for Improvement:

1. The lack of object-centric representations has not held back the adoption of ALE. ALE has indeed made available a wide range of domains featuring a common interface, with resources requirements, making ALE the go-to framework for testing RL methods. The observed trend in the emergence of object-centric and neurosymbolic methods is notable, as reflected in Fig.1. OCAtari provides benefits for a wide range of domains. We added a paragraph in the introduction to strengthen this claim. \

2. Authors might not continue the development to cover more games. Since the paper submission, our coverage has expanded to include 7 additional games (compared to our original manuscript), with enhanced detection performance for two of them. In our provided codebase, the "scripts" folder contains various tools for reverse engineering games. Notably, our coverage surpasses that of AtariARI, now spanning more than twice as many games. As you can see, we plan to provide as many games as possible and do not require guidance by AtariARI to do so. \

3. Games are already modifiable within ALE. Yes they are. However, to modify the behavior of the games, one first needs to understand how the information is processed and where it is stored. Our Open Source OCAtari framework makes this information process transparent. For example, to understand how to remove particular types of enemies, one first needs to know the RAM positions encoding the enemy type, but also what values correspond to these enemies, explicitly provided by OCAtari. We made this point clearer in the manuscript in section 2.3. \

4. REM might not save time. As you said, for very few games, such as MsPacman, the background is not encoded in our object-centric description. This is why, for such games, hybrid approaches might be necessary, as described in our limitations section. But even hybrid approaches could benefit from turning off expensive object detection modules, and thus save many resources.
   The background changes after some levels, but that still constitutes a fixed number of backgrounds, these are not procedurally generated (thus equivalent to a problem of having one). Extracting a maze representation to use e.g. the A* algorithm, together with an object-centric RL trained algorithm might be an interesting approach that also would benefit from OCAtaris object extractions. We expanded our future work section to now include this discussion.

Question:

1. Why not creating directly object-centric environments ? In other works, we had created more modern environments for the development of different neurosymbolic approaches, but have been many times asked to compare to deep approaches on ALE games. Beyond having been adopted by most deep RL practitioners, ALE provides a wide range of tasks featuring a common interface, with many RL specific caveats available (e.g. sparse reward, difficult credit assignment, …etc). These environments have been developed to challenge humans, and not AI, and thus don’t include any experimenter's bias. They are also cheap to run compared to more modern games. We added a paragraph to strengthen our motivation about using Atari in the introduction.

Dear Reviewer,

we really want to thank you for all the effort you have put in the reviews, and the valuable feedback you have provided us. We agree that OCAtari can serve a nice tool to test the abstraction ability of RL methods in their ability to produce human-like object-centric reasoning. Your raised points have greatly helped us to improve our manuscript.

Let us address the opportunities (which we enumerated to make the assignment easier).

Opportunities for Improvement:

1. What benefit for classic deep methods? The OCAtari framework, serves as an important asset for advancing object-centric reinforcement learning (RL) and facilitating comparisons with established deep RL methodologies. OCAtari extends its utility to conventional deep RL approaches, as it can e.g. be employed together with Language Model (LLM) that guide agents training using on game manuals’ instructions (and object grounding) [1, 2]. Moreover, OCAtari's reveals games’ inner workings, allowing the development of modified game versions for the evaluation of e.g. continual RL methods, as done in [3] on Doom. \

2. Baselines of representation learning methods. We provide comparison in detection capabilities of the two SOTA methods evaluated on Atari: SPACE [4] and MOC [5] in table 3. These methods obtain an average few shot classifier that goes up to 22% for SPACE and 84% for MOC. OCAtari directly provides the object classes, that can be used to train the classifiers. In our F-Score computations, we only consider that an object is correctly detected if it has the correct class. We added this comment in our manuscript in section 3.1

3. Not much contribution over AtariARI. AtariARI only indicates the location of raw information within the RAM, whereas our framework provides reconstructed object-centric information. This involves intricate computations, such as deriving the objects’ positions from anchors and offsets, looking up potential presence indicators (e.g. for objects that have been destroyed), ensuring that our models genuinely acquire object-centric state descriptions. Moreover, our coverage spans 35 games, a substantial increase compared to AtariARI's 16, and continues to expand. We rephrased section 3.3 to better emphasize this.

Questions:

1. What is the learning contribution? Our key contribution is the introduction of the OCAtari environments. We showcased the feasibility of training PPO agents that use a 2 layer MLP on the object-centric state descriptions across different games. Our efficient ram extraction method (REM) stands out for getting object-centric information from RAM values, surpassing the energy efficiency of neural networks or hard-coded vision detection algorithms like VEM. The development of efficient or reliable learning algorithms is not the focus of this paper. We are providing a resource-efficient alternative for developing, evaluating and training object-centric RL reasoning agents, that can be based on e.g. logic, as done in [6, 7, 8]. \

2. Is object-centricity necessary to solve a task? While object-centricity (OC) is not inherently indispensable for solving RL games, numerous studies in RL research highlight its importance in elucidating agent reasoning, prevent misalignment, and potentially correct them. Notably, studies such as [1, 2] leverage Language Model-based Learning (LLMs) to realign agents, relying on natural language descriptions of environment goals for this purpose with great success. This underscores why we need to enhance our understanding of agent behavior and ensuring their proper alignment with the intended objectives. According to these studies, OC is a necessary step to achieve it. We added this in our update of the introduction.

3. How is VEM working? We have improved the the VEM: the Vision Extraction Method paragraph in our manuscript, page 3: “At each stage, objects are extracted using color-based filtering and an object position priority. For example, Pong consists of 3 primary objects and 2 HUD objects, each assigned constant RGB values (cf. Figure 2). Contrary to the paddles, the HUD scores always appear above the white threshold, which allow us to distinguish these two same-colored objects. This technique allows to reliably extract the depicted objects.”

4. How does OCAtari overcome evaluation on non-deterministic approaches? All our OCAtari environments are compatible with the Deterministic, NoFrameskip and v0 to v5 variations of gymnasium. This helps to reproduce experiments and make it easier to provide benchmarks and datasets. We added this information into our experimental details section in the appendix. Thank you for raising this up, we add this comment to our manuscript. But you are right, the non-determinism is not the important aspect of OCAtari, we modified the paper to just express that RL benchmarks are important.

[1] Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and brain sciences, 40, e253.

[2] Farebrother, Jesse, Marlos C. Machado, and Michael Bowling. "Generalization and regularization in dqn." arXiv preprint arXiv:1810.00123 (2018).

[3] Hausknecht, M., & Stone, P. (2015, September). Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series.

[4] Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018, April). Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).

[5] Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., ... & Kavukcuoglu, K. (2018, July). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning (pp. 1407-1416). PMLR.

[6] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba:

Mastering Atari with Discrete World Models. ICLR 2021

[7] Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018). Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61, 523-562.

[8] https://gymnasium.farama.org/environments/atari/complete_list/

[9] Wu, Y., Fan, Y., Liang, P. P., Azaria, A., Li, Y., & Mitchell, T. M. (2023). Read and reap the rewards: Learning to play atari with the help of instruction manuals. Advances in Neural Information Processing Systems, 36.