A2Perf: Real-World Autonomous Agents Benchmark

When presenting a project such as A2Perf, one must also show where it would be used within the life cycle of a (research or production) project. For instance, some libraries offer tools to train a model and make it clear that the goal is to achieve the best performance given an architecture and an algorithm, others focus on reproducing results in the literature... Here the goal is very clear but the way the library is to be used (end-to-end vs post-training to validate claims) is a bit unclear. I believe writing a comprehensive RL library with many SOTA algorithms isn't really the point here, so I'm wondering how one could choose an algorithm, architecture, dataset or problem that is not in the toolkit of the library and use the metrics provided to assess that specific scenario. How easy would that be done compared to training two different algorithms with a common RL library (SB3, torchrl, tianshou)? It would be good to give the reader some idea of how easy it would be to use A2Perf for a problem that is not already accounted for by the lib.

1. The authors aim to create a unified benchmark focused on real-world application and deployability, with the goal of it being widely adopted by the research community. However, the exact contribution is unclear—they primarily use pre-existing benchmarks and metrics previously proposed/used in other works. Is the novelty here simply the combination of these elements in a single accessible suite? While this can prove valuable, it still feels limited by the small set of available environments.
2. Although the term 'autonomous agent' is used frequently here, it remains undefined. How much human intervention is permitted? Are there separate training and testing phases involving human input? Is a continual learning setup, as described in [1], considered?
3. Generalization as a key metric—This metric seems vaguely defined. Why should it be reasonable to expect a policy trained on task 1 to perform well on task 2 without additional training or fine-tuning? Does generalization here only refer to task performance, or does it also encompass task representation and knowledge of the environment?
4. Although the paper addresses real-world applications, its evaluation is limited to simulation benchmarks. For instance, how would evaluation differ with a real robot dog? What about factors like reset time between episodes or the need for human intervention?
5. The paper also overlooks the robustness of hyper-parameters. Given that deep RL algorithms are notoriously sensitive to hyperparameter choices, shouldn't this be considered part of the training costs?
6. Justification for metrics specified in Table 3 is insufficient. The authors should consider additional metrics from [3] and [4] when evaluating application performance.
7. The paper lacks citations for large-scale benchmarks such as RoboHive [5], which supports a wide range of robot embodiments.

References

1. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024). Loss of plasticity in deep continual learning. Nature, 632(8026), 768-774.
2. Ceron, J. S. O., Araújo, J. G. M., Courville, A., & Castro, P. S. (2024). On the consistency of hyper-parameter selection in value-based deep reinforcement learning. Reinforcement Learning Journal, 3, 1037–1059.
3. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.
4. Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical design in reinforcement learning. arXiv preprint arXiv:2304.01315.
5. Kumar, V., Shah, R., Zhou, G., Moens, V., Caggiano, V., Gupta, A., & Rajeswaran, A. (2024). Robohive: A unified framework for robot learning. Advances in Neural Information Processing Systems, 36.

1. Motivation
   a) While I do find the general motivation of this work timely, it is not clear to me why these specific environments were chosen and how they relate.

2. Claims and Novelty
   a) It seems that the novelty of this manuscript is limited as most metrics are already known and the environments that are being considered already exist. It seems that the main contribution is to assign existing metrics to existing benchmarks. The exception is the novel data metric that is being introduced. However, as I outline later I’m not convinced of the usefulness of this metric.
   b) The motivation for the work seems to be largely based on the claim that evaluation in other benchmarks is missing. One of the initial claims is that “There is currently no benchmarking suite that defines the environments, datasets, and metrics which can be used to develop reference implementations and seed leaderboards with baselines” (L18) This seems to be a very strong statement given the rise of a large number of benchmarks recently.
   c) “Most [other benchmarks] only evaluate an agent’s raw performance on the same task on which it was trained, without considering numerous other metrics that matter in real-world production training and deployment scenarios.” (L46) There are various environments and datasets with proposed metrics that measure benchmark related quantities. Some examples:

• [1b, 2b] Continual learning benchmarks measure metrics such as forgetting and collapse.
• [3b] Suggests to evaluate various dataset sizes which seems similar to the data metric proposed in this manuscript.
• [4b, 13b] Measure constraint violation for practical safety features.
  d) I am going to interpret Table 1 in the paper as a claim. I do not agree that Safety Gym and CoinRun do not measure reliability. Safety gym explicitly measures violations of constraints which can be interpreted as reliable. The number of tasks an agent can solve out of distribution in CoinRun should have to count as a measure of how reliable a system is to perturbation. Similarly it is unclear to me why Safety Gym tasks are considered realistic but DM Control is not.

3. Related work
   a) The treatment of related work could be improved.

• Given the recent rise of new benchmarks after the introduction of the datasets and benchmarks track at NeurIPS. 8/9 of the benchmarks listed in Table 1 are from 2020 or older. Here is a non-exhaustive list of recent potentially relevant benchmarks [1b, 2b, 3b, 4b, 7b, 8b, 9b, 10b, 11b, 12b, 13b] and datasets [14b, 15b, 16b, 17b]
• The reliability metrics only get cited in section 3 even though the intro states "our reliability metrics" (L92) which might lead to confusion about attribution.

4. Metrics
   a) Most of the suggested metrics already exist, see [8, 12, 50] in the paper. Systems performance metrics can quite easily be tracked using frameworks such was Weights&Biases nowadays too.
   b) The data cost metric seems inconsistent since any method can use a fixed dataset that already exist. The major upside of using datasets is that the cost of data collection is amortized over time. Once a dataset exists, it can be re-used and it is unclear to me why existing data should be tied to specific methods. Further, it seems odd that all RL agents show 0 data cost as they do need to collect data. In general, RL sample efficiency is significantly lower and it seems incorrect to claim that offline methods have lower data cost. I suppose the argument is supposed to be that this data collection requires human effort to gather expert demonstrations but then the metric is measured using energy consumed to train RL policies which do not require human effort.
   c) It seems a little contrived to consider system metrics algorithm dependent rather than hyperparameter dependent. For instance, I could simply change the batch size of my DDQN algorithm an increase the VRAM usage. In section 5.2, if I simply decrease the number of parallel processes, the peak RAM usage would be lower but other metrics would not be affected. I think in order for this to be useful, clearer guidelines are needed on how exactly to measure these metrics.

Overall, I agree with the final assessment that we need more holistic metrics to measure progress in L532. However, it's not entirely clear what unique advantages this benchmark offers beyond existing methods. I think the scientific contributions are limited and the proposed metrics either already exist, are possibly of low utility or at least require more detailed elaboration on how to use them. The benchmark environments also already exist. The experiments provide an interesting study but do not provide sufficient novel insights for me to argue acceptance to ICLR at this time.

[1b] Continual World: A Robotic Benchmark For Continual Reinforcement Learning. Maciej Wolczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, Piotr Miłoś. NeurIPS 2021.
[2b] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone. NeurIPS 2023.
[3b] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill bench-mark with large-scale demonstrations. NeurIPS D&B 2024.
[4b] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
[5b] Model-Based Reinforcement Learning for Atari. Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, Henryk Michalewski. ICLR 2020.
[6b] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. ICML 2022.
[7b] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. ICLR 2021. [8b] Jorge A. Mendez, Marcel Hussing, Meghna Gummadi, and Eric Eaton. CompoSuite: A compositional reinforcement learning benchmark. CoLLAs 2022.
[9b] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. ICLR 2024.
[10b] DACBench: A Benchmark Library for Dynamic Algorithm Configuration. Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer. ICJAI 2021.
[11b] Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a diverse suite of scalable reinforcement learning environments in jax, 2024.
[12b] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. NeuRIPS 2020.
[13b] Zhaocong Yuan, Adam W. Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P. Schoellig. Safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation 2022.
[14b] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research 2024.
[15b] Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. NeurIPS 2022 D&B.
[16b] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Lin Liu, Yang Jun Feng, Lin Lai, Hongyang Qin, Minwen Deng, Juchao Zhuo, Deheng Ye, Qiang Fu, Yang Guang, Yang Wei, Lanxiao Huang, and Xiangyang Ji. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. NeurIPS 2023 D&B.
[17b] Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning. Marcel Hussing, Jorge A. Mendez, Anisha Singrodia, Cassandra Kent, Eric Eaton. RLC 2024.

• Limited baselines were evaluated (one offline, BC, and two online baselines per environment).
• The authors did include standard deviations, but it is unclear which results are statistically significant. The paper would benefit from stating this explicitly.
• The paper would benefit from summaries across the environments about which methods perform the best (summary figure, bolding, etc.)
• Given this paper is proposing a benchmark without any algorithmic advances, the limited evaluation and issues with rigor (see questions) make it difficult to evaluate the contribution