CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents

We thank reviewer 8arZ for the valuable time and constructive feedback.

Q1: As a work in the area of datasets and benchmarks, I think a lot of the details are in the code, and I don't have good confirmation of those details just by virtue of the text in the paper. This environment is based on an open-source game http://freeciv.org/. I don't know how different is the open-source game and the environment introduced by the authors. It would be excellent if authors could give reviewers an early release of code to review.

A1: That is an excellent question. We have released our code on GitHub at an anonymous account for review purposes: https://github.com/civrealm. and you can also access our documentation at: https://civrealm.github.io/civrealm/. This GitHub account hosts four distinct repositories https://github.com/civrealm?tab=repositories:

• civrealm: this repository contains the core code of the CivRealm environment, encompassing both tensor-based and language-based APIs, along with comprehensive documentation.
• civrealm-sav: this repository contains the code responsible for creating the mini-games, which also enables users to customize their own games and generate new maps.
• civrealm-tensor-baseline: this repository provides the code for our baseline tensor-based RL agents.
• civrealm-llm-baseline: this repository hosts the code for our LLM agents, including BaseLang and Mastaba.

In the following, we would like to explain the relationship between Civrealm and the Freeciv game. To put it briefly, it is analogous to the relationship between pysc2 (link) [ref1] and the Starcraft game.

Specifically, Freeciv is a game that supports graphical interfaces for human players. However, this interface type is not designed for RL or language agents, i.e., accessing states and executing actions through tensor or language. To expose Freeciv as a research environment, we have developed CivRealm with an OpenAI Gym-style API to facilitate interaction between AI agents and Freeciv. In other words, Civrealm implements a proxy API that enables programmatic control of Freeciv. Through this API, AI agents can start/join a game, obtain game states, perform actions, and capture screenshots to generate game replays.

To implement the API, we conducted an in-depth exploration of the internal workings of Freeciv. This involved:
(1) delving into the process of parsing game states and issuing action commands through communication packets,
(2) building concurrent communication flows and thoroughly testing the environment to ensure smooth training and testing processes,
(3) configuring server settings for diverse game modes,
(4) developing infrastructure for parallel training, and
(5) incorporating various other supporting features for research with the game, including quantitative metrics.

In summary, we have constructed a comprehensive infrastructure that encompasses various controller, state, and action classes designed for multifaceted game control, thus facilitating interaction with the Freeciv server.

Additionally, we have added support for tensor-based and LLM-based agents and have created a new game mode: "mini-game (doc link)". For more detailed documentation of our implementation, please refer to: https://civrealm.github.io/civrealm/.

We sincerely appreciate your thoughtful review, and we hope we have addressed your concerns. Please do not hesitate to reach out if you require any further information.

[ref1] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge.", 2019.

Thanks for your explanation! Hope you can understand that it is important for reviewer to have codes to review for these kinds of paper. I am happy to raise your score. :)

Another question: In the table one, why Dota is not regarded as a stochastic game? I think the monster in the wild and the items drop from them are random. Why it is not multi-goal game? One may argue that the goal is single -- win the game, but I guess CivRealm is also to "win the game".

We thank reviewer 8arZ for the valuable time and constructive feedback.

Q1: Some details are not clear enough. As this is a multi-agent environment, what are the baseline RL and LLM agents play against? I didn't seem to find such details in the description.

A1: Thank you very much for bringing this to our attention. The opponent players that our baseline methods play against are built-in AIs implemented by the original Freeciv team. We have added this detail in Appendix A.1 of the revision.

Q2: The proposed baselines are not working in the (full game) environment. Maybe it is better to also set different difficulty levels (e.g. by different task horizons) for the full game so that later research can more easily be evaluated on the benchmark.

A2: Thank you for the suggestion! We will be adding APIs to allow users to configure the length of a full game. This will enable later research projects to evaluate themselves on full games with varying time horizons.

The current full game is also highly customizable in several ways:

• The number of opponents and the map size can be adjusted.
• The level of resources on the map can be customized.
• It offers different levels of difficulty in terms of the built-in AI opponents.

Q3: Is Figure 2 the real situation that the LLM agents discover or the expected situation? If it is what the LLM agents discover, why does Tit still behave poorly?

A3: Thank you for bringing this confusing part to our attention. This is an expected situation simulated by our built-in rule-based AI, which we utilize to illustrate the complexity of the Freeciv game. Figure 8 displays what the LLM agents have discovered.

Q4: The text in Figure 4 is too small to view.

A4: Thank you for the reminder. We have updated Figure 4 in the revised paper.

Q5: Why the success rate of the RL agent in some mini-games (e.g., SettlerBuildCity) is higher for the hard setting than for the easier settings?

A5: We have also observed this interesting phenomenon. This counterintuitive situation arises from a shortcut strategy discovered by the RL agent. Specifically, as outlined in Appendix A.2 of the paper and detailed at https://civrealm.github.io/civrealm/advanced_materials/minigame.html, the winning condition for the hard mode of the SettlerBuildCity task is determined by averaging the potential city positions on the minigame map. In the harder mode, the land terrain is considerably less favorable, with an abundance of deserts, tundra, and swamps, resulting in lower tile production values for food, production, and trade. Consequently, the winning score is lower due to the averaging across these poorer terrains, resulting in a ranking of minimal winning scores as follows: easy > normal > hard.

Originally, the goal of the hard mode was to locate and settle in a resource-rich land area (e.g., forests or rivers) amidst these challenging terrains. However, sea and ocean tiles always offer more resources compared to the widely distributed poor land terrains. We have observed that tensor-based agents tend to exploit this unique scenario and learn a shortcut strategy of exclusively building coastal cities. This approach easily results in above-average production values for inland terrain, meeting the winning criteria for settlerBuildCity[hard].

Q6: Can the authors comment on the reason for instability and the sudden drop in performance in Figure 9?

A6: The sudden drop is attributed to the player encountering opponents or a pirate invasion, which could trigger when the player has reached a certain development stage. Our method, Mastaba, has outpaced the development of BaseLang; therefore, Mastaba encountered pirate invasions and was defeated at an earlier stage. It is worth noting that in the game, defending against pirate invasions is challenging, even for human players, as the pirates' technology, attack strength, and other factors are typically set to a high difficulty level.

Q7: It would be great if the authors could provide videos of the experiments or visualization ways in the future code repo.

A7: We included two videos in the supplementary materials: "Mastaba" and "BaseLang," each showcasing a complete game. In terms of visualization, our codebase offers users two functionalities: 1) the ability to visually observe the agent playing the game and 2) the option to automatically record screenshots of the game. We also customized the FCIV-NET version of the game for users, which provides a 3D visualization (shown in Fig 1).

We appreciate again your thoughtful review and we hope we addressed your concerns. Please let us know if you'd like any further information.

We thank reviewer 3D5Q for the valuable time and constructive feedback.

Q1: How robust is the server-client architecture? Did you do any experiments to test it?

A1: That is an excellent question! During the development of CivRealm, we adhered to the test-driven development practice. We created unit tests, integration tests, and end-to-end tests to verify the reliability of various game operations and the training environment. Additionally, we conducted testing on multiple platforms, including Linux, Mac OS, and Windows, to ensure smooth gameplay for our agents to the best of our efforts. Most of the test code is available in the civrealm/tests folder within the released civrealm repository for your reference.

Q2: Is there support for designing custom games? I see Lua script being mentioned to specify the reward structure, new goals, etc. Is there also a support for new maps?

A2: Yes, we have uploaded a civrealm-sav repository in our released code. This repository enables users to customize new games and generate new maps. For a detailed tutorial, please refer to the 'Create new Mini-Game' content at: https://civrealm.github.io/civrealm/advanced_materials/minigame.html.

We sincerely appreciate your thoughtful review, and we hope we have addressed your concerns. Please let us know if you would like any further information.

Thank you for your prompt response. We are genuinely grateful for your thoughtful feedback, which further improves the clarity of our paper.

Best,

Authors

Thank you for your insightful feedback on our work, and we appreciate your positive comments. We agree that discussing prior work on multi-LLM agents for the study of cooperative settings [1] would be a valuable addition to our paper. Your feedback is much appreciated.