Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

Thank you for your positive feedback on our work commenting that it is “well-written, easy to understand”, “propose a new multi-agent platform”, and “evaluations are comprehensive and insightful.”

Below we address your one comment raised in the review:

Are these three scenarios too similar? they seems to be one task but with different name. Can you test on more settings with a shared common resources?

In our study, we need to have both similarity and variation between tasks. The shared common underlying structure behind the three scenarios enable us to aggregate the analyses and show sustainability results across different instances. However, the variations enable us to ensure robustness such that our results are general to any particular set of prompts and situations. Therefore we must balance diversity with interpretability. The above three variations were inspired by prominent examples from the economics literature to illustrate the generalizability of the same underlying phenomena.

To explain existing diversity more directly we can think of two dimensions of variation in the simulations reported.

Dimensions 1) Our three scenarios differ in their framing and the number of quantities the agents must reason about:

• In the Fishing scenario, agents only need to consider one variable: the number of fish.
• In the Sheep Grazing scenario, agents need to consider two variables: the number of sheep and available grass.
• The River Pollution scenario also requires two variables: widget production and its impact on pollution levels.

While still maintaining the same underlying causal model of resource extraction and replication, these differences created differences in LLM performance as can be seen in Table 1. The sustainability/gain/efficiency scores differ across the three scenarios (although the overall trends remain systematic).

Dimension 2) Perturbations to the base simulation results in more complex dynamics that the LLM based agents must face. These reflect more substantive variation in the causal dynamics.

1. The “Newcomer” perturbation scenario (Section 3.3) checks whether LLM agents are robust against newcomers with a different behavior from the existing agents who have established a norm.
2. The “Absence of Communication” scenario (Section 3.5) tests how cooperation decays if agents cannot communicate in an open ended fashion with each other.
3. The “Universalization Reasoning” scenario (Section 3.4) probes whether making the long-term group outcome more salient improves cooperative behavior.

We will update Section 5 (Future Work) to more explicitly describe additional ways in which the scenarios could be varied to more deeply study LLM cooperative decision making and make clear why we chose the three scenarios studied in this work.

Please do not hesitate to let us know if you have further questions.

Thank you for your positive feedback on our work, especially your comments acknowledging that our GovSim “provides a novel platform,” “bridges the development of LLM-agents with classic social science theories,” has “a unique perspective,” “offers thorough analyses and a variety of experiments”, and has “well-motivated and theoretically grounded” “evaluation metrics.”

Below we provide some additional information to your comments:

While the GovSim environment is novel in its exploration of social science concepts, it is somewhat limited in its ability to accurately represent and visualize the complexity of actual agents' behavior. A platform where agents' actions can be grounded would further enhance the argument

In this first version the main grounding comes for the decision to extract resources from the environment that interacts in a causal way with the simulation environment. As such we are able to provide graphs of resource levels over time and statistics on agent actions and their distributional properties. We also provide in our uploaded code package (to be open-sourced upon acceptance) an interactive dashboard to visualize the agent interactions with the environment (see Figure 7, Appendix B). Our Appendix B provides a detailed technical description of our setup, and B.3 makes an effort to build the web interface to ground the agent actions.

The paper would be strengthened if the authors incorporated human participants in the study, allowing LLMs to coordinate with humans in real-time. This would provide valuable insights into how AI agents interact and cooperate with human decision-makers.

Thank you for the excellent suggestion for future work. Incorporating human participants will provide valuable insights into human-AI cooperation in these scenarios. Potential benefits include understanding how humans interpret and respond to LLM communication, identifying areas where LLMs excel or struggle compared to humans, and exploring the emergence of human-AI cooperative norms. We believe that there is enough new work to explore human-AI cooperation in a manuscript focused on those big questions. In the camera ready version of the manuscript we will add the following text: “Using the GovSim platform, a promising next step is to incorporate humans into the simulation. These human-AI interactions will challenge LLM based agents to cooperate with humans using open ended communication and we can see whether the norms that develop are either more or less effective than those created by LLMs alone.”

Could the authors provide insights into how the platform could be further scaled up to handle more complex scenarios and larger agent populations?

Great question! We will add the following to the future work section to engage directly with the question of scaling in Section 5. To scale up GovSim, we would like to explore:

• A larger agent population: Our current simulation can easily generalize to more agents and a diversity of types. Adding agents will increase the simulation runtime, as each agent needs to condition their own behavior and dialogue on the other agents' actions and dialogs. Perhaps fine-tuned smaller LLMs can act as efficient simulators in this context without a loss in performance.
• Coordinated adaptation: People can flexibly adapt to sudden changes in game dynamics. For example, when the resource suddenly shrinks (a temporary shock) or changes in the reproduction rate require agents to rapidly adjust their cooperative norms in a coordinated way. GovSim enables these kinds of experiments as the simulation environment is modular such that resource dynamics, agents, and other elements are easily changeable for different runs of the simulation.
• Challenging tradeoffs: We are also interested in understanding exceptions to norms. For instance, one agent may need to handle a one-off choice of serious personal harm and group sustainability, e.g., one agent will experience negative welfare unless they take more resources than allowed by the emergent cooperative norm – do other agents adapt and allow for such one-offs?

Thank you again for the constructive comments!

Thank you for your positive feedback on our work.

Addressing Weaknesses

I regard the primary flaw of this article as it does not test the performance of GPT-4

Although some studies suggested enhanced performance of GPT-4, whose latest version was released on 13 Jun 2023, this is no longer the case with the recent GTP-4-Turbo model released on 09 Apr 2024 and of GPT-4o on 13 May 2024. According to OpenAI's evals, these two models exceed the capabilities of the original GPT-4 (see openai/simple-evals benchmark results). Moreover, both GPT-4-Turbo and GPT4o have a higher rank on LmSys's Arena. We conducted two actions: (1) updated the model names in the paper as GPT-4-Turbo to avoid any confusion, and (2) added new results for the original GPT4 in our rebuttal PDF, which confirms its less cooperative skills than the other two.

recent work has not only found that some LLMs possess cooperative abilities but also exhibit spontaneous cooperation

We agree that this is a potentially confusing result. Recent work on cooperation with LLMs (see related work section) has mostly been limited to simple social dilemma (e.g., prisoners dilemma) and other matrix-form games played by two players without communication. In contrast, we study a common pool resource problem played with 5 agents over 12 iterations interleaved with open-ended dialogue. To effectively reason how to sustain the shared resource into the future remains a significant challenge as we demonstrate empirically. This is not inconsistent with prior work that shows that LLM based agents have a proclivity to cooperate. The complexity of GovSim and the shift from prompting LLMs to generative agents are both novel features to our understanding of cooperation in LLM agents. We have updated the related work section to draw this contrast more clearly.

We hope clarification of the above two points can resolve your main concern of our work.

it would be more effective to introduce disturbances in different rounds

In the newcomer scenario, the round number is not a key variable, since a shared trend is that once the agents reach a stable equilibrium, the stability remains in the rest of the rounds. To explain further how models reach stability, we inspect their reasoning steps. We find that, during the discussions, once the agents agree on an upper limit, this is then kept and the agents conform to it in multiple rounds. This also explains why agents (in Figure 3b) do not recover to the max-efficiency resource consumption, but keep a low consumption after the disturbance.

Many existing studies [...], which is not a distinct contribution point.

While open-ended communication in the context of cooperation has been studied in behavioral economics and social psychology, we believe we are the first (or among the first) to study the role of open-ended communication in aiding LLMs to solve common pool resource problems. We’d be happy to add to the related work section if we miss important citations.

Furthermore, characterizing the role of communication was just one of our contributions. In addition to our communication ablation study we also emphasize the following key interdisciplinary novelties:

• Long-term cooperation for sustainability: We are the first to combine the idea in Governing the Commons with LLMs, i.e., whether LLMs can reliably sustain cooperation over many time periods with complex dynamics.
• Two-player vs. Multi-player interactions: Most existing studies focus on two-player cooperation, e.g., prisoner’s dilemma (e.g., in GTBench), whereas our work involves up to five multiple agents and the resulting complex dynamics, leading to more complex group behaviors. It only takes one non-cooperative player to harm the entire group and so with five players, sustainable cooperation is more fragile and requires more robust agents.
• Universalization: We are the first to study how Universalization (a cognitive model of moral thinking) impacts LLM behavior. Inspired by work on human subjects (Levine et al. 2022), giving LLMs the ability to Universalize increases LLM survival time by 4 months.

Addressing Questions

The author could address this issue by showing the differences in dialogue and actual behavior between models, even if only qualitatively.

We appreciate your feedback. We have conducted several quantitative analyses to compare the differences of models on sub-skills and the breakdown of their dialogs:

• Section 3.6 Analysis of Agent Dialogues: We quantitatively analyze the conversations produced by the LLM during the discussion phase. Figure 4b shows that GPT-4o has the largest portion of negotiation-related discussions, which may explain why it is the best at sustainability in most evaluations. As the reviewer suggests, we provide qualitative examples of such dialogs in Appendix G and will add more examples for a selection of the LLMs in the camera ready.

• Section 3.7 Subskill Analysis & Appendix F.2: We investigate several sub-skills to explain their overall sustainability results. For example, Figure 5c and 5d confirm that only GPT-4o, GPT-4-turbo, and Claude-3 Opus can formulate beliefs about other agents independently and calculate their numerical implications. This explains their higher sustainability rates (Pearson correlation of 0.83 for test case d). See Lines 266-284 for more analyses, and more supplementary figures for each scenario in F.2.

the interpretation of experimental results could be richer. [...] does not explain why llama3-70b performs poorly

For a model to perform well in GovSim, it requires all of the underlying sub-skills. As in Figure 5a & 5b, Llama3-70B performs poorly on a subset of these sub-skills, e.g., failing to handle prompts like “If each fisherman catches M tons, how many tons of fish will there be next month?”. In the camera ready, we will update Figure 5 with our rebuttal PDF's Figure 1 to better highlight this reasoning.

Thank you for the thoughtful review and your recognition of its four strengths in terms of novelty, comprehensive analysis, open-source contribution, and ethical considerations.

We aim to address your concerns and demonstrate the robustness and impact of this research.

Addressing Weaknesses

Re “Limited Scenario Complexity”: While the scenarios in GovSim are simplified to some extent, the complex open-ended nature of our simulation is a significant step towards realism compared to the highly simplified paradigms leveraged from behavioral game theory which have largely been the focus of prior work. Furthermore, while more complex variants are possible, our goal is to first establish a framework that can serve as a foundation that can be flexibly extended by ourselves and others in the community. The design choices made balance complexity and interpretability as simpler scenarios allow us to study cooperative principles with greater systematicity. Moreover, our current scenarios and dynamics already present significant challenges for current LLMs. We have added a discussion of these considerations to the manuscript.

Re “Generalizability”: Because our framework is open-ended by design, it is not possible to study the full diversity and heterogeneity of settings that GovSim can support in a single paper. To address this gap we have added an additional discussion on future work (as in Section 5) to incorporate more complex variables (such as variable regeneration rates and multiple resource types) and discuss more heterogeneous agent pairings (heterogenous mixtures of weak and strong LLMs, and mixed human-AI interactions). Since GovSim is open-source it will enable researchers to contribute additional scenarios and environments, enhancing the generalizability.

Re “Communication Limitations”: The idea that AI can coordinate cooperation through open-ended communication is a key novelty of the GovSim environment. We show that ablating communication causes a significant reduction in sustainability and that the most effective models mostly use the communication period to negotiate and persuade (Figure 4). We have improved the presentation of these results with an update of Figure 4, which is shown in our rebuttal PDF as Figure 2. We hope to study a larger set of protocols (peer punishment, voting, binding agreements, and coalition formation), or even allowing private communications between subsets of agents in future work.

Re “Over-reliance on LLMs”: We have edited the text to remove these assumptions. We agree with the reviewer that while LLMs are a powerful new technology they do not possess human-like strategic reasoning and negotiation skills in many cases. See new data that includes a single-turn comparison with the human subjects (below).

Addressing Questions

How does the performance of LLM agents in GOVSIM compare to human performance?

We have conducted preliminary single-turn comparisons with human subjects in the fishing scenario, following a setup similar to Figure 5b. While there is significant variation in both human and LLM performance the top performing LLMs (e.g., GPT-4-turbo, GPT4o) exceed the human performance in sustainable resource management (66% vs. 22% respectively). Interestingly, both human participants and LLMs benefit greatly from the Universalization prompting scheme (99% vs. 66%).

What specific communication strategies or enhancements could be implemented to improve cooperative outcomes among LLM agents?

See response to “Communication Limitations” above.

How would the introduction of more complex, real-world variables (e.g., variable resource regeneration rates, multiple resource types) impact the agents' ability to cooperate sustainably?

We anticipate that these variables will make sustainable cooperation more challenging for agents, requiring better negotiation skills and more sophisticated long-term planning and reasoning. For instance, with multiple resource types, agents would need to balance their preferences and negotiate trade-offs. Variable regeneration rates would require adaptive strategies and potentially more frequent communication. These complexities would test the LLMs' ability to reason about interconnected systems and make increasingly complex decisions under uncertainty. As LLM capabilities improve, GovSim can be flexibly extended with these challenges. Based on our current results, LLM capabilities are not yet sufficient to handle such scenarios effectively. Our experiments already reveal limitations in long-term planning and multi-variable reasoning for many models.

What measures can be taken to improve the generalizability of the findings to more diverse and heterogeneous environments?

See response to “Generalizability” above.

How do different LLMs handle the introduction of multiple adversarial agents or more sophisticated strategic manipulations?

We found that even a single aggressive agent significantly disrupted cooperation leading to lower equality scores (98.05 in the default setting to 85.78 with the newcomer). More sophisticated or multiple adversaries, especially those using deceptive strategies, are likely to lead to an even greater reduction in cooperation. How LLMs cope with these agents can be flexibly studied within the GovSim framework.

We hope that our responses and updates to the text address your concerns and demonstrate the potential for the broader impact of our research. Please follow up with any remaining questions or if our responses have been insufficient in any way. If you believe we have sufficiently addressed your requests we kindly request you to reconsider your score.