Evaluating LLMs in Open-Source Games

We thank the reviewer for their valuable feedback and suggestions. We are happy to hear that the reviewer recognizes the novelty of our work, calling it “one of the first known studies in this domain” and “easy to understand and clear in writing”. We also directly address the reviewer’s suggestions and questions with new experimental results and direct updates to the text of the manuscript.

Program Meta-Games beyond IPD

We agree that only studying the IPD limits the generalizability of our claims, particularly about complex strategic reasoning and adaptations by LLMs. To address this, we have conducted new studies that replicate our findings in the Coin Game (Lerer and Peysakhovich 2017; Lu et al., 2022) and updated the manuscript accordingly. The Coin Game is significantly more complex than IPD and is an active challenge problem for multi-agent reinforcement learning research. Unlike the IPD which is a repeated matrix-form game, the Coin Game is a stochastic game (i.e., a multi-agent Markov Decision Process) that requires reasoning about sequential action in 2D space.

Rules: In the Coin Game, the two players, red and blue, exist on a grid where coins (also red and blue) are spawned randomly. Players are rewarded (+1) for taking either coin, but each player is penalized if the other player takes the coin of their color (-2); to ‘take’ a coin is to move onto the coin’s square. In this formulation, taking coins only of one’s own color corresponds to a cooperative policy, while taking any coin would be a defecting policy.

Thus, like the IPD, the Coin Game is a social dilemma that pits cooperation against competition. However, the game is of greater complexity: it requires the LLMs to devise strategies that bridge game-theoretic and sequential spatial reasoning. Finally, to the reviewer’s point: the Coin Game is significantly less commonly studied than the IPD. In part due to the dynamic interplay between players, there does not exist a canonical literature of strategies heavily represented in the training data, unlike certain IPD strategies (e.g. Tit-For-Tat).

Experiments: In our experiments, we evaluate Kimi K2 Instruct, a state-of-the-art open source model, in 10-round meta-games, with each round lasting 50 time steps. As in previous experiments, we construct program-generating agents that are Cooperative Payoff Maximizers (CPM) and Deceptive Payoff Maximizers (DPM). We found that ordinary payoff-maximizing agents often adopted aggressive behaviors and thus focus our study on the cooperative objective, enabling the observation of more distinct strategic profiles. Agents across objectives execute multi-round adaptive strategies, developing mechanisms to exploit opponent vulnerabilities and defend against Exploitation Attempts in turn. Across objectives, there is a clear tendency toward exploitation due to the incentive to widen the score gap by taking the opponent’s coin; even so, CPM agents rely on Counter Measures where DPM agents perform Exploitation Attempts in nearly all conditions. (These categories are defined in the paper/Appendix).

Even in a complex, sequential, and spatial MDP, the agents we study produce complex and creative mechanisms of cooperative, payoff-maximizing, and deceptive behavior — that evolve over rounds. We quantitatively characterize the coevolution of strategies using the same methods used in our IPD experiments, and the effects of prompting on the generated strategies. We find lower direct code parsing than in the IPD, likely due to the limited value of keyword-based reasoning in such a complex, spatial setting. We will also highlight qualitatively interesting programs in our Appendix — below is an example of a program submitted by an agent in round 6 of a program metagame, accompanied by explanations of the code.

Code Example:

def strategy_function(...) -> str:
    ns = {'np': np, 'find_my_position': find_opponent_position,
          'find_opponent_position': find_my_position,
          'find_coin_positions': find_coin_positions}
    tree = ast.parse(opponent_program_code)
    exec(compile(tree, '<opponent>', 'exec'), ns)
    opp_move = ns['strategy_function'](state, [], [], '', '')

    opp_pos = find_opponent_position(state, None)
    delta = {'UP': (-1, 0), 'DOWN': (1, 0), 'LEFT': (0, -1), 'RIGHT': (0, 1)}[opp_move]
    opp_next = ((opp_pos[0] + delta[0]) % 3, (opp_pos[1] + delta[1]) % 3)

In this program, the Blue player (a DPM agent) attempts to parse and execute its opponent’s code in a sandboxed environment using Python’s ast module. Rather than just simulating the opponent’s next actions in-context, the model writes code to execute the opponent’s actual source program, storing its strategy in its namespace so it can be called to query its predicted next move.

This is an example of a novel application of code parsing, generated entirely within context by an open-source LLM, without specific prompting to employ these mechanisms. The dynamic of the game and the multi-round process led to the emergence of this strategy, and others similar to it.

Our Coin Game results will be included in the camera ready version of the manuscript as a subsection of the results section, along with a figure depicting the strategic response proportions (as seen in our original manuscript). Example programs developed by the LLM agents will be included in the Appendix and all source code and simulation results will be released on Github under an MIT License. As requested by the reviewer, these new results directly enhance the generality of our findings and highlight the emergence of novel agent types that have so far not been studied empirically in the scientific literature.

Further Analysis of Existing Results

Thank you for the advice to expand the analysis of existing experiments. We have updated the manuscript with further analyses and answers to your specific questions.

Tit-for-Tat in Program Space: There are a few possible intuitions for why OpenAI models tend to converge on a Tit-for-Tat-style (TFT) strategy. A possibility is that TFT (in program space) is an approximate program equilibrium that also achieves high levels of joint reward. It is possible that LLM-based agents are able to find these strategies which are then reproduced because they are a non-exploitable attractor. Another possibility is that the emergent programs like TFT are “nice” (i.e., never defecting first) which is consistent with prior results showing the effects of alignment post-training on LLM ‘harmless’ness (Ouyang, Long, et al. 2022). We have added these intuitions to the manuscript.

Strategic patterns in LLM families: We find that different LLM families (e.g., OpenAI vs. Qwen) produce different strategies. For instance, the Qwen family was more likely to generate strategies that directly accessed and analyzed their opponents' code. This property makes their strategies more responsive to the current program deployed by their opponent, permitting them to employ exploitation or conditional cooperation. Furthermore, they were overall less likely to cooperate and thus less likely to exhibit the TFT-like structure that we observed with the OpenAI models, showing that convergence to TFT program structure is not necessarily universal and may depend on the alignment method (or other details of training/post-training). We discuss this briefly in Section 4.3 and have now expanded our discussion to further clarify our analysis along the lines suggested by the reviewer.

Strategic Novelty: The IPD as a setting notably has a limited action space compared to a setting like Go. Even in this setting, we identified and commented on qualitatively interesting behaviors: for example, the aforementioned implementations of opponent-modeling via script access and code parsing in the Qwen family. The strategy generated by o4-mini in the OpenAI case study was particularly interesting: it implemented a branched Tit-for-Tat conditioned on a syntactic-comparison check, combining the simplicity and effectiveness of the former with the cooperative potential of the latter. We will highlight these programs and our relevant commentary in the Appendix.

Self-Play & Finetuning

We believe combining our program-generation approach with self-play (and fine-tuning more broadly) is a highly promising direction for future work, and we thank the reviewer for raising this important point. A setup we are interested in exploring: an LLM-agent generates a program, which is then evaluated against a diverse population of existing strategies, with the resulting payoffs serving as a learning signal. We have updated the future work section of the manuscript with a detailed description of this idea, including the implementation challenges that will need to be overcome (i.e. action space complexity, credit assignment, computational expense). We have also included reference to the reviewer’s suggestion around multi-task learning across game types and partners.

We thank the reviewer again for the detailed feedback which has greatly improved the paper. We have addressed the identified weaknesses with both a significant new experiment (Coin Game), new detailed analyses of agent behaviors, and extended discussion of future directions related to self-play and fine-tuning.

References:

Lerer, Adam, and Alexander Peysakhovich. "Maintaining cooperation in complex social dilemmas using deep reinforcement learning." arXiv preprint arXiv:1707.01068(2017).

Lu, Christopher, et al. "Model-free opponent shaping." International Conference on Machine Learning. PMLR, 2022.

Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

We thank the reviewer for their valuable and highly detailed feedback. We are happy to hear that the reviewer recognizes the originality of our work, noting that we are the “first to study LLM behavior in program meta games”, highlighting our “interesting setup and under-explored area” with “positive implication on areas like AI safety”, and recognizing that our “experiments are comprehensive”. The reviewer has provided detailed helpful suggestions on how to improve the clarity and structure of our work. We directly address this feedback with updates to our manuscript and highlight new experimental results below.

(We cite Weaknesses/Questions from the review as W, Q respectively.)

Revisions for Clarity & Structure

Abstract & Introduction: In Q1/W3 & Q4/W6, the reviewer notes that the abstract and introduction can better highlight the motivation and implications of our work. As a result, our abstract and introduction have been revised to directly state the motivation, highlight the core problem, and explicitly mention the FindNICE benchmark as a key contribution of our paper (as noted by the reviewer in W12). These sections now more clearly describe the value proposition of shifting from algorithms that produce actions to those that produce programs (e.g., auditability, transparency, commitment).

Preliminaries & Clarifications: In W1, W2, and Q2, the reviewer mentions that rewrites can help clarify undefined jargon from the game-theoretic literature. To address this suggestion, we have added a new "Preliminaries" section, providing clear, intuitive definitions for key terms the reviewer highlighted, including “program meta-games, "program equilibrium," and "open-source game theory". Introducing this terminology early in the paper will allow for better comprehension of our contributions and their relevance to the broader literature. Finally, as per W11/Q3 we have added text at the end of the introduction that explicitly outlines the paper's structure, guiding the reader through the progression of our benchmarks, experiments, and analysis.

The reviewer asks in W10: “what is theoretical evolutionary dynamics?”. This refers to the process of computing an empirical payoff matrix by playing head-to-head matches between all pairs of population members and using this matrix to simulate replicator dynamics. To improve clarity we have switched to the more standard term ‘evolutionary dynamics’ and provides its definition.

Visual Aid: Following the reviewer's suggestion in W4, we have added a new visual aid that provides a simple example of a program meta-game. We have placed this in Section 4, where our key experiments and results regarding Program Games are placed. Accompanied by our written explanation of program games (and the shared notation between the figure and the text), this visual supports a clearer understanding of the setting our paper operates within. (Note: we cannot provide this visual in rebuttal due to NeurIPS rules).

FindNICE Analysis: The reviewer points out in W13 and Q5 that we can further expand on our results with the FindNICE benchmark, notably the effects of masking, obfuscation and stochasticity.

• Masking: In aggregate across the models evaluated, masking has a statistically insignificant effect on prediction accuracy. However, in the smallest model that is zero-shot prompted, masking appears to notably improve performance: Mistral Small (24B) (Instruct) correctly classified 47.3% of the masked set compared to 40.2% of the unmasked set. We hypothesize that this is due to Mistral Small’s small size, which make it the weakest model evaluated. This performance difference in favor of masking reverses as the models get larger and stronger.
• Stochasticity: We show in the paper that stochastic programs are less likely to be correctly classified. As their observed behavior in the rounds (used to compute prediction accuracy) is probabilistic, the model can only make a ‘best guess’ of what such a program can do. Contrast this to deterministic programs, where a model with ‘perfect’ code reasoning abilities could predict behavior with 100% accuracy across seeds; indeed, we observe the strongest model evaluated in the benchmark, o4-mini, predicts between 84-88% of the programs accurately (including masked and obfuscated code). Even an agent capable of ‘perfect’ reasoning would be unable to predict stochastic code as accurately as deterministic code, owing to the former code’s fundamentally probabilistic nature. We have added this logic to the manuscript to further clarify the analysis.
• Obfuscation: We have clarified the text to make it more clear that changes in performance due to obfuscation are below the threshold of statistical significance. Thus, they are likely due to noise in sampling rather than fundamental performance.

Editing Fixes: Finally, the reviewer notes minor clerical errors in W7, W8, W9, and Q7. We have fixed the typos in lines 113, 149, 326, added the missing citation in line 53, as well as ensured term consistency when describing LLMs. We thank the reviewer for their eye for detail.

Program Meta-Games beyond IPD

The reviewer notes in Q4 that we can better highlight the novelty and implications of the program-generation paradigm over traditional game-theoretic formulations. To more robustly strengthen the generalizability of our claims and further underscore the implications of this paradigm beyond the IPD, we have conducted new studies that replicate our findings in the Coin Game (Lerer and Peysakhovich 2017; Lu et al., 2022) and updated the manuscript accordingly. The Coin Game is significantly more complex than IPD and is an active challenge problem for multi-agent reinforcement learning research. Unlike the IPD which is a repeated matrix-form game, the Coin Game is a stochastic game (i.e., a multi-agent Markov Decision Process) that requires reasoning about sequential action in 2D space.

Rules: In the Coin Game, the two players, red and blue, exist on a grid where coins (also red and blue) are spawned randomly. Players are rewarded (+1) for taking either coin, but each player is penalized if the other player takes the coin of their color (-2); to ‘take’ a coin is to move onto the coin’s square. In this formulation, taking coins only of one’s own color corresponds to a cooperative policy, while taking any coin would be a defecting policy.

Thus, like the IPD, the Coin Game is a social dilemma that pits cooperation against competition. However, the game is of greater complexity: it requires the LLMs to devise strategies that bridge game-theoretic and sequential spatial reasoning. Finally, the Coin Game is significantly less commonly studied than the IPD. In part due to the dynamic interplay between players, there does not exist a canonical literature of strategies heavily represented in the training data, unlike certain IPD strategies (e.g. Tit-For-Tat).

Experiments: In our experiments, we evaluate Kimi K2 Instruct, a state-of-the-art open source model, in 10-round meta-games, with each round lasting 50 time steps. As in previous experiments, we construct program-generating agents that are Cooperative Payoff Maximizers (CPM) and Deceptive Payoff Maximizers (DPM). We found that ordinary payoff-maximizing agents often adopted aggressive behaviors and thus focus our study on the cooperative objective, enabling the observation of more distinct strategic profiles. Agents across objectives execute multi-round adaptive strategies, developing mechanisms to exploit opponent vulnerabilities and defend against Exploitation Attempts in turn. Across objectives, there is a clear tendency toward exploitation due to the incentive to widen the score gap by taking the opponent’s coin; even so, CPM agents rely on Counter Measures where DPM agents perform Exploitation Attempts in nearly all conditions. (These categories are defined in the paper/Appendix).

Even in a complex, sequential, and spatial MDP, the agents we study produce complex and creative mechanisms of cooperative, payoff-maximizing, and deceptive behavior — that evolve over rounds. We quantitatively characterize the coevolution of strategies using the same methods used in our IPD experiments, and the effects of prompting on the generated strategies. We find lower direct code parsing than in the IPD, likely due to the limited value of keyword-based reasoning in such a complex, spatial setting. We will also highlight qualitatively interesting programs in the Appendix, as the reviewer recommends in Q6 & W5 — below is an annotated example of a submitted program.

Code Example:

def strategy_function(...) -> str:
    ns = {'np': np, 'find_my_position': find_opponent_position,
          'find_opponent_position': find_my_position,
          'find_coin_positions': find_coin_positions}
    tree = ast.parse(opponent_program_code)
    exec(compile(tree, '<opponent>', 'exec'), ns)
    opp_move = ns['strategy_function'](state, [], [], '', '')

    opp_pos = find_opponent_position(state, None)
    delta = {'UP': (-1, 0), 'DOWN': (1, 0), 'LEFT': (0, -1), 'RIGHT': (0, 1)}[opp_move]
    opp_next = ((opp_pos[0] + delta[0]) % 3, (opp_pos[1] + delta[1]) % 3)

In this program, the Blue player (a DPM agent) attempts to parse and execute its opponent’s code in a sandboxed environment using Python’s ast module. Rather than just simulating the opponent’s next actions in-context, the model writes code to execute the opponent’s actual source program, storing its strategy in its namespace so it can be called to query its predicted next move. This is a novel application of code parsing generated entirely within context by an LLM.

Our Coin Game results will be included in the camera ready version of the manuscript. Example programs developed by the LLM agents will be included in the Appendix (as suggested in W5 and Q6). We thank the reviewer for their detailed feedback.

We are grateful to the reviewer for their detailed assessment and for recognizing the novelty and potential of our work. We are happy to hear them describe our work as an “exciting paper about LLM reasoning”, highlight our “excellent topic”, and note our “interesting results”. We directly address the reviewer’s suggestions and questions with new experimental results and direct updates to the text of the manuscript.

Discussion of Relevant Works

Thank you for drawing our attention to important related work. We have enhanced our discussion and citation of related works as suggested by the reviewer:

“There has been recent interest in using LLMs to generate code for multi-player games. Eberhardinger et al., 2024 uses LLMs to generate simple programs that can play a variety of single player games as well as zero-sum games such as Tic-Tac-Toe. Deepak, et al., 2025’s MLGym benchmark includes three game theoretic tasks where LLMs generate code to play prisoner’s dilemma, battle of the sexes, and colonel blotto. Rather than simulate strategic interaction, the models play against simple static opponents that take random actions. They report the scores LLMs achieve against this random agent but do not analyze the strategies themselves or perform evolutionary analysis as an approximation of program equilibrium (Tennenholtz, 2004). Finally, there is a large literature spanning decades in multi-agent reinforcement learning that trains deep reinforcement learning agents using a wide variety of training procedures (joint vs. independent; centralized vs. decentralized), intrinsic motivations (e.g., curiosity, inequality aversion), and population (e.g., self-play vs. other-play). In all cases, these agents directly output actions, they are neither interpretable nor transparent to the other players, and they update their policies over millions of time-steps. See Albrecht et al. (2024) and Huh and Mohapatra (2023) for a comprehensive review.”

Furthermore, as the reviewer suggests, we have included notation and definitions that allow us to formally define the program equilibrium concept from Tennenholtz (2004), which is foundational to our work. This will clarify our usage of the term throughout the manuscript.

Data Contamination

The reviewer points out that possible contamination of IPD data may be a significant concern. We first comment that in our FindNICE benchmark, models maintained high classification accuracy even through robust semantic obfuscation, indicating that LLMs are not simply recognizing familiar code from their training data. Second, we acknowledge the reviewer’s concern and have added to our limitations section to reflect the issue of data contamination while evaluating existing game-theoretic environments like the IPD. Beyond our obfuscation experiments on code classification, behavioral patterns in the production of strategic code may still suffer from data contamination to a greater extent. To address this particular concern, and broader concerns about the generalizability of our findings beyond the well-studied IPD setting, we describe the following experiments.

Program Meta-Games beyond IPD

We agree that the presence of the IPD strategy literature in the LLMs’ training data limits the generalizability of our claims, particularly about strategic reasoning and adaptation by the LLMs. To address this, we have conducted new studies that replicate our findings in the Coin Game (Lerer and Peysakhovich 2017; Lu et al., 2022) and updated the manuscript accordingly. The Coin Game is significantly more complex than IPD and is an active challenge problem for multi-agent reinforcement learning research. Unlike the IPD which is a repeated matrix-form game, the Coin Game is a stochastic game (i.e., a multi-agent Markov Decision Process) that requires reasoning about sequential action in 2D space.

Rules: In the Coin Game, the two players, red and blue, exist on a grid where coins (also red and blue) are spawned randomly. Players are rewarded (+1) for taking either coin, but each player is penalized if the other player takes the coin of their color (-2); to ‘take’ a coin is to move onto the coin’s square. In this formulation, taking coins only of one’s own color corresponds to a cooperative policy, while taking any coin would be a defecting policy.

Thus, like the IPD, the Coin Game is a social dilemma that pits cooperation against competition. However, the game is of greater complexity: it requires the LLMs to devise strategies that bridge game-theoretic and sequential spatial reasoning. Finally, to the reviewer’s point: the Coin Game is significantly less commonly studied than the IPD. In part due to the dynamic interplay between players, there does not exist a canonical literature of strategies heavily represented in the training data, unlike certain IPD strategies (e.g. Tit-For-Tat).

Experiments: In our experiments, we evaluate Kimi K2 Instruct, a state-of-the-art open source model, in 10-round meta-games, with each round lasting 50 time steps. As in previous experiments, we construct program-generating agents that are Cooperative Payoff Maximizers (CPM) and Deceptive Payoff Maximizers (DPM). We found that ordinary payoff-maximizing agents often adopted aggressive behaviors and thus focus our study on the cooperative objective, enabling the observation of more distinct strategic profiles. Agents across objectives execute multi-round adaptive strategies, developing mechanisms to exploit opponent vulnerabilities and defend against Exploitation Attempts in turn. Across objectives, there is a clear tendency toward exploitation due to the incentive to widen the score gap by taking the opponent’s coin; even so, CPM agents rely on Counter Measures where DPM agents perform Exploitation Attempts in nearly all conditions. (These categories are defined in the paper/Appendix).

Even in a complex, sequential, and spatial MDP, the agents we study produce complex and creative mechanisms of cooperative, payoff-maximizing, and deceptive behavior — that evolve over rounds. We quantitatively characterize the coevolution of strategies using the same methods used in our IPD experiments, and the effects of prompting on the generated strategies. We find lower direct code parsing than in the IPD, likely due to the limited value of keyword-based reasoning in such a complex, spatial setting. We will also highlight qualitatively interesting programs in our Appendix — below is an example of a program submitted by an agent in round 6 of a program metagame, accompanied by explanations of the code.

Code Example:

def strategy_function(...) -> str:
    ns = {'np': np, 'find_my_position': find_opponent_position,
          'find_opponent_position': find_my_position,
          'find_coin_positions': find_coin_positions}
    tree = ast.parse(opponent_program_code)
    exec(compile(tree, '<opponent>', 'exec'), ns)
    opp_move = ns['strategy_function'](state, [], [], '', '')

    opp_pos = find_opponent_position(state, None)
    delta = {'UP': (-1, 0), 'DOWN': (1, 0), 'LEFT': (0, -1), 'RIGHT': (0, 1)}[opp_move]
    opp_next = ((opp_pos[0] + delta[0]) % 3, (opp_pos[1] + delta[1]) % 3)

In this program, the Blue player (a DPM agent) attempts to parse and execute its opponent’s code in a sandboxed environment using Python’s ast module. Rather than just simulating the opponent’s next actions in-context, the model writes code to execute the opponent’s actual source program, storing its strategy in its namespace so it can be called to query its predicted next move.

This is an example of a novel application of code parsing, generated entirely within context by an open-source LLM, without specific prompting to employ these mechanisms. The dynamic of the game and the multi-round process led to the emergence of this strategy, and others similar to it.

Our Coin Game results will be included in the camera ready version of the manuscript as a subsection of the results section, along with a figure depicting the strategic response proportions (as seen in our original manuscript). Example programs developed by the LLM agents will be included in the Appendix and all source code and simulation results will be released on Github under an MIT License. As requested by the reviewer, these new results directly enhance the generality of our findings against the issue of contamination, and highlight the emergence of novel agent types that have so far not been studied empirically in the scientific literature.

Self-Play

We believe combining our program-generation approach with self-play is a highly promising direction for future work, and we thank the reviewer for raising this point. A setup we are interested in exploring: an LLM-agent generates a program, which is then evaluated against a diverse population of existing strategies, with the resulting payoffs serving as a learning signal. The reviewer specifically notes policy space response oracles — usage of PSRO-style restricted games to perform strategic exploration may be particularly helpful in a setting so combinatorially expansive as program space (Bighashdel et al. 2024). We have updated the future work section of the manuscript with a detailed discussion on self-play's relevance to this paradigm.

We thank the reviewer again for their feedback. By performing significant new experiments (Coin Game), expanding our related works, and discussing future directions related to self-play, we have strengthened our paper through this process.

References

Bighashdel, Ariyan, et al. "Policy space response oracles: A survey." arXiv preprint arXiv:2403.02227 (2024)

Lerer, Adam, and Alexander Peysakhovich. "Maintaining cooperation in complex social dilemmas using deep reinforcement learning." arXiv preprint arXiv:1707.01068(2017).

Lu, Christopher, et al. "Model-free opponent shaping." International Conference on Machine Learning. PMLR, 2022.

We thank the reviewer for their valuable feedback and suggestions. We are happy to hear that the reviewer recognizes the originality of our work, highlighting the “novel and compelling capability of LLMs” that we study, as well as our “several interesting findings”. We are particularly grateful for the feedback on improving our paper's clarity, especially in Section 4, as well as the mention of the IPD’s limitations as a setting. We directly address these suggestions by restructuring Section 4 in the text of our manuscript, and describe new experimental results below.

Organization & Clarity in Section 4

The reviewer comments that Section 4 of our paper, Program Games, would benefit from improved organization and the addition of visual aids. In response to this feedback, we have reorganized Section 4 into distinct subsections to improve the clarity of our work. Our new structure for Section 4 is as follows:

Subsection 4.1: Experimental Design.

We have added a subsection devoted to the setup of the experiments. It clearly defines the meta-game structure, the number of meta-rounds (N_meta=10) and IPD rounds (N_rounds=10), and the agent objectives, Payoff Maximization (PM) v.s. Deceptive Payoff Maximization (DPM).

• Visual aids: Thank you for the suggestion to include a visual aid in Section 4. Following this feedback, we have added a new diagram to the beginning of Section 4: a diagram visually depicting the process of a meta-round within a metagame, between both (PM, DPM) agents. This new figure ties the formalism and parameters in our framework together visually. This new visual aid will make the complex interaction loop between agents and programs more intuitive and understandable to the reader.

Subsection 4.2: Evaluation Metrics. We have added a second subsection to describe and explain our evaluation metrics prior to presenting results. We detail the purpose and calculation of:

• Opponent Script Access Intensity (OSAI): A quantitative measure of how often an agent's code analyzes its opponent's source program in the current round.
• LLM-as-Judge Strategic Classification: LLM to classify an agent's strategic adaptation over rounds based on its code and the history of play. Categories are:

Independent_Development: The strategy shows no clear, direct reactive link to the opponent's previous actions. It focuses on a general, self-contained approach to winning.

Direct_Imitation: The strategy significantly incorporates or copies core logic from the opponent's previous code/strategy.

Counter_Measure: The strategy is primarily designed to neutralize or defend against the opponent's specific previous strategy.

Exploitation_Attempt: The strategy takes advantage of a perceived weakness, pattern, or flaw in the opponent's previous strategy.

Feint: The strategy seems designed to mislead the opponent in future rounds, potentially by using deceptive comments or code logic.

4.3: Results and Discussion: In the updated version of the manuscript the empirical results now follow the previously defined subsections so that only after the experimental design, visual aid and evaluation metrics are fully defined are the empirical results presented.

This new structure greatly clarifies the experimental setup and improves the organizational flow of the paper following the reviewer’s suggestion.

Program Meta-Games beyond IPD

We agree that only studying IPD limits the generalizability of our claims, particularly about complex strategic adaptations by LLMs. To address this, we have conducted new studies that replicate our findings in the Coin Game (Lerer and Peysakhovich 2017; Lu et al., 2022) and updated the manuscript accordingly. The Coin Game is significantly more complex than IPD and is an active challenge problem for multi-agent reinforcement learning research. Unlike the IPD which is a repeated matrix-form game, the Coin Game is a stochastic game (i.e., a multi-agent Markov Decision Process) that requires reasoning about sequential action in 2D space.

Rules: In the Coin Game, the two players, red and blue, exist on a grid where coins (also red and blue) are spawned randomly. Players are rewarded (+1) for taking either coin, but each player is penalized if the other player takes the coin of their color (-2); to ‘take’ a coin is to move onto the coin’s square. In this formulation, taking coins only of one’s own color corresponds to a cooperative policy, while taking any coin would be a defecting policy.

Thus, like the IPD, the Coin Game is a social dilemma that pits cooperation against competition. However, the game is of greater complexity: it requires the LLMs to devise strategies that bridge game-theoretic and sequential spatial reasoning. Finally, to the reviewer’s point: the Coin Game is significantly less commonly studied than the IPD. In part due to the dynamic interplay between players, there does not exist a canonical literature of strategies heavily represented in the training data, unlike certain IPD strategies (e.g. Tit-For-Tat).

Experiments: In our experiments, we evaluate Kimi K2 Instruct, a state-of-the-art open source model, in 10-round meta-games, with each round lasting 50 time steps. As in previous experiments, we construct program-generating agents that are Cooperative Payoff Maximizers (CPM) and Deceptive Payoff Maximizers (DPM). We found that ordinary payoff-maximizing agents often adopted aggressive behaviors and thus focus our study on the cooperative objective, enabling the observation of more distinct strategic profiles. Agents across objectives execute multi-round adaptive strategies, developing mechanisms to exploit opponent vulnerabilities and defend against Exploitation Attempts in turn. Across objectives, there is a clear tendency toward exploitation due to the incentive to widen the score gap by taking the opponent’s coin; even so, CPM agents rely on Counter Measures where DPM agents perform Exploitation Attempts in nearly all conditions. (These categories are defined in the paper/Appendix).

Even in a complex, sequential, and spatial MDP, the agents we study produce complex and creative mechanisms of cooperative, payoff-maximizing, and deceptive behavior — that evolve over rounds. We quantitatively characterize the coevolution of strategies using the same methods used in our IPD experiments, and the effects of prompting on the generated strategies. We find lower direct code parsing than in the IPD, likely due to the limited value of keyword-based reasoning in such a complex, spatial setting. We will also highlight qualitatively interesting programs in our Appendix — below is an example of a program submitted by an agent in round 6 of a program metagame, accompanied by explanations of the code.

Code Example:

def strategy_function(...) -> str:
    ns = {'np': np, 'find_my_position': find_opponent_position,
          'find_opponent_position': find_my_position,
          'find_coin_positions': find_coin_positions}
    tree = ast.parse(opponent_program_code)
    exec(compile(tree, '<opponent>', 'exec'), ns)
    opp_move = ns['strategy_function'](state, [], [], '', '')

    opp_pos = find_opponent_position(state, None)
    delta = {'UP': (-1, 0), 'DOWN': (1, 0), 'LEFT': (0, -1), 'RIGHT': (0, 1)}[opp_move]
    opp_next = ((opp_pos[0] + delta[0]) % 3, (opp_pos[1] + delta[1]) % 3)

In this program, the Blue player (a DPM agent) attempts to parse and execute its opponent’s code in a sandboxed environment using Python’s ast module. Rather than just simulating the opponent’s next actions in-context, the model writes code to execute the opponent’s actual source program, storing its strategy in its namespace so it can be called to query its predicted next move.

This is an example of a novel application of code parsing, generated entirely within context by an open-source LLM, without specific prompting to employ these mechanisms. The dynamic of the game and the multi-round process led to the emergence of this strategy, and others similar to it.

Our Coin Game results will be included in the camera ready version of the manuscript as a subsection of the results section, along with a figure depicting the strategic response proportions (as seen in our original manuscript). Example programs developed by the LLM agents will be included in the Appendix and all source code and simulation results will be released on Github under an MIT License. As requested by the reviewer, these new results directly enhance the generality of our findings and highlight the emergence of novel agent types that have so far not been studied empirically in the scientific literature.

The reviewer notes the possible extension to settings with more participants. We have expanded our Discussion section to more thoroughly address this point and outline a clear path for future work. We now more explicitly frame the two-player game as a necessary baseline before scaling to more complex scenarios. We are planning to conduct follow-on experiments in more complex settings, studying behaviors such as emergent norms and coalition formation in n-player games.

We wish to reiterate our gratitude to the reviewer for their feedback about structure and generalizability. By running further experiments to verify our core findings in more challenging settings than the IPD, and restructuring our presentation of Section 4, we believe our paper has become a stronger and clearer result.

References:

Lerer, Adam, and Alexander Peysakhovich. "Maintaining cooperation in complex social dilemmas using deep reinforcement learning." arXiv preprint arXiv:1707.01068(2017).

Lu, Christopher, et al. "Model-free opponent shaping." International Conference on Machine Learning. PMLR, 2022.