We thank you for genuinely providing valuable comments on our paper. We will address your concerns one by one.

Originality:

We appreciate your question regarding the games covered in our benchmark relative to “A Survey of Strategic Reasoning with Large Language Models” and examples involving Theory of Mind (ToM). We acknowledge that strategic reasoning is a vast domain with many forms, and we chose to focus on a specific subset of games (within the subset, the atomic games have a common form but different configurations and descriptions), particularly those resembling scenarios like the Prisoner’s Dilemma, Stag Hunt, etc., where strategic reasoning is fundamental. These games are a subset of the larger landscape of strategic reasoning tasks, and even within this subset, we found that current LLMs struggle with reasoning consistently and accurately.

In summary, compared to previous work, our benchmark: (1) considers a more comprehensive set of 144 types of 2x2 games; (2) explores different contextual framings of the same game structure in greater depth; and (3) introduces three novel complex game forms—sequential, parallel, and nested—based on atomic games, which were not designed in prior studies.

By focusing on this category of problems, we aim to reveal specific deficiencies in LLMs’ strategic reasoning abilities, such as the “asymmetric pattern” observed in symmetric games, which has not been well-studied in prior benchmarks. While other benchmarks may explore broader game-theoretic concepts, our contribution lies in systematically evaluating LLMs on a targeted and relevant class of strategic reasoning tasks.

---

Quality:

Thank you for suggesting the incorporation of statistical tests to assess differences between models. We fully agree that this would enhance the rigor of the paper and strengthen our findings. In future revisions and subsequent work, we plan to include such statistical analyses.

We also share your observation that some games, such as the Prisoner’s Dilemma, might be more commonly exposed to LLMs during training, potentially leading to better performance on those games. To investigate this, we collected experimental data and computed metrics for two distinct sets of games: the counter-diagonal set (which includes well-known games like the Prisoner’s Dilemma, Stag Hunt, and Battle of the Sexes) and the non-counter-diagonal set. The results are summarized in the table below (using DA prompting):

From the table, we observe that for advanced models such as gpt-4o, claude-3-5-sonnet, and Qwen2-72B, performance on the famous set of games does not consistently surpass (and in some cases is lower than) performance across all games. Conversely, for models like Llama-3.1-70B and gpt-4o-mini, the famous game set appears to be relatively easier. This is a fascinating finding and may indicate potential training data leakage for the more well-known games.

We acknowledge that this raises a significant and valuable research question, and we plan to explore this direction further in future work. Your observation has been instrumental in highlighting an area that warrants deeper investigation. Thank you for bringing this to our attention.

---

Clarity:

We appreciate your suggestions for improving clarity. We revise the introduction in our new revision on line 91 and line 94 respectively. Also we include another appendix section to provide clearer explanations for some terms. Regarding Figure 6, we will explore alternative visualizations that better highlight the differences between the classic and story-based settings, as suggested.

We sincerely thank you for your detailed comments and constructive feedback. Below, we address the specific concerns you raised:

Question: The sequential/parallel form games seem not to hold the task of testing for strategic reasoning.

Response: We sincerely apologize for not clearly articulating the significance and importance of complex games in our benchmark. It is worth further emphasizing that our motivation for designing complex form games is to highlight that the atomic games in TMGBench resemble the “primary components” of complex real-world social scenarios. Specifically:

• For sequential games, real-life scenarios often require making decisions one after another to solve problems.
• For parallel games, such as in diplomatic contexts, governments often need to simultaneously make decisions in multiple domains, including technology, military, politics, and culture.
• For nested games, as seen in scenarios like auctions, the decisions made in earlier auctions often influence subsequent ones.

Thus, the complex game forms we designed can effectively represent various complex strategic scenarios in real life, enabling a more in-depth evaluation of large language models’ strategic reasoning capabilities.

---

Question: Potential ambiguous prompt expression.

Response: We understand your concerns. Prior to our formal experiments, we conducted extensive prompt engineering to ensure that the prompts we currently use can reliably test the models. Additionally, the experimental results in our formal tests have validated that the prompts effectively and consistently evaluate the models.

---

Question: Proofreading errors.

Response: We are grateful that you meticulously pointed out parts of the text that may not be easy for readers to understand. We have revised and improved these sections in the paper accordingly based on your guidance (respectively on line 124, line 224, line 235, line 364, line 317, line 476).

---

Question: The potential cause of Llama70B performing worse on DA than 8B.

Response: We acknowledge your observation regarding the underperformance of the DA prompt (Llama70B performing worse than Llama8B). Here is our interpretation of this finding:

• Positional Bias: One possible explanation is positional bias. In some cases, larger models may exhibit stronger biases towards certain choices rather than random ones, leading to suboptimal performance. We will analyze this phenomenon further in the data.
• Emergent Abilities: The poor performance of both Llama70B and Llama8B under the DA prompt suggests that these models do not exhibit emergent abilities without CoT prompting. This highlights the limitations of current LLMs in strategic reasoning tasks, even as model size increases. Future Work: We plan to include additional statistical analysis to strengthen our interpretation and discuss this observation more thoroughly in the experimental results section of the revised paper.

We look forward to engaging in further discussions with you and receiving your additional guidance and feedback. Thank you!

First, we thank you for giving kind advice and insightful opinions, and we will address your questions one by one.

Of Weakness 1:

Question: Lack of a clear definition for strategic reasoning.

Response: We fully understand your concern about the lack of a clear definition of strategic reasoning and would like to provide further clarification. In the introduction of our paper, we referenced a survey paper titled "A Survey of Strategic Reasoning with Large Language Models", which defines strategic reasoning and discusses the key characteristics of tasks that require it. We used the concept derived from that, and we have already added more detailed explanations in the appendix to make it clearer for readers. Please refer to Appendix A of the latest revised paper. Here is a short explanation:

1. Definition of Strategic Reasoning: Strategic reasoning can be defined as the ability to make decisions based not only on one’s own actions but also on predicting and responding to the actions of others, especially in environments where outcomes depend on the interdependence of agents’ decisions. This definition distinguishes strategic reasoning from other forms of reasoning, such as common-sense reasoning or logical reasoning.

2. Characteristics of Strategic Reasoning: The core feature of strategic reasoning is its reliance on anticipating and responding to the behavior of other participants.

3. Necessity and Applications of Strategic Reasoning: Strategic reasoning is vital in various fields, including decision-making intelligence and social intelligence. For example, strategic reasoning is required in games like poker and chess, where players must predict and counter their opponents’ moves.

---

Question: Differences Between Our Benchmark and Existing Ones.

Response:

1. Game Coverage: TMGBench focuses on a specific but symmetric subset of 2x2 matrix games. Rather than incorporating a wide range of game types, we diversify the configurations and descriptions within this subset. While these games may share a common form representation, slight changes in numerical parameters or contextual framing can result in significantly different challenges for LLMs. Our experiments demonstrate that LLMs often become confused and perform poorly when only a few numbers are altered, highlighting their generalization issues in strategic reasoning.

2. Data Volume: Each game in TMGBench contains several data points, ensuring a more comprehensive evaluation. This diversity in examples helps create a more robust and stable evaluation.

3. Scenario Diversity: TMGBench introduces a wide range of scenarios within its game structures, offering diverse contexts in both numeric setups and story-based reframing. For example, existing benchmarks often focus solely on abstract payoff matrices, whereas TMGBench incorporates narrative elements to test how well LLMs adapt to different descriptions of the same strategic challenges. This diversity reflects real-world complexities, where decisions are rarely presented in isolated, numeric forms but instead in nuanced, contextual settings.

4. Game Extensibility: TMGBench enables researchers to expand the benchmark by introducing additional forms of game compositions or more complex scenarios. For example, nested games can simulate hierarchical decision-making scenarios, like auctions or negotiations. This extensibility allows TMGBench to adapt to future developments in strategic reasoning research, offering long-term utility for evaluating increasingly sophisticated LLMs.

---

Question: Differences Between Strategic Reasoning and Other Types of Reasoning.

Response: Strategic reasoning differs fundamentally from other types of reasoning, such as common-sense reasoning, in that it involves considering the actions of other participants and predicting their behavior. For example, in common-sense reasoning, the focus is on making inferences based on factual knowledge, while in strategic reasoning, the focus is on understanding and anticipating the intentions and actions of other participants.

We thank you for giving a high degree of recognition to our work and for effectively summarizing our core contributions.

One regret of this work is that we were unable to provide a more detailed background in the main text, due to the page limit, to help readers unfamiliar with game theory quickly grasp the designs of our TMGBench. We will revise the paper to include more introductory parts in the appendix to clarify some concepts.

We address your concern as follows:

Question: How to compute the standard answers to complex form games?

Response: We present three kinds of complex forms in our work: sequential, parallel, and nested. In the sequential and parallel forms, the atomic games are independent of each other, so we directly compute the standard answer for each atomic game using the conclusions from the Robinson-Goforth topology. However, in the nested form (we explore the 2-folded nested form in our work), we compute the conditional Nash equilibrium using the functions below (This means that all answers can be automatically computed based on the rules, ensuring their strict correctness):

def get_Nash_equilibrium(pA, pB, ban=None, banp=None):
    # pA: player A's payoff matrix
    # pB: player B's payoff matrix
    # ban: the restricted situation
    # banp: the restricted player
    Nash_equilibrium_choice, Nash_equilibrium_result = [], []
    for row in range(2):
        for column in range(2):
            alter_row, alter_column = 1 - row, 1 - column
            if ban is not None and (row + 1, column + 1) == ban: continue
            if (banp == "A" and (alter_row + 1, column + 1) == ban or pA[row][column] >= pA[alter_row][column]) \
            and (banp == "B" and (row + 1, alter_column + 1) == ban or pB[row][column] >= pB[row][alter_column]):
                Nash_equilibrium_choice.append((f"A{row + 1}", f"B{column + 1}"))
                Nash_equilibrium_result.append((pA[row][column], pB[row][column]))
    return Nash_equilibrium_choice, Nash_equilibrium_result
 
def calc_conditional_NEs(task_id, ban, banp):
    info = json.load(open(f"dataset/families/{task_id}.json"))
    pA, pB = info["row payoffs"], info["column payoffs"]
    return get_Nash_equilibrium(pA, pB, ban, banp)[0]
 
actual_optimal_situation_pre_task = calc_conditional_NEs(pre_task_id, restricted_situation, restricted_player)

We hope these clarifications address your questions and look forward to further discussions and receiving your valuable guidance and feedback. Thank you!