We sincerely appreciate your thoughtful and detailed review of our work. It means a great deal to us that you recognize SPIN-Bench contribution to strategic planning and social reasoning.

Limited Complexity on "Social Intelligence"

We are in full agreement that it is a deeply multifaceted construct, and that any single evaluation will inevitably focus on specific facets.  To illustrate this breadth, Social Intelligence, Goleman 2006[1] outlines two broad categories: 'social awareness,' defined as “what we sense about others,” and 'social facility,' described as “what we then do with that awareness”. Each of these categories encompasses more granular competencies; for instance, social awareness includes aspects like primal empathy and attunement, while social facility involves skills such as persuasion and self-presentation.

Within this framework, SPIN-Bench is intentionally scoped to address one key aspect from each of Goleman's broad categories: Theory of Mind, which aligns with 'social awareness', and negotiation, which falls under 'social facility'. With SPIN-Bench, we attempted to provide analysis on these two specific components of social intelligence. We chose this focus because Theory of Mind and negotiation are not only critical components of social reasoning but also represent areas where current AI models face significant challenges.

We share the reviewer's enthusiasm for further discussions and iterations that delve deeper into the broader dimensions of social intelligence. We propose several compelling directions for future work:

1. Deepening Theory of Mind: Currently, we only provide information about current game states. Exploring scenarios where agents play multiple rounds in Hanabi or Diplomacy could offer AI agents richer contexts for modeling evolving collaborator or competitor intentions, behaviors, and knowledge states.
2. Broadening Social Intelligence Tasks: Extending evaluations to include tasks like maintaining alliances in Diplomacy and enhancing intention prediction would further enrich our understanding of AI social capabilities.

Human Performance Comparisons in Diplomacy

For the first few tasks in our benchmark (competitive, cooperative games), we have found suitable solver-based or human baselines to compare the gap between LLMs and human/solver-level players. However, Diplomacy has no known “solver” or exhaustive engine capable of optimal play, nor a standardized, scalable platform that supports large‐scale human‐vs‐LLM tournaments. We acknowledge the fact that incorporating both human players and LLM players, and comparing their social behaviors, can offer more interesting insights. To address this limitation, we are working on a unified multiplayer social game interface that will allow public users to upload their favorite models or participate in games themselves. After the interface is released, we anticipate collecting data among human players and LLMs, and releasing that data to support open research in socially intelligent LLM.

Neutral powers' impact in Diplomacy

Thank you for the question. In our simplified “all‐neutral” Diplomacy setup (where every other power is a neutral dummy), the game’s complexity is dramatically reduced. Neutral powers in our evaluation act as passive placeholders: they never issue any orders or pursue their own strategies. Instead, whenever an active player captures one of a neutral power’s supply centers, the neutral player simply disbands one of its units at random. In effect, the neutral powers cannot resist or negotiate—they exist only to provide additional territory for an active agent to claim. So, the same LLM may get higher scores in an all-neutral game, compared to the game where other players are also LLMs/humans.

We use this “all‐neutral” configuration purely in our first‐round evaluation to isolate each model’s basic game proficiency. By setting all other powers “neutral/dummy” will make the game super simple and easy. If a player can understand the game rules and basic actions, they can win the game easily. Once we identify models that demonstrate reliable core mechanics under these simplified conditions, we advance them to a second evaluation stage involving full multiplayer scenarios where social reasoning, negotiation dynamics, and long‐term planning become critical. In short, the neutral‐power setup serves as a stress test for rule comprehension and basic strategic reasoning; any performance gains here do not carry over obviously into the fully interactive, adversarial Diplomacy environment because the social and cooperative dimensions are absent.

Reference:

[1] Goleman, D. (2006). Social intelligence: The new science of human relationships.

Thank you for your comprehensive review of our paper.

In-depth Discussion of Theory of Mind Capabilities

You are absolutely correct that even no perspective taking in negotiation messages can still reflect theory of mind capabilities from another angle. Below, we address your concern in two parts:

Why can high values of perspective taking show advanced theory of mind capabilities?

In our metric, we assign “1” whenever an agent explicitly references or empathizes with the recipient’s perspective—an unambiguous, observable signal that the agent is actively modeling the other party’s mental state. Explicit perspective-taking often leads to more persuasive proposals in negotiation, which reflects a deeper level of social intelligence. To verify that these explicit signals truly correspond to accurate mental-state reasoning, we conducted an additional experiment:

For every message flagged as containing perspective-taking, we asked LLM annotators to use the full game context and subsequent actions, to judge whether the agent’s interpretation of the opponent’s perspective was correct. Across all sampled messages, the model achieved a perspective-taking correctness rate between 88% and 100%. This high accuracy indicates that when the agent explicitly adopts the other’s viewpoint, it is not merely using generic or templated language but is genuinely inferring the opponent’s goals or intentions. Consequently, a high count of such explicit references provides strong evidence of advanced ToM reasoning.

Low values could also be a part of the theory of mind capabilities

You are correct that even infrequent perspective-taking can reveal ToM capabilities, especially in uncertain or adversarial settings. While our current work focuses on explicit, message-level signals, we have designed a preliminary approach for evaluating implicit ToM:

When agents generate an action (e.g., a move or offer), we also ask them to output their own negotiation strategy alongside a prediction of their opponent’s likely strategy. After the game concludes, we compare each agent’s predicted opponent strategy with the opponent’s self-generated strategy. A high alignment between these two suggests that the agent was reasoning about the opponent’s mental state, even if it did not verbalize that reasoning. In our current evaluation, we can see a very high alignment ratio between LLMs’ generated strategies and their actual actions, so it provides plausibility support for the above approach of implicit intention prediction.

We recognize that ToM and social intelligence are multifaceted and cannot be fully captured by a single binary metric. Our explicit perspective-taking score represents one clear, verifiable facet of ToM. In future work, we plan to expand on implicit evaluation methods and make the multi-agent game platform more complete to encourage the community to explore additional lenses for measuring social reasoning under uncertainty.

o1 Score on Competitive Games

We believe the key reason is that classical planning, collaborative games, and competitive games each demand quite different skill sets, so strengths in one category do not always translate directly to another. Classical planning and collaborative games primarily test an agent’s ability to perform deterministic, multi‐step reasoning and to coordinate openly with partners under partial information. By contrast, purely adversarial or mixed‐motivation environments require not only planning but also anticipating and in many cases misleading opponents. In SPIN-Bench, these three game categories are deliberately chosen to probe distinct facets of strategic and social intelligence.

In competitive games, we can see o1 still ranks at or near the top among all evaluated models. o1’s core planning ability remains very capable, but it does not pull as far ahead in contexts where opponent modeling and deception handling are critical. This is also reflected in our negotiation-enabled Diplomacy setting: we observe o1 and o1-preview models a large performance drop in a more interactive strategic environment (Section 4.4.2).

Thank you for your insightful review. We have addressed your comments in our response above. As the discussion deadline approaches, please let us know if you have any further questions or topics you’d like to discuss. We will be available over the next few days to assist.

Dear reviewer CDqy,

Thank you for your positive feedback and thoughtful review. The deadline is approaching soon, and we want to ensure we have addressed your concerns and incorporated your feedback into our final version. We remain fully open to any additional discussion before the review period ends and would greatly value your perspective on these findings. Thank you again for your thorough and constructive review.

We appreciate your thorough review and constructive comments on our work.

Missing References

We will add these missing references in our paper.

1. arxiv:2310.11667: authors present an environment for simulating social interactions and evaluation. They also showed GPT-4 could automate agent evaluation, which indirectly supports the rationality of our using LLM for social reasoning annotation. We focus specifically on strategic games as a natural extension for multi-agent tasks.
2. arxiv:2404.11023: authors outline several obstacles to advancing social intelligence. SPIN-Bench concretizes these abstract challenges by placing them in strategic-game environments. By doing so, our benchmark enables fine-grained analysis of how LLMs handle these very challenges.
3. arxiv:2403.14659: authors develop a data infrastructure for social AI tasks, emphasizing the need for more interactive, multi-agent data. Ours complement this infrastructure by supplying strategic, game-based interaction logs and metrics.
4. arxiv:2412.12175: authors propose a framework for generating diverse, challenging Theory-of-Mind data through stories and question-answering. We extend their ideas by embedding ToM challenges into strategic or competitive contexts.

Human Annotation on LLM-assisted Metrics

We agree that relying solely on LLM-generated labels can introduce bias and noise, so we conducted human validation study within our research team:

We provided each annotator with the same information that the LLM used:

1. Negotiation message
2. Message sender and recipient
3. Current game state
4. Issued orders after the negotiation phase
5. Sender’s negotiation strategy

We randomly sampled 5% of negotiation messages. Three trained human annotators independently labeled each message for the metrics defined in Appendix B.5.2.

We used majority vote among the three annotators to produce the human label. We measured inter-annotator agreement as the percentage of cases where all three annotators assigned the same label. The three annotators agreed 87% of the time—indicating a strong consistency. So this level of concordance suggests that our LLM-based approach provides reliable estimates of the negotiation behaviors.

Actionable insights

First, across cooperative and strategic games, we observed that many agents struggle to infer their partners’ or opponents’ hidden intentions. To address this, future training pipelines could explicitly model others' intentions alongside standard actions. Supervising a model to predict hidden intentions should strengthen its ability to perform theory‐of‐mind reasoning.

Second, our experiments highlight that long‐horizon planning remains a bottleneck in games. Agents should learn to stick to their strategy, or dynamically change their strategy based on opponents' moves. Future works can further model their strategy during the game play, or design reward models for their planning behavior that explicitly reward both strategic consistency and adaptability.

Third, a promising direction is cross-game curriculum learning, for example, PDDL tasks teach a model to search deterministic state spaces, while Diplomacy forces the model to infer hidden intentions. Combining these skills can help a model better balance deterministic planning and belief inference.

Contribution to the community

A Unified, User-Friendly Interface and Game Engine

We have developed an open-source framework that lets users run all tasks locally. Through a single configuration script, researchers can launch and evaluate tournaments between different LLM agents, solver-level engines, and even human players across all supported games. And we have also been actively expanding this platform by adding new games over time.

A Reasoning Data Collection Toolkit

Our framework not only records each agent’s actions but also captures the their natural-language reasoning trace. Researchers can use this toolkit to generate game reasoning data. We think those reasoning data will serve as a valuable supplement for further training on those well-performed game trajectories. We collected 10k Connect Four samples. We then fine-tuned R1-Distill-Qwen-1.5B under two settings: (1) using both action and reasoning as supervision vs. (2) using only the action labels. Samples are selected by the quality of the action.

On our 1k testset, the model trained with reasoning achieved a best-move accuracy of 46.3%, compared to 40.5% when trained on actions alone. This result demonstrates that reasoning traces can improve performance. By releasing this data collection pipeline, we hope to catalyze future work on leveraging rich reasoning data for improved strategic capabilities.

LLM Gaming Competition Platform

It offers a competition platform where users can upload their own models to participate. We will also host a large-scale social-game competition at an upcoming ML conference, and all the collected data will be public for future research.

Thank you for addressing my concerns. Including the missing references and expanding the discussion of annotator agreement and validation will strengthen the paper's longer-term validity and contribution. I have updated my score.

Thank you for your thoughtful review and positive feedback on our work. We are especially encouraged by your recognition of our integrated tasks, extensive experimentation, and insights into long-horizon reasoning and multi-agent coordination challenges. We will promptly release the benchmark, code, collected data, and other resources with clear documentation, with the hope that they will facilitate further advances in strategic planning and social intelligence research within the LLM community.

Thank you for your detailed review and valuable insight on your questions.

Additional Details of LLM-assisted Annotations

We acknowledge that relying solely on LLM-generated annotations can introduce noise or skewed judgments. Unfortunately, these social reasoning capabilities are challenging to evaluate through rule-based methods and are not scalable via purely manual annotation. LLM-based annotation is a common approach for large-scale analysis. [1]

We investigated LLM’s annotation alignment with human annotators in Appendix C.3.3. We also compared Claude 3.7 Sonnet with o1 model for annotating a subset of negotiation messages. Claude 3.7 Sonnet demonstrated higher alignment with human annotators.

Here's our human annotation design: To ensure consistency between human and LLM annotations, we provided each annotator with the same information that the LLM used:

1. Negotiation message
2. Message sender and recipient
3. Current game state (exactly what a player sees on the board)
4. Issued orders from the immediately preceding negotiation phase (to show whether prior proposals were acted upon)
5. Sender’s negotiation strategy

We randomly sampled 5% of all negotiation messages. Then, three trained human annotators (members of our research team) independently labeled each message for the metrics defined in Appendix B.5.2.

For each metric, we used majority vote among the three annotators to produce the human label. We measured inter-annotator agreement as the percentage of cases where all three annotators assigned the same binary label. Across all metrics, the three annotators agreed 87% of the time—indicating strong consistency for this classification task. We then compared Claude 3.7 Sonnet’s labels against the majority-vote human labels. Claude’s agreement rate with human annotators ranged from 91% to 100% across different metrics. So this level of concordance suggests that our LLM-based approach provides reliable estimates of the negotiation behaviors in Diplomacy.

Looking ahead, we plan to further reduce potential annotation biases by adopting an ensemble of LLMs. We hope this strategy will yield more robust and multifaceted social intelligence evaluation.

Planning vs. Intention Prediction

In our analysis within SPIN-Bench, we observed notable performance discrepancies between various LLMs, particularly regarding intention prediction and planning capabilities. We conducted a detailed analysis on a subset of game trajectories, relying primarily on human empirical evaluations. We share an interesting finding here:

Specifically, we observed distinctive failure modes between o3-mini and Claude 3.5 Sonnet in the context of Hanabi—a cooperative game that requires planning, constrained communication, and intention prediction. Although both models achieved similar overall scores, their limitations differed significantly.

o3-mini excelled at generating coherent planning strategies and providing effective hints but frequently struggled with accurately interpreting its partner’s intentions (see example below in Detailed Example). In contrast, Claude 3.5 Sonnet displayed the opposite pattern: it more reliably interpreted hint signals (demonstrating stronger intention inference) but occasionally produced misleading or implausible hints.

These observations indicate a clear divergence in their respective planning and intention prediction abilities. In our classical planning metric, o3-mini achieved an accuracy of 51.25%, significantly outperforming Claude 3.5 Sonnet’s 20.55%. This demonstrates that even though o3-mini excels at generating a coherent multi-step strategy, it struggles with forward-looking intention inference. In other words, strong planning does not guarantee accurate intention prediction, and vice versa. This indicates that maybe a more balanced training approach that addresses both skill sets is needed in the future.

Detailed Example

A concrete example is provided in paper's trajectory viewer. For o3-mini in two-player mode (steps 3–8), player 1 effectively hint a blue card for player 2’s safe play; however, player 2 fails to correctly interpret this strategic hint. Also in later cases, player 2 can give a similar hint, but player 1 failed to interpret the hint.

Conversely, at step 14 in the two-player game between Claude 3.5 Sonnet, player 2 incorrectly hints a previously played white card to player 1, but that information is actually misleading for player 1. Nonetheless, it successfully interprets other useful hints in subsequent turns, demonstrating stronger intention recognition capabilities despite occasional strategic miscalculations.

This nuanced observation underscores the importance of distinguishing between planning capability and intention inference during evaluation.

Reference:

[1] Zhou, X.,etc SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. In Proceedings of the (ICLR).

Thank you for your insightful review. We have addressed your comments in our response above. As the discussion deadline approaches, please let us know if you have any further questions or topics you’d like to discuss. We will be available over the next few days to assist.

Dear Reviewer nB76,

As the discussion period draws to a close, we would like to extend our sincere gratitude for your insightful feedback and valuable contributions. In response to your comments, we have conducted additional experiments and compiled detailed results to address each of your concerns. Please let us know if any questions remain or if you would like further clarification. We remain fully open for any additional discussion before the review period ends.

Thank you once again for your time and effort.