LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models

Thank you for your valuable feedback and encouraging comments. We would like to take this opportunity to clarify the raised questions.

Response To Question "How scenarios uniquely test the reasoning capabilities of LLMs?"

We acknowledge the necessity for a detailed discussion regarding specific tasks within each scenario, and correspondingly discussed the advantages of LLMs in various scenarios, such as Societal Simulation, where "LLMs are utilized for the modeling and prediction of human behavior within intricate societal frameworks", and "LLMs are applied to understand and predict the outcomes of economic decisions, simulating scenarios like market competition, resource allocation, and investment strategies" for Economic Simulation.

The further dicussion will cover the high-level capabilities of LLM required for different scenarios, such as planning, problem decomposition, and world modeling [1], and it will facilitate more accurate assessments of LLM strengths and limitations.

Response To Criticism "The survey could benefit from a deeper analysis of the related works rather than just listing them."

We appreciate your suggestion on a deeper analysis of the related works. We have made efforts in this regard. For instance, in Section 4 (Methods), we categorized the methods based on their underlying motivations and implementations. Additionally, we utilized a timeline (Figure 4) to delineate the developmental roadmap of these methods, providing readers with a clearer developmental trajectory.

Response To Concern "It would be beneficial to include the authors' perspectives on practical approaches to realizing them"

As mentioned in 6.2, the establishment of a universal and standardized benchmark is highly needed. Such a benchmark should adhere to three principles: Well-Controlled Environments, Reasonable Evaluation, and Scalability Across Scenarios. From a practical standpoint, we consider Welfare Diplomacy [2] a good initiative. This game, which incorporates cooperative and deceptive elements typical of multi-agent systems, stems from classic diplomacy games [3]. However, there is a need for the development of more quantitative metrics to evaluate the strategic reasoning abilities in such scenarios.

[1] Reasoning with Language Model is Planning with World Model.

[2] Welfare diplomacy: Benchmarking language model cooperation.

[3] Human-level play in the game of diplomacy by combining language models with strategic reasoning

Thank you for affirming our work and providing valuable suggestions regarding the existing issues.

Response to the Concern "The lack of discussion on the applicable conditions of metrics and explanation for the selection of metrics".

The selection of these metrics is intricately linked to specific contextual scenarios. For instance, in survival games, metrics such as agent survival rates are pertinent, whereas, in games without elimination, metrics like win rates or rewards are more relevant. Additionally, different scenarios necessitate distinct focuses on the reasoning capabilities of the employed strategies.

Regarding the rationale behind selecting these quantitative metrics, we aimed at two primary objectives:

1. Rationality Assessment of LLMs: These metrics evaluate the rationality of LLMs, highlighting the disparity between LLMs and traditional methods like reinforcement learning. This comparison helps showcase the differences in rationality between LLMs and conventional approaches. Moreover, these metrics provide a benchmark for comparing the relative rationality of different methods.
2. Enhanced Transparency in Decision-Making: As discussed in our introduction, LLMs aim to make decision-making more transparent and interpretable. Therefore, these metrics analyze the performance of LLMs in the process of strategic reasoning.

We greatly appreciate your additional references, which undoubtedly enhance the quality of our work.

Once again, thank you for your valuable feedback and constructive recommendations.

Regarding your main concern, "whether a survey paper fits within the COLM technical track," we have inquired with the Program Chairs:

Q: As this is the first COLM, we seek your clear guidance on whether a survey paper is appropriate for the COLM track. … If the survey paper will naturally be rejected by COLM2024, we will withdraw the submission accordingly.

A: Please address this in your rebuttal. We don't have a specific policy about survey papers, and they will be judged according to the contribution, similar to conferences like ICLR/NeurIPS and ACL venues. -Yoav

As noted by the PC, survey papers are also valued as a form of contribution by COLM'24, similar to other top-tier conferences like ACL, which even categorizes surveys as a distinct type of contribution on its submission site. One can find survey papers such as "Reasoning with Language Model Prompting: A Survey" accepted by prestigious venues like ACL/EMNLP/NAACL. Unlike well-defined reasoning tasks, Strategic Reasoning with LLMs remains unorganized and lacks systematic consolidation despite numerous related efforts. Our paper precisely provides a timely and comprehensive review, contributing to the community by standardizing research efforts in this field.

Response to "What major challenges and opportunities exist in applying LMs to strategic reasoning scenarios, and which areas are receiving more or less attention in LLM research?"

In our paper, we highlighted three key benefits of utilizing LLMs in the introduction. Regarding the replacement of traditional methods, we acknowledge that LLMs have not yet reached a stage where they can outperform state-of-the-art RL models in domains such as poker or Go. However, LLMs have shown advantages in novel research areas like conversational games (e.g., Diplomacy), where traditional RL methods may fall short. LLMs have broadened the application scenarios of strategic reasoning, presenting new challenges and opportunities.

We aim to offer significant value through comprehensive synthesis and critical analysis of existing works, alongside our in-depth insights on this direction. To address your concern about the depth of discussion, we will expand the "scenarios" section and delve into specific challenges and opportunities across scenarios, explaining the motivation for using LLMs and assessing the prospects for success.

Thank you for your insightful suggestions. We look forward to further discussions with you.

Thank you for your careful review and valuable comments on our work.

Response to Question "How could we define any proposed benchmark as unified and standard enough so that every related research should evaluate with?"

We believe a unified benchmark should embody three key characteristics:

1. Well-Controlled Environments: The benchmark should operate within controlled environments with minimal randomness, facilitating reproducibility and ease of experimentation.
2. Reasonable (Objective or Subjective) Evaluation: As discussed in section 5, a unified benchmark for strategic reasoning with Large Language Models (LLMs) should be capable of objectively or subjectively assessing their strategic reasoning abilities. This includes evaluating long-term planning capabilities and dynamic adaptability, among other facets of strategic reasoning.
3. Scalability Across Scenarios: The benchmark should be adaptable to a broad spectrum of strategic reasoning scenarios, enabling researchers to explore diverse applications and challenges.

As mentioned in our paper, some benchmarks derived from classical game-theoretical problems for strategic reasoning lack higher-level abstraction, which limits their scalability and applicability to real-world applications. One feasible approach involves building upon existing RL-based environments, such as Robot Warehouse (RWARE)[1] and the StarCraft Multi-Agent Challenge [2]. These environments offer well-defined and controllable settings, laying a solid foundation for benchmarking strategic reasoning abilities. However, further refinement is necessary to ensure the benchmark meets the criteria of reasonable and comprehensive evaluation and scalability.

By leveraging established environments as starting points, we can enhance the benchmark's capacity to assess LLMs' strategic reasoning capabilities while fostering innovation and advancement in the field. Thank you for prompting this critical discussion, and we welcome any additional insights or suggestions you may have.

[1] F. Christianos, L. Schäfer, and S. V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. 2020.

[2] M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.