KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

Thank you very much for your valuable feedback and suggestions on KORGYM. We sincerely apologize for the shortcomings in our work and have conducted a series of experiments and improvements to address these concerns:

---

Response to Weakness 1

(1) Question 1:The core claim that the benchmark is knowledge-orthogonal is not validated.

(1) Answer 1:

Although we formally defined the concept of KOR in Appendix C of our submitted paper, it did not permit empirical quantification of pre-trained knowledge leakage, thus failing to validate our core claim of knowledge orthogonality. To rectify this, we have extended the formal definition and introduced the Knowledge Leakage Coefficient, which can be empirically calculated through experiments, to further substantiate the validity of KORGYM.

1.1 Notational Definitions

For a game within KORGYM, the required reasoning information comprises:

• $K$: General background or domain‑specific knowledge acquired during pre‑training and post-training.
• $T$: Single‑turn or multi‑turn tasks that must be completed during gameplay.
• $R$: Core game‑rule information specifically designed for solving task $T$.
• $A$: The set of actions generated by the LLM to solve task $T$.
• $\rightarrow$: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.
• $S$: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:
  • $S(T \rightarrow A \mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.
  • $S(T \rightarrow A \mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.

---

1.2 Analysis of Game Redundancy

1. Knowledge‑Rule Decoupling
   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:

   $$R \perp K$$

2. Knowledge Leakage Coefficient
   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\beta$:

   $$\beta = \frac{S(T \rightarrow A \mid K)}{S(T \rightarrow A \mid R, K)}, \quad \beta \in [0,1].$$

A smaller $\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.

---

1.3 Quantitative Experiment on Knowledge Leakage

Based on the above definition, we select a subset of games from KORGYM, including some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the $\beta$:

From the above results, we can draw the following preliminary conclusions:

• Model perspective: Models such as o3‑mini and DeepSeek‑R1 exhibit relatively notable knowledge leakage, whereas other models (e.g., Doubao‑1‑5‑thinking‑pro) demonstrate comparatively lower leakage. This observation supports the view that RL processes genuinely enhance a model's reasoning capabilities, rather than merely enabling it to memorize training questions.
• Game perspective: Knowledge leakage is more prominent in classic games, while the corresponding β values for original/adapted games remain at a relatively low level. Given that more than 80% of the games in KORGYM are originally developed or adapted by our engineering team, this strongly supports the validity and reliability of KORGYM.
• Methodological perspective: We believe that the knowledge leakage quantification experiment itself offers an interesting insight and methodology for analyzing models’ genuine reasoning abilities and uncovering information contained within the pre-training data.

Thus, we have demonstrated the effectiveness of KORGYM in achieving knowledge orthogonality. Additionally, we plan to introduce more original and adapted games in the future and exclude games with higher β values to further enhance the degree of knowledge orthogonality in KORGYM.

---

Response to Weakness 2

(2) Question 2:The control, interaction, and task reasoning part of the benchmark feels weak, as it consists of relatively simple tasks in terms of control and interaction complexity.

(2) Answer 2:

Due to time constraints and considerations for overall system coherence, we regretfully did not include highly complex games, such as Minecraft, within the control, interaction, and task reasoning components. We aimed to address these reasoning dimensions by decomposing them into specific sub-capabilities, each assessed through dedicated games in KORGYM. For example:

• Minigrid evaluates the model's capability to interact with environments using tools.
• PVZ assesses the model's capacity for resource management and allocation (e.g., sunlight) within dynamic and continuous environments, as well as its multi-step strategic planning based on opponent progression.
• Minesweeper examines logical reasoning, probabilistic estimation, and risk assessment abilities under incomplete information.

By decomposing these dimensions into sub-capabilities and designing corresponding games, we provided a comprehensive evaluation framework for control, interaction, and task reasoning. Furthermore, in Response to Question 1, we quantitatively demonstrated the effectiveness of this approach through Spearman coefficient analysis.

---

Response to Question 1

(3) Question 3: Could the authors show correlation with other tasks or benchmarks which feature more difficult or specialized tasks? For example control and interaction could measure correlation with alfworld or minecraft.

(3) Answer 3:

To further analyze the correlation with other tasks or benchmarks featuring more difficult or specialized tasks, we conducted a Spearman correlation analysis with the following domain-specific benchmarks/projects:

• MathArena: Specialized assessment of model capabilities in mathematics;
• BBEH: An extended benchmark for BBH proposed by Google;
• MC Bench: Evaluates the model's ability to construct structures described in prompts using Minecraft blocks;
• OmniSpatial: A benchmark designed for comprehensive spatial reasoning evaluation;
• lmARENA-Vision: A comprehensive benchmark assessing visual reasoning capabilities;
• Ace Attorney : A project leveraging the classic game Ace Attorney to test interaction and reasoning capabilities of LLMs;
• BizFinBench: A benchmark for evaluating long-term strategic planning capabilities in financial contexts;
• SnakeBench: A project employing LLM versus LLM scenarios to evaluate competitive gaming capabilities.

The corresponding meanings of the Capability Dimension fields are as follows:

• MLR : Mathematical and Logical Reasoning
• CIR : Control Interaction Reasoning
• PR: Puzzle Reasoning
• SGR: Spatial and Geometric Reasoning
• SR: Strategic Reasoning
• MR: Multimodal Reasoning

The results of our analysis are as follows:

Thus, it can be inferred that KORGYM truly has a correlation with specialized tasks, and can reliably reflect the true reasoning capabilities of models across various dimensions.

---

Response to Question 2

(4) Question 4: How is the reasoning paradigm disabling achieved?

(4) Answer 4:

To implement the reasoning paradigm disabling, we added explicit instructions in the game rules prohibiting the use of specific reasoning paradigms as auxiliary methods. We also provided illustrative examples, such as:

"Remember, you cannot use the Code analysis method to help you solve the problem, e.g., "import math; a = 0; for i in range(...): ..." is forbidden."

Finally, we sincerely thank you once again for your valuable feedback and suggestions on KORGYM, and we look forward to further discussions with you.

We sincerely appreciate your recognition and support for our work, and we deeply apologize for the oversight regarding the Figure and related work sections in the paper.

---

Response to Weakness 1

(1) Question 1: Figure 2 is not straightforward to understand. I appreciate the effort put into it but simplifying it will help the paper.

(1) Answer 1:

We have simplified the content of the figure to enhance readability and provided additional explanations and supplements in the body text. We will update these adjustments once the paper is accepted.

Response to Weakness 2

---

(2) Question 2: For related work I believe a few citations are a must: PlanBench and the works that followed it [1, 2], as it is one of the first works to evaluate reasoning in LLMs, and GameTraversalBenchmark [3], as it evaluates spatial reasoning and is the first benchmark to evaluate LLMs on games.

(2) Answer 2:

We sincerely appreciate your insightful feedback on the shortcomings in our related work section and are grateful for recommending exceptionally engaging and inspiring studies. We found the PlanBench [1] and the accompanying Evaluation work [2], along with the GameTraversalBenchmark [3] that you suggested, to be truly fascinating and substantial contributions to the field. Reading these works significantly enriched our understanding and provided invaluable inspiration during the revision of our paper. We have eagerly integrated the relevant citations into our updated related work section, and we will reflect these enhancements comprehensively in the final version upon acceptance. Moreover, we intend to actively reference these insightful studies in our future research endeavors and surveys, thereby amplifying their visibility and sharing their valuable contributions with a broader scholarly audience.

---

Once again, thank you very much for your support and constructive suggestions on KORGYM. We look forward to further discussions with you.

First and foremost, we sincerely thank you for your recognition of KORGYM and for your valuable feedback on the analysis of game overlap and RL impact. We deeply apologize for our oversight in this regard. To quantitatively analyze the influence of game overlap and RL impact, we formulated the following formal definitions and conducted two sets of ablation experiments:

Response to Question 1 and Question 2-1

(1) Question 1:

• Is it fair to compare Duobao's performance on these games with models that have not seen such games during pretraining?
• What fraction of KORGym games (if any) overlaps with Duobao's RL training set?

(1) Answer 1:

In Doubao's technical report, only examples of Sudoku, Maze, and 21-Point were provided, without detailed data being included. Therefore, to address these two questions, we need to introduce a parameter capable of quantifying the impact of data leakage on model performance and address the problem through analytical experiments. Although we formally defined the concept of KOR in Appendix C of our submitted paper, it did not permit empirical quantification of pre-trained knowledge leakage. To rectify this, we extend the formal definition and introduce the Knowledge Leakage Coefficient, which can be empirically calculated through experiments, to further substantiate the validity of KORGYM.

---

1.1 Notational Definitions

For a game within KORGYM, the required reasoning information comprises:

• $K$: General background or domain‑specific knowledge acquired during pre‑training and post-training.
• $T$: Single‑turn or multi‑turn tasks that must be completed during gameplay.
• $R$: Core game‑rule information specifically designed for solving task $T$.
• $A$: The set of actions generated by the LLM to solve task $T$.
• $\rightarrow$: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.
• $S$: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:
  • $S(T \rightarrow A \mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.
  • $S(T \rightarrow A \mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.

---

1.2 Analysis of Game Redundancy

1. Knowledge‑Rule Decoupling
   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:

   $$R \perp K$$

2. Knowledge Leakage Coefficient
   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\beta$:

   $$\beta = 1 - \frac{S(T \rightarrow A \mid R, K) - S(T \rightarrow A \mid K)}{S(T \rightarrow A \mid R, K)} = \frac{S(T \rightarrow A \mid K)}{S(T \rightarrow A \mid R, K)}, \quad \beta \in [0,1].$$

• A smaller $\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.
• Conversely, a larger $\beta$ indicates significant overlap between the LLM’s pre-trained knowledge and the game rules, allowing the LLM to complete the game task and achieve optimal scores even without explicit game rules provided in the prompt.

---

1.3 Quantitative Experiment on Knowledge Leakage

Based on the above definition, we select a subset of games from KORGYM, including the games mentioned in Doubao's technical report(Sudoku and Maze), some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the Knowledge Impact Coefficient $\beta$. The results are presented below.

From the above results, we can draw the following preliminary conclusions:

• For Question 1, we observe that: - For games explicitly trained on by Doubao-1-5-thinking-pro (Sudoku and Maze), the β values are indeed higher but remain at a moderate level among the sampled models, which is within an acceptable range; - For classic games (Snake and 8-Puzzle), while Doubao-1-5-thinking-pro demonstrates strong performance (ranking 3rd on Snake and 4th on 8-Puzzle among all models in the main experiments), its β values remain the lowest among the sampled models. This observation supports the view that reinforcement learning (RL) processes genuinely enhance a model’s reasoning capabilities rather than simply enabling it to memorize training questions or regurgitate pretraining knowledge, thus demonstrating strong generalization. - For games originally developed or adapted by our engineering team, all models exhibit very low β values, and given that at least 80% of KORGYM games are original or adapted, this further validates the effectiveness of KORGYM.

• For Question 2-1, we confirm that the original/adapted games (80%+) enable Doubao-1-5-thinking-pro to maintain relatively low β values. Even for classic games, the β values remain at a low level. Hence, we estimate that the overlap between games and Doubao’s RL training set does not exceed 15%. In future work, we will leverage this methodology for further experiments to determine a more precise overlap ratio.

• From a methodological perspective, we believe that this Knowledge Leakage Coefficient evaluation framework can effectively reveal the proportion of data types used in the LLM training process. For instance, although the training data composition is not explicitly disclosed in the reports, our experimental results suggest that o3-mini likely involved Snake-related data and Doubao-1-5-thinking-pro likely involved Sokoban-related data. Moving forward, we plan to further develop this methodology to uncover additional interesting insights.

---

Response to Question 2-2

(2) Question 2: Whether there is an ablation isolating the RL impact from other improvements in the "thinking" model variant?

(2) Answer 2:

We sincerely apologize for the oversight regarding the ablation analysis on RL impact from other improvements in the "thinking" model variant. Here, we provide additional supplementary analysis. To isolate the effect of RL training from other structural or training environment factors contributing to the performance difference from the non-thinking to the thinking variants of the model, we selected three LLMs that offer both non-thinking and thinking modes. For Gemini-2.0-Flash and Claude-3.7, the results under both modes are already reported in Table 1 and Figure 5 of our paper. The final experimental results are as follows:

From these results, we observe that the thinking variants of the same model consistently outperform their non-thinking counterparts, which supports the validity of the performance improvements attributed to the transition from the non-thinking to the thinking variant of the model.

---

Finally, we sincerely thank you for your valuable feedback on KORGYM again, and we look forward to further discussions with you.

Thank you for your insightful suggestions. We apologize for not clearly addressing this aspect in our KORGYM submission. Below, we will elaborate on the measures taken in the design of KORGYM to minimize overlap with pre-training data. In addition, building on the formal definition of Knowledge Orthogonal Reasoning (KOR) in our paper, we will introduce our quantifiable metrics to assess knowledge leakage and select exemplar games within KORGYM to empirically analyze the extent of such leakage.

Response to Weakness

(1) Question1: Potential Domain and Task Distribution Bias: Although the knowledge-orthogonality concept aims to decouple evaluation from prior knowledge, the game/task design process is not fully transparent regarding avoidance of distributional overlap with LLM pre-training corpora.

(1) Answer1:

In the design phase of KORGYM, significant efforts were made to minimize potential knowledge leakage:

• Unlike traditional reasoning benchmarks, KORGYM is designed as dynamically generated games. The nearly infinite decision space and symbolic diversity significantly reduce the likelihood of overlap with pre-training data.
• Games, especially multi-turn scenarios, require long-term strategic planning, which is typically underrepresented in pre-training corpora.
• During the selection and creation of games, we developed numerous original games and modified traditional games extensively to further prevent overlap with pre-training data.

---

(2) Question2: The paper does not concretely quantify the degree of knowledge leakage, which may influence some of the comparative results or limit conclusions about pure reasoning.

(2) Answer2：

Thank you for your valuable suggestion and we will provide a formal definition of the degree of knowledge leakage and supplement it with quantitative experiments:

2.1 Formal Definition of the Knowledge Leakage in KORGYM

In Appendix C of our submitted paper, we formally defined the concept of "Knowledge Orthogonal Reasoning" (KOR). However, it does not allow for empirical quantification of the extent of pre-trained knowledge leakage. This limitation arises from the inherent impossibility of fully eliminating the influence of pre-trained knowledge $K$ on the reasoning process. Additionally, the modeling scenario of KORBench[1], structured as a question-and-answer format, differs significantly from that of KORGYM. Consequently, we extend the formal definition originally proposed in KORBench and present an experimental approach to quantitatively measure the degree of pre-trained knowledge leakage.

[1] Ma et al. KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks. arXiv.2410.06526.

---

2.2 Notational Definitions

For a game within KORGYM, the required reasoning information comprises:

• $K$: General background or domain‑specific knowledge acquired during pre‑training and post-training.
• $T$: Single‑turn or multi‑turn tasks that must be completed during gameplay.
• $R$: Core game‑rule information specifically designed for solving task $T$.
• $A$: The set of actions generated by the LLM to solve task $T$.
• $\rightarrow$: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.
• $S$: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:
  • $S(T \rightarrow A \mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.
  • $S(T \rightarrow A \mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.

---

2.3 Analysis of Game Redundancy

1. Knowledge‑Rule Decoupling
   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:

   $$R \perp K$$

2. Knowledge Leakage Coefficient
   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\beta$:

   $$\beta = 1 - \frac{S(T \rightarrow A \mid R, K) - S(T \rightarrow A \mid K)}{S(T \rightarrow A \mid R, K)} = \frac{S(T \rightarrow A \mid K)}{S(T \rightarrow A \mid R, K)}, \quad \beta \in [0,1].$$

• A smaller $\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.
• Conversely, a larger $\beta$ indicates significant overlap between the LLM’s pre-trained knowledge and the game rules, allowing the LLM to complete the game task and achieve optimal scores even without explicit game rules provided in the prompt.

---

2.4 Quantitative Experiment on Knowledge Leakage

Based on the above definition, we select a subset of games from KORGYM, including some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the Knowledge Impact Coefficient $\beta$. The results are presented below.

From the above results, we can draw the following preliminary conclusions:

• Model perspective: Models such as o3‑mini and DeepSeek‑R1 exhibit relatively notable knowledge leakage, whereas other models (e.g., Doubao‑1‑5‑thinking‑pro) demonstrate comparatively lower leakage. This observation indirectly supports the view that reinforcement learning (RL) processes genuinely enhance a model's reasoning capabilities, rather than merely enabling it to memorize training questions or regurgitate knowledge from pretraining.
• Game perspective: Knowledge leakage is more prominent in classic games, while the corresponding β values for original/adapted games remain at a relatively low level. Given that more than 80% of the games in KORGYM are originally developed or adapted by our engineering team, this strongly supports the validity and reliability of KORGYM.
• Methodological perspective: We believe that the knowledge leakage quantification experiment itself offers an interesting insight and methodology for analyzing models’ genuine reasoning abilities and uncovering information contained within the pre-training data. Future work could extend this approach to conduct deeper analyses and evaluations of reasoning capabilities.

Once again, we sincerely thank you for your valuable feedback on KORGYM. Your comments have provided insightful inspiration and an important reminder. Moving forward, we plan to further increase the proportion of original/adapted games in KORGYM to more accurately capture models' genuine reasoning capabilities.