Thank you for recognizing the value of our exploration of LLM agents for partially observable games and appreciating our dataset contribution. These are indeed core strengths of our work.

LLM utilization: LLMs are key components in our system to achieve the claimed performance. Our work represents an important attempt to effectively utilizing LLMs in complex two-player game environments: While opponent stat estimation could potentially be learned from data alone, LLMs provide crucial domain knowledge that would otherwise require extensive labeled data and model training. For instance, they understand type matchups, move effectiveness, and Pokemon meta-strategies with minimal prompting (as shown in Figure 5, page 6, where PokéChamp adapts to complex mechanics like Terastallization and Dynamax). The one-step lookahead mechanism combines LLM knowledge with damage calculations to determine the best action for the game by extrapolating over short horizon information.

While opponent action prediction is challenging (13-16% accuracy for min-player, Table 1, page 5), this is still valuable for narrowing the search space in minimax and substantially better than random (which would be <1% given the large action space). Even imperfect opponent modeling significantly improves overall performance, as demonstrated by our 76% win rate against the strongest LLM-based bot and 84% against the strongest heuristic bot (Table 3, page 7).

Pokemon as illustrative settings for LLM: Pokémon battle is a text-based two-player game featuring a vast pool of Pokémon, types, items, moves, and game mechanics, all described in text, resulting in a combinatorial explosion of possible configurations. It exemplifies a complex, partially observable environment with a massive state space (~10^354 for the first turn alone, page 2). This complexity makes it an excellent testbed for demonstrating how LLMs can constrain search to human-like strategies without explicit training, which is our core contribution.

ablation studies: We appreciate this suggestion. We do provide a partial ablation by comparing PokéChamp with GPT-4o versus Llama 3.1:8b (Tables 3-4, page 7-8), demonstrating that our approach works well even with smaller models. Though frontier models such as GPT-4o perform best. We also compare against the One Step Lookahead agent without the full minimax framework (Tables 3-4), showing the value of our complete approach.

Regarding the value function quality, as mentioned in our initial response, the best evaluation is performance in games. Our expert-level performance (1500 Elo rating, top 10%, Figure 1, page 1) demonstrates the effectiveness of our value function approximation. The value function is able to correctly prioritize actions that lead to winning strategies, as shown by our significantly higher win rates compared to all baselines.

the significance of 90% Elo: Pokémon Showdown is a very active competitive ladder with roughly 3000 active games at any given time, and over millions of games occurring every month. This makes the top 10% achievement substantial. Reaching this level requires deep understanding of complex strategies, team compositions, and meta-game knowledge. The 1500 Elo rating on Pokémon Showdown is considered expert-level play among the community.

We thank you for your suggestions regarding model naming conventions and will address these in our revision.

Thank you for highlighting our framework's novelty and effectiveness in integrating minimax search with LLMs, and for recognizing PokéChamp's strong performance across multiple benchmarks.

generalizability beyond Pokémon battles: Our framework naturally applies to any two-player zero-sum competitive games beyond Pokemon where minimax tree search is feasible. As described in Section 3 (page 3-4), our framework implements three LLM-powered components that can generalize to any POMG: "(1) Action sampling via LLM and tool-assisted action generation, (2) Opponent modeling through historical data and LLM-based prediction, and (3) LLM-generated value function for leaf nodes. This method implements basic components of the minimax search tree with LLM predictions that can generalize to any POMG."

The core innovation here is the replacement of traditional minimax components with LLM-based alternatives. While we demonstrate this in Pokémon, the structure is applicable to any two-player partially observable game where states can be described in natural language. Unlike RL-based methods that require extensive task-specific training, our approach only requires observation space descriptions in natural language. This makes implementation no more difficult than developing custom RL environments and reward functions.

search depth trade-offs: As detailed in Section 5.3 (page 7), we implement strict time management to comply with the 15-second per turn limit: "Our search depth is limited by the 15 second time cutoff, which is a maximum of a two step lookahead with a branching factor of 16 (4 max player actions, 4 min player actions)." We address this trade-off dynamically throughout gameplay, as "the decision time when there is only 1 pokemon left is very fast due to the limited number of states to expand." This demonstrates our system's adaptability to different computational constraints.

opponent modeling accuracy: While the raw prediction accuracies (13-16%) may appear low, this reflects the inherent challenge of predicting exact moves in a complex, partially observable environment with a large action space. Importantly, even imperfect opponent models provide significant value in minimax search by narrowing the branch exploration to more likely opponent responses. The comparative performances against state-of-the-art bots (76% win rate against the strongest LLM-based bot and 84% against the most advanced heuristic bot) demonstrate that our opponent modeling is effective despite these challenges.

We appreciate your overall positive assessment of our work and hope our clarifications address your concerns about generalizability and search depth optimization.

Thank you for your thoughtful review. We appreciate your recognition of PokéChamp's strong performance against prior bots and human players.

Elo ratings and active players: The Elo system on Pokémon Showdown (which we use for evaluation) only includes active players by design. Inactive accounts are reset to a base Elo of 1000 regularly and are not included in the percentile, ensuring our 90th percentile achievement reflects performance against the current active competitive community. As detailed in Figure 1 (page 1), PokéChamp's 1500 Elo rating places it firmly among expert players in this active ecosystem. This is comparable to how chess platforms like chess.com evaluate player performance. Pokemon Showdown has a smaller scale: roughly 3000 active games at any given time with millions of games played every month.

LLM overhead in real-time play: We address this important constraint in Section 5.3 (page 7), noting that "33% of games were lost due to time constraints." Our system implements strict time management to ensure compliance with the 15-second per turn limit, which restricts our search depth to a maximum of two-step lookahead with a branching factor of 16 (4 max player actions, 4 min player actions). In practice, the decision time varies based on game state complexity significantly faster when fewer Pokémon remain in play due to the reduced state space. We have further developed a faster version of this using better coding practices and systems techniques to increase speed that we will release with our code that achieves the same performance as mentioned in the paper.

Pokémon-specific terms: We agree that better explanation of Pokémon-specific terms would improve accessibility. Elo is a standard rating system widely used in competitive gaming beyond Pokémon (originated in chess), and we define the Abyssal bot in Section 5.1 (page 6) as "a rule-based heuristic bot used in official Pokémon games." We'll expand these definitions in the revised version.

Higher prediction accuracy at 1800+ Elo: This interesting observation likely stems from the narrower strategy space employed by elite players. At this high level, players tend to favor optimal, established strategies rather than unpredictable or sub-optimal plays, making their decisions more consistent and thus more predictable.

Damage calculator: The damage calculator example you noted (dragondarts: 161 turns) is actually a technical implementation detail. We cap the maximum turns to KO for cases where a Pokémon is effectively immune to an attack (as with Dragon-type moves against Fairy-types). By avoiding infinite values, we find that the LLM performs better at understanding and comparing all options.

Three LLM-based components: Our three LLM-based components (action sampling, opponent modeling, and value estimation) form a general framework that is naturally applicable to any two-player partially observable game, not just Pokémon. These components could be adapted to other strategic games with similar characteristics.

Remaining comments: We will address the missing ablation studies and hyperparameter sensitivity analysis description in our revised manuscript. We provide a link to a double-blind project website within our submitted project. The code link within the website is already announced “not for anonymous review” during the review period, but is available to reviewers after the review period is over. Thank you for your valuable feedback that will help us improve the final version.

More detailed model behavior analysis beyond Section C: In addition to Section C, our paper provides the following analyses on our model/method with respect to the mechanics and strategies present in this game. In Section 4.3 (page 5-6), we present benchmark puzzles specifically designed to test PokéChamp's strategic decision-making abilities with special mechanics like Terastallization and Dynamax. Figure 5 illustrates how PokéChamp demonstrates understanding of complex type matchups and strategic mechanics usage rather than exploiting defects. Additionally, we evaluate against both bots and human players (Section 5.3), showing that our approach generalizes beyond potential weaknesses in other models.

probability distribution over possible moves: We assess this in Table 1 (page 6), where we evaluate action prediction accuracy from Top-1 through Top-5 (ranking according to the action probabilities), effectively showing a distribution over moves compared to actual human actions.There may be multiple valid strategies at any point, which is why we include the Top-K metrics. The player prediction accuracy for PokéChamp varies between 26-30% for Top-1 and improves to 43-66% for Top-5 as Elo increases. We appreciate your suggestion to look at perplexity on human moves, which we can incorporate in future work.

the "black box" nature of using an LLM: We have taken great care to make our approach transparent. As illustrated in Figure 2 (page 3), we explicitly replace three components of minimax search with LLM-based generations and clearly explain the contribution of each component. The LLM serves as a prior to constrain the search space to human-like strategies (page 2), leveraging general knowledge rather than Pokemon-specific training. Our ablation studies comparing PokéChamp with Llama 3.1 versus GPT-4o (Tables 3-4) demonstrate that some performance gains come from the intrinsic model capacity. However, we also provide ablations to show that all tested models greatly benefit from our methodology (comparing PokéChamp with PokéLLMon, Tables 3-4). In fact, being able to switch in the latest frontier model or open-source model when new capabilities emerge is an important benefit of this method.

We also demonstrate robustness across different formats (Gen 8 Random Battles, Gen 9 OU) and against human players (achieving 1500 Elo, top 10% of players), showing that our approach generalizes well beyond specific test environments.

Thank you for the positive feedback on our evaluation metrics and recognition of our work's relevance to both RL and LLM integration into task planning.