We appreciate the thoughtful comments from the reviewers and address your questions in detail below.

… what are the more advantages of modifying the search process compared with performing on the final action decision?

The key advantage of SE-MCTS over SA-MCTS is its potential for enhanced human learning and explainability. First, as the reviewer noted, SE-MCTS better aligns with the desired strength rank, which is crucial for real-world applications like learning systems for Go or chess players. Second, as SE-MCTS modifies the search tree itself, it aligns more closely with the thought process of the desired rank, opening opportunities for improved explainability to help players understand and improve their decision-making. In contrast, while SA-MCTS allows a single search to produce actions at multiple strength levels, it leads to inconsistencies – such as weak players performing advanced moves due to the randomness – resulting in unrealistic playing styles.

… with a sufficient number of simulations, SE-MCTS might become ineffective … provide a ablation study of SE-MCTS under different numbers of MCTS simulations?

Yes, this phenomenon is common in all MCTS-based programs, including SA-MCTS, due to the inherent trade-off between exploration and exploitation in UCB. As the number of simulations increases, MCTS prioritizes exploitation, causing move selections to converge. As the reviewer suggested, we conducted experiments using 800 (same as the paper), 1200, and 1600 simulations. The results below show that while accuracy slightly decreases for all MCTS-based programs as the number of simulations increases, SE-MCTS consistently outperforms MCTS and SA-MCTS, demonstrating its robustness.

In Figure 4, why the win rate of SE-MCTS so low?

We have addressed this issue in the third paragraph of subsection 4.2 in our initial version. This arises because MCTS inherently explores low-probability moves – those less likely to be selected by human players – due to its exploration mechanism. Without incorporating an additional $r_{\infty}$ rank during training, SE-MCTS struggles to handle these rarely seen actions. If the explanation in subsection 4.2 is insufficient, we welcome the reviewer's feedback for further clarification.

… have pairwise matches between SE-MCTS, SA-MCTS, and MCTS with different levels, then rank them using Elo rating system …

Running the pairwise matches like in Figure 4 is highly resource-demanding, as each match includes 250 games and requires approximately 100 GPU hours on an NVIDIA RTX A5000.

We attempt to conduct a smaller round-robin experiment during the rebuttal period. Due to the computational challenges, we selected three ranks ($r_4$, $r_6$, and $r_8$) and two representative methods (SA-MCTS and SE$_\infty$-MCTS), excluding SE-MCTS due to its ineffective strength adjustment. Each combination involves 250 games. The results below demonstrate our method's robustness across different baselines. The win rates in each cell are from the perspective of the y-axis player playing against the x-axis player. To calculate the Elo rating of each model, we initialize the rating at 1500 and iteratively update them to align the expected win rates with the observed pairwise win rates. The rightmost column of the table presents the coverage Elo ratings.

If the reviewer finds this table helpful, we are open to extending the round-robin table with more models and including the table in the appendix of our final version.

Thank you again for your valuable reviews. We are open to further discussions or conducting additional experiments.

Dear Reviewer 651t,

Thank you for your review and valuable feedback. As the discussion phase is ending, we would greatly appreciate it if you could review our rebuttal.

To further strengthen our empirical results, we have included a round-robin tournament and rank the players based on Elo rating system in the Appendix. We believe these additional experiments provide more convincing results and hope the reviewer will consider a more favorable recommendation.

We are more than willing to provide additional explanations or experiments within the remaining discussion period. Thank you again for your time and consideration.

Sincerely,

Authors

Is it possible to train and test SE-MCTS on Chess like on GO?

We initially excluded SE-MCTS on chess because it performs worse than SE$_{\infty}$-MCTS in the Go experiment. However, it is possible to train and test SE-MCTS on chess. We have updated both Figure 6 and Table 2 to include the results for SE-MCTS. For Figure 7, the experiments are still in progress due to the higher computational demands, as explained in the previous question. We will update the revision as soon as these experiments are completed.

What is the exact value of z for each rank?

The exact values of $z$ for each rank are listed below.

Which strength estimator is used in Figure 3?

The strength estimator used in Figure 3 is SE$_{\infty}$. We have revised the caption of Figure 3.

What does the strength score of vanilla MCTS and SA-MCTS look like?

To conduct this experiment, we run 100 self-play games for both vanilla MCTS and SA-MCTS at each rank. The strength scores for each model are listed below. For vanilla MCTS, its strength score is around 1, corresponding to a rank close to $r_1$. Interestingly, for SA-MCTS, the strength score looks arbitrary. This is because SA-MCTS incorporates random action selection after the MCTS search, leading to inconsistent performance (sometimes playing very weak moves and other times very strong moves). This further corroborates that SA-MCTS is less human-like and less conducive to human learning.

Which paper is the reference for SA-MCTS?

We apologize for the confusion. SA-MCTS was proposed by Wu et al. (2019). We have added the citation to the second paragraph of subsection 4.3, where SA-MCTS is first introduced.

We hope the additional experiments and revisions improve the clarity and soundness of our work. We are open to further discussions or conducting additional experiments.

We appreciate the constructive feedback and have conducted additional experiments to provide detailed answers to reviewer's questions below.

… the strength estimator method is relevant to learning to rank [1], [2] … some related background could be discussed to make the novelty more clear.

Strength estimation is similar to learning to rank but differs in key aspects. To clarify the distinction between strength estimation and learning to rank, the revision includes a discussion at the end of subsection 2.4, as follows:

"In addition, strength estimation is similar to ranking problems (Burges et al., 2005; Xia et al., 2008), but it differs in a key aspect. Ranking problems often focus on ordering items based on a single query, whereas in games, strength is assessed as overall skills across multiple positions or games. This challenge requires aggregating rankings across various scenarios to capture a player's ability."

Thank you for pointing this out.

… expand each column to a 2D map (except for MCTS). That means comparing SA-MCTS, SE-MCTS, and SE∞-MCTS for all combinations of ranks …

We understand that a round-robin table would provide more comprehensive results. However, this experiment is highly resource-demanding. In Figure 4, each cell includes 250 games to ensure statistically robust results, following the SA-MCTS paper. Running each cell requires approximately 100 GPU hours on an NVIDIA RTX A5000.

A full round-robin table, as suggested by the reviewer, comparing SE-MCTS, SA-MCTS, and SE$_{\infty}$-MCTS would involve 528 combinations (11 ranks * 3 methods * 32 / 2 matches), requiring 52,800 GPU hours or 2,200 GPU days, which is impractical. Therefore, we simply follow the SA-MCTS approach, comparing all variants against a baseline to reduce computational overhead.

Nevertheless, we attempt to conduct a smaller round-robin experiment during the rebuttal period. Due to the computational challenges, we selected three ranks ($r_4$, $r_6$, and $r_8$) and two representative methods (SA-MCTS and SE$_\infty$-MCTS), excluding SE-MCTS due to its ineffective strength adjustment. Each combination involves 250 games. The results below demonstrate our method's robustness across different baselines. The win rates in each cell are from the perspective of the y-axis player playing against the x-axis player. To calculate the Elo rating of each model, we initialize the rating at 1500 and iteratively update them to align the expected win rates with the observed pairwise win rates. The rightmost column of the table presents the coverage Elo ratings.

If the reviewer finds this table helpful, we are open to extending the round-robin table with more models and including the table in the appendix of our final version.

We thank the reviewer for providing valuable comments and address concerns as follows.

… A salient limitation is its inability to capture intransitivity … game plays by a player at rank r′>r would win against game play by a player at rank r when the lower ranked play may have played an effective exploiter strategy …

We would like to clarify that strength represents a player's overall skill or ranking within a group. For instance, on online Go platforms like FoxWeiqi, a player ranked as 2 dan reflects a higher overall strength (or win rate) compared to other 1 dan players. However, a 2 dan player may not always achieve a higher win rate against every 1 dan player due to exploiter strategies. Based on this foundation, a player with a higher strength score indicates a higher ranking or Elo rating and is highly likely (though not always guaranteed) to achieve a higher win rate against players with lower strength scores.

We understand the limitations of the Bradley-Terry model, particularly its inability to account for intransitivity. However, the Bradley-Terry model remains widely used in games and sports such as Go, chess, and football due to its simplicity and reliable strength estimation. Our experiments focus on predicting a player's ranking as it would appear on online platforms, and we do not claim to address individual transitivity cases, as it is beyond the scope of this paper. We recognize this as an interesting challenge for future research and have revised our discussion accordingly.

… exactly what "strength" means as it should depend on both the player and its opponent's strategies …

L53: "..., with higher scores indicating stronger actions": … what stronger implies here? …

As mentioned in the previous question, "strength" represents a player's overall skill or ranking within a group. Each player's strength score is independent of opponent strategies. Regarding higher scores indicating stronger actions, selecting actions with higher strength scores leads to a higher expected ranking compared to those with lower strength scores.

… the issue of reach-probability of states in defining the loss function to the SE network …

L58: … the quality of the strength adjustment of SE-MCTS in state s relies heavily on the quality of SE(s, a)? … players at a certain rank would rarely reach state s in the first place and lead to inaccurate prediction by SE-MCTS?

We kindly request clarification on the "issue of reach-probability of states in defining the loss function", as we are uncertain of its meaning. If this refers to whether all possible states can be examined during training, the answer is no. It is well-known that, when training neural networks on human games, many states rarely appear or are never reached. This phenomenon is not unique to the SE network – it also affects policy and value networks. This is why AlphaGo relies on neural network generalizability to extrapolate from seen to unseen states. Regarding whether states rarely visited by players at certain ranks could lead to inaccurate predictions by SE-MCTS: yes, this can occur, as it can with MCTS and SA-MCTS. However, we do not view this phenomenon as a significant issue.

We apologize if our explanation does not fully address the reviewer's concern and would appreciate further clarification for a more comprehensive discussion.

L173: "... aggregating using the geometric mean": it would help readers if the use of the geometric mean can be better motivated …

We kindly remind the reviewer that we have explained the reasoning for using geometric mean in the fourth paragraph in subsection 3.1. Specifically, we explain that the geometric mean ensures stable estimations and reflects rank ability across scenarios, immediately after first mentioning "aggregating all individual strengths using the geometric mean". We are willing to provide further clarification if the reviewer feels additional explanation is necessary. Please let us know if there are specific aspects you would like us to elaborate on.

L238: "we sequentially perform the softmax loss...", typo? minimise the softmax loss?

We have revised the sentence to "we sequentially minimize each softmax loss as defined by the equation 5." Thank you for pointing this out.

… optimise a rank-conditioned policy network π(⋅|s,r) with r the set of player ranks … SA-MCTS does not require ranked game-play data whereas the proposed methods do

The idea of training a rank-conditioned policy network is interesting and was widely explored nearly a decade ago (Tian & Zhu, 2015). This approach was deprecated after AlphaGo due to its ineffectiveness in distinguishing ranks with input ranking features. Furthermore, as our goal is to establish a comprehensive strength system, it is unclear how such networks could contribute to strength estimation.

Although our method requires ranked data for human-likeness, SA-MCTS, which does not rely on ranked data, produces less human-like moves. We view this as a trade-off for achieving human-likeness rather than a limitation. It is important to note that, as shown in subsection 4.4, our approach remains effective even with limited data.

• Tian, Yuandong, and Yan Zhu. "Better computer go player with neural network and long-term prediction." arXiv preprint arXiv:1511.06410 (2015).

We hope the above responses address the reviewer's concerns and strengthen the contributions of our strength system, which has significant potential for broad applications, similar to most real-world Elo rating systems, despite intransitivity issues. We kindly ask the reviewer to consider reevaluating our paper and are open to further discussions.

We thank the reviewer for finding our work interesting and providing insightful feedback. We would like to clarify the concerns raised, particularly regarding the issue of data leakage.

What is the training/testing split? … partition by players between training and testing … simply learn the preferred openings of each player and use that to predict skill …

We would like to clarify a potential misunderstanding regarding the objective of our work compared to [1]. Playing strength differs fundamentally from playing style or player-specific preferences. Strength reflects the overall skill level of a group of players with similar rankings, regardless of individual styles. For instance, two Go players of the same rank may use different strategies (e.g., offensive vs. defensive), but our approach classifies them into the same rank, while [1] distinguishes players based on style. On the other hand, if a player improves from 1 dan to 2 dan, our strength estimator classifies the player's games by rank, classifying them into 1 dan and 2 dan accordingly, while [1] classifies all games under the same individual, ignoring rank progression.

Consequently, splitting training/testing datasets by players is unnecessary, as each strength corresponds to a specific rank involving many players with diverse styles. For instance, Figure 3 shows that strength scores are nearly identical during the opening, likely because early moves in Go have limited variety, and weaker players can easily imitate stronger ones. This confirms that the models cannot rely on play-specific opening preferences to predict strength, unlike [1].

In our experiment, training and testing datasets contain entirely distinct games, though overlapping players may be present, which is acceptable. We apologize for the unclear description in the initial version and have revised the paper to explicitly describe datasets for both Go and chess in subsections 4.1 and 4.5. Thank you for pointing this out!

[1] McIlroy-Young, Reid, et al. "Detecting individual decision-making style: Exploring behavioral stylometry in chess." Advances in Neural Information Processing Systems 34 (2021): 24482-24497.

The lack of baseline or consideration … compare their results to the raw Elo estimates given by the 100/15 games …

Our paper already includes comparable baseline models (Moudˇrík & Neruda, 2016) for strength estimation, specifically $SL_{sum}$ and $SL_{vote}$, which provide direct comparisons for evaluating the accuracy of predicting playing strength from game records. Regarding the comparison to raw Elo estimates over 100/15 games, we would greatly appreciate more detailed guidance or references to related papers from the reviewer on how to conduct this experiment. We are more than willing to perform additional experiments to address your concerns.

I am also concerned by lack of code release …

We understand the importance of releasing code. As stated in our REPRODUCIBILITY STATEMENT: "The source code, along with a README file containing instructions, will be released to ensure reproducibility once this paper is accepted." However, if the reviewer finds it essential to examine the code during the review process, we are happy to upload it to an anonymous repository for review.

compare their results to any of the models of human-like play …

The primary focus of our paper is on a strength system that (a) estimates player strength from their historical game records and (b) adjusts strength to provide suitable levels, with the additional benefit of a more human-like playing style. Given these goals and objectives, we compare our strength estimator with two $SL$ networks and evaluate SE-MCTS against MCTS and SA-MCTS for strength adjustment.

We understand that several studies in chess, such as Maia (McIlroy-Young et al., 2020) and Maia-2 (Tang et al., 2024), focus on human-like play. However, these works mainly aim to achieve high accuracy in predicting human moves, without estimating the strength or verifying whether the actual playing strength aligns with specific rankings. Given the distinct goals and objectives of our work, we believe a direct comparison may not be fully applicable.

• McIlroy-Young, Reid, et al. "Aligning superhuman AI with human behavior: Chess as a model system." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.
• Tang, Zhenwei, et al. "Maia-2: A Unified Model for Human-AI Alignment in Chess." arXiv preprint arXiv:2409.20553 (2024).

I'm unclear on the reasoning for the claim on line 534 … how would AlphaZero's Elo be relevant for humans without human games?

Yes, Elo is a relative measure, and our strength estimator similarly learns relative strength scores for state-action pairs, with higher-ranking players associated with higher scores. It does not provide absolute rankings or Elo ratings. Therefore, human games are necessary during testing to calibrate these scores with specific rankings. For instance, as shown in Figure 3, a value around 1 to 2 is aligned to a rank like 9 dan using human data. On the other hand, to reduce reliance on human games during training, especially for niche games, we propose using AlphaZero self-play games to learn strength scores, which can be used to predict human ranks.

… plots are difficult to read … how the confidence intervals are calculated

We describe the method of evaluation in the third paragraph of subsection 4.2. Specifically, we randomly select $N$ games (from 1 to 100) for each $r_i$ from the query dataset. Since a single sample can only result in an accuracy of 1 (correct prediction) or 0 (incorrect prediction), we repeat this process 500 times to ensure a stable estimation. For example, when $N=5$, we randomly sample five games from the query dataset and check whether the strength estimator predicts the correct rank based on these games. This process is repeated 500 times, and the resulting prediction distribution is used to calculate the confidence intervals.

Regarding the plots, we have revised the figures to improve readability by using bold and smoothed lines and lighter colors. Please let us know if the reviewer finds the revision clearer or has additional suggestions.

We hope the above clarification resolves the reviewer's potential confusion between our work and [1], and demonstrates that our experimental results are robust and reliable. We remain open to further discussions. Thank you again for your thoughtful review.

Dear Reviewer eKyZ,

Thank you for your review and valuable feedback. As the discussion phase is ending, we would greatly appreciate it if you could review our rebuttal, particularly regarding the training and testing dataset. We believe our experimental results are robust and hope the reviewer will consider a more favorable recommendation after reviewing our rebuttal.

We are more than willing to provide additional explanations or experiments within the remaining discussion period. Thank you again for your time and consideration.

Sincerely,

Authors