We appreciate your review and constructive feedback, and we are grateful that you consider our writing & presentation clear, and our evaluation rigorous.

The design of ALLIE is akin to AlphaGo without the self-play phase

The key contribution of our work does not lie in the use of AlphaGo-style MCTS itself, but rather, we show that the use of adaptive search enables chess gameplay at a Grandmaster level (>2500 Elo) without loss of “human-likeness” — which to the best of our knowledge is not possible with prior techniques.

Like the reviewer pointed out, our work 1) shows machines can mimic the depth of human reasoning by learning to predict the think time of humans, and 2) proposes an time-adaptive MCTS procedure that enables strong gameplay and skill calibration, which are quite different from the AlphaGo line of research.

The authors do not explain why adaptive search attains a better skill calibration error even against mid-level opponents.

Adaptive search leads to better skill calibration than the no-search baseline (Allie-policy) because search itself is essential to avoid blunders. Even against players with Elo ~ 1700, baselines that don’t do search underperform humans by > 50 Elo points, and both regular and adaptive search help close this gap and leads to much better calibration.

As the authors point out, the users knew they were playing against an AI before they filled out the survey. Therefore, the human alignment result from the user study might not be as objective as if they conducted a Turing test.

We acknowledge that the knowledge of playing against a bot may subtly bias behavior, but we don’t expect drastic differences in playing strength or style. In addition, all systems (Maia, Allie-Policy, Allie-Adaptive-Search, etc.) are deployed identically, which should minimize the impact of this knowledge on the validity of our findings. We further note that the terms of service of Lichess (the website we deployed our chess bot on) require disclosure if playing using an engine, and it would be simply impossible to evaluate against thousands of human players if we didn’t deploy our model on Lichess. We would love to see a Chess Turing test study done in the future, which evaluates how well Allie and other chessbots can fool human players.

It would better support the superiority of adaptive search in skill calibration if ALLIE with adaptive search were compared against an AlphaZero-like agent, where the skill level is adjusted using a temperature parameter.

We certainly could “water-down” any strong chess AI like AlphaZero to achieve perfect skill calibration (e.g., with a temperature parameter, or making it play a random move with some probability p depending on the strength level), but we argue such skill calibration is uninteresting in itself. The core challenge that we try to solve in this paper is to train human-like chess AI models that are well-calibrated to actual humans at varying strength levels, and this is not possible through sampling from chess engines such as AlphaZero.

For example, the Maia paper [1] reports direct comparisons against Stockfish and Leela (an open source version of AlphaZero), and shows that neither engine matches human behavior well (Figure 4).

[1] Aligning Superhuman AI with Human Behavior: Chess as a Model System. Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson.

Is the bin range in Figure 5 set to 200? The datasets are separated every 100 Elo ratings. Why does the bin range differ?

We chose the bins so we had at least 50 human games in each bucket. Using a small bin range would lead to fewer games in each bucket and more noisy system performance estimates. This choice is independent from how we processed the pre-training dataset.

the explanation of why we cannot use GPT-4 for evaluation is unclear.

Getting an OpenAI model to play chess requires a non-chat model (specifically, gpt-3.5-turbo-instruct, the latest non-chat model from OpenAI) for feeding in a chess game record (in PGN format), and converting next move prediction to text generation. Assistant models such as GPT-4o cannot be used in this way, and does not seem to exhibit the same ability to play chess.

How to determine the constant c in the time-adaptive MCTS procedure? It seems it is not explicitly stated in the main text.

We follow AlphaZero’s original implementation and use c=1.25. We have clarified this choice in the revision.

For the resignation conditions, you consider both the resignation token and values. What if it only thinks of resign tokens? Would the result be better or worse than TPR of 86.4%?

The true positive rate would be strictly higher, since Allie only resigns if both of the following conditions are true: [RESIGN] is the most likely token Board value below a threshold (very close to losing). If we take away condition 2, Allie resigns strictly more often, and we would end up with a higher TPR.

Why does MCTS improve move accuracy, especially in expert-level human games in this work?

Chess is a game fundamentally about computation and search, and it is plausible that expert-level players can search sufficiently deeply such that next token prediction (or more generally, methods that use finite compute) just cannot simulate this computation. In such cases, we can reasonably expect search-based methods to better model expert behavior.

Evaluation set excludes moves made under time pressure

We acknowledge that “blunders” under time pressure is an interesting behavior to learn, but we decided to follow the experimental setup of Maia to make our training procedure consistent with the previous state-of-the-art method, so our results are more comparable. We agree that adding time remaining for each move can further improve the ability of the model to model human pondering time, this would be a relatively small change and we don’t expect it to change our results significantly.

Should show results on other games to demonstrate generalizability

Chess is a longstanding challenge in AI development, dating to the 1960s. We strongly believe that contributions to chess AI are meaningful in their own right, without the need to demonstrate generalizability. However, we do think the generalizability of our method is a very interesting question. We expect techniques that lead to more human-aligned chess play would transfer well to other two-player combinatorial games, and could potentially prove useful for domains in which machine learning models are trained to model human-like decision making. We look forward to follow-up work studying these questions.

Do you enable resignation for ALLIE as mentioned in Section 5.2 for the LARGE-SCALE HUMAN STUDY?

Yes, resignation is enabled.

Do ALLIE-SEARCH and ALLIE-ADAPTIVE-SEARCH use the softmax function for final move decisions, as ALLIE-policy does? If not, the experiment may be unfair.

Both search variants use softmax to pick a move like Allie-policy.

Thank you for your detailed read and feedback on our paper. We address some of your concerns below.

the novelty of this paper is very limited

The reviewer may have missed the main contribution of our work—we combine a policy learned from real human games with an adaptive search. The adaptive search method is a new (and more effective) way of scaling inference-time compute compared to traditional MCTS. In reviewer onxn’s words: “This time-adaptive search proves to be crucial for closing the skill gap between Allie and the human opponent, as shown in the human study the authors conducted.”

Allie uses a Transformer architecture with many more parameters than Maia

We first want to highlight that our main contribution, the time-adaptive search procedure, is the key algorithmic innovation that enables Allie to play at a grandmaster level (2528 Elo) against 2500 Elo human opponents, while no-search Allie baseline performs much worse (2136 Elo). This gain is independent of the model architecture and parameters.

That said, we completely acknowledge that using a much larger Transformer architecture, which uses more parameters and inference compute plays a role in the improvement of move-matching accuracy over Maia, but Maia is the best open-source chess model that we have access to, and we simply don’t have access to a Maia-like model with equal parameters to compare against. To test the impact of model size on the performance of Allie, we introduced an Allie variant with half the parameters in our ablation study (Appendix F, Figure 9), and the move-matching accuracy (54.2% with half parameters) still convincingly outperformed the Maia model (51.6%). To provide more evidence that model architecture and parameter count clearly isn’t the entire story: when comparing Allie’s move-matching accuracy against GPT-3.5 – a much larger Transformer model that was partly trained on chess data, Allie predicts human moves more accurately on average.

Thank you for your detailed review of our work. We hope that our response below addresses your questions.

Entirely based on offline data, based off recordings of human gameplays

We argue that to train a humanlike chess model, we have to rely on offline human data as the main ingredient: as an analogy, you can’t really teach a language model to generate human language without records of how humans write. While it is plausible that additional self-training could improve the strength of our models, it is not clear such methods would help with our objective of creating a “human-aligned” model, since we are not after creating a “superhuman” chess AI.

Moreover, large parts of the data have been removed for training (downsampling low-Elo games, removing transitions under time pressure), so it seems training ALLIE-like agents require a specific training regimen.

In terms of downsampling low Elo games and pre-processing, we would like to point out that similar data pre-processing steps are standard in model training. For example OLMo uses text quality and deduplication filters, and ResNet uses random crops for image augmentation. Removing moves under time pressure and downsampling low Elo games are similar in spirit to standard practice, and they should not limit the generality of our approach.

There does not seem to be any related work on skill calibration. Although I am not an expert in that area, I would be curious about what works have been done in that regard.

We agree that skill calibration is an understudied area and there doesn’t seem to be many papers on this. One very recent paper is Maia-2 that adds skill embeddings into the neural network, and reports better move prediction compared to the original Maia models, but it is difficult to directly assess its skill calibration, since that would require evaluation against actual humans players.

I find the claim that ALLIE has a "reliable board value estimate" a bit misleading. The reviewer may have misinterpreted the results on time prediction (Figure 3) for value prediction.

Low correlations (<0.3) of Allie’s value prediction with game outcomes when the game is 30+ moves from being finished is very much expected: the outcome of a chess game is fundamentally difficult to predict, and the uncertainty in the outcome decreases naturally as the game progresses (e.g., one of the two player usually gains a decisive advantage). Allie performs almost as well as stockfish (the state-of-the-art chess engine) in predicting the player who wins a chess game at intermediate positions, which we consider to be quite impressive.

In addition, we would like to point out that learning which player would win a chess game is not our main contribution, and we simply need a reasonable value function to guide the time-adaptive MCTS procedure.

Would a fine-tuned GPT3.5 on the same dataset as ALLIE (but in PGN format) result in a human-aligned agent?

In principle, GPT-3.5 could be fine-tuned to match human behavior, and it would probably outperform Allie. However, there are two reasons for not doing doing this.

• Limited scientific insight since we don’t know what into the pre-training data of this model in order to give it Chess-playing capabilities.
• No longer possible to do since OpenAI deprecated finetuning its non-instruction-tuned models. More recent instruction-following models such as gpt-4o don’t seem to exhibit the same strong chess ability as gpt-3.5.

We appreciate your positive feedback.

I don't think the term human-aligned is good in this paper.

We agree that the term “alignment” is typically used to describe “being helpful and harmless to humans”, but we define human-aligned as being both “humanlike” and “skill-calibrated” (see our introduction). Hopefully this is unambiguous in the context of chess.

I'm not clear how the <resign> token can be used in practice, even with a false positive rate of 0.1%.

You might have misinterpreted our results. The average game has about 35 moves, and with a false positive rate of 0.1% per move, the model would incorrectly resign once in 28.5 games. When deployed online, Allie resigns when down substantial material or when an opponent is about to promote a pawn (two FENs of positions where Allie resigns against humans are provided below), similarly to how humans would resign.

1N6/5pk1/1R2n1b1/R5pp/3Pp3/4P1P1/5PP1/4N1K1 b - - 0 38

3k4/pp2RP2/3p4/6B1/3P4/2Pr4/Pn4K1/8 b - - 0 34

No code available

Our code is already publicly available on Github, and our models are already available for download on HuggingFace. We have uploaded a zipfile that contains an anonymized version of our code.

We really appreciate your comprehensive review of our paper and constructive feedback, both positive and negative. We are encouraged that you find our methodology convincing, and consider our approach as advancing the state of the art.

There is some missing related work. I realize you may not have seen some of these papers in time, but hope they are helpful in revising your paper.

Thank you for pointing out the relevant papers, including the quite recent Maia-2 paper. Maia-2 and our work share the common goal of better aligning chess AIs with human behavior. We have added a discussion summarizing the similarities and differences among Maia, Maia-2, NeurIPS'21 and KDD'22 and Allie in the revision. On the use of the term “human-aligned:” building a human-aligned chess model is precisely the goal of Maia, but we argue that better strength matching with humans can be seen as a component in this alignment, which as you pointed out, we help achieve through a behavior-cloning model with inference-time search. We have revised our language regarding the use of “human-aligned”, particularly in the context of prior work.

The size difference between Allie and Maia's architectures (and its connection to increased move-matching accuracy).

The size of Allie’s architecture is indeed much larger than Maia’s and we definitely agree it is one of the factors behind the stronger performance of Maia. We have mentioned both the larger size of the training set, as well as additional model parameters as a factor that explains the increase in move-matching accuracy in our revision.

The difference between Allie's value head and Stockfish's evaluation. How accurately does Allie's value head predict game outcomes?

The reviewer may have misunderstood Figure 4—we are reporting the correlation between Allie’s value predictions and human game outcomes. Since we don’t know if Allie’s predictions are any good if we don’t have a point of comparison, we also computed Stockfish evaluation as an “objective” metric, and show that Allie predicts human game outcomes about as reliably as Stockfish.

The non-human-like aspects of MCTS search. Can you report move-matching accuracy as a function of search time? / How sensitive are your results (e.g., move-matching accuracy) to the constant 'c' in the rollouts function?

We did not try other search budgets in our experiments, but we agree with the reviewer that this would be an interesting analysis to run—we may explore adding this analysis in our revision in the next few days.

How did you choose the constant 'c' in the number of rollouts function?

$c$ is chosen such that we perform 50 rollouts on average across all positions, based on statistics computed on the training dataset. We do 50 rollouts for both adaptive-search and standard MCTS in line with prior work [1] which studied augmenting behavior-cloned chess and go policies with MCTS.

Explain why Allie with adaptive search does better than AlphaZero-like MCTS search against 2500 Elo opponents.

Thanks for bringing [2] to our attention! Adaptive and non-adaptive search use the same amount of compute on average throughout a game, and the key difference is that adaptive search allocates more compute to positions that humans spend additional time in. [2] suggests that humans spend more time at positions that search is more valuable. Since Allie’s time predictions are well-correlated with actual human time usage (Figure 3), it is not a complete surprise that doing search proportional to predicted human time usage results in significantly stronger gameplay than uniform allocation of search budget across every position.

Are soft tokens encoded in a way that also suffers from sparsity?

Since the soft tokens are basically interpolated embeddings based on Elo, they don’t require observing games for every Elo value which helps with the sparsity issue a lot. As an example, suppose that the model never observes games by a 950 Elo player. With interpolated soft tokens, the skill embeddings would still be very close to say that of a 955 Elo player, which we may have training data for.

Any normalization/weighting applied to the loss function terms?

We normalize labels of pondering time to have variance 1. All three losses (move, value and time prediction) have equal weighting in the training objective. We will clarify this in the revision.

[1] Modeling Strong and Human-Like Gameplay with KL-Regularized Search. Athul Paul Jacob, David J. Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown.

[2] Time spent thinking in online chess reflects the value of computation. Evan Russek, Daniel Acosta-Kane, Bas van Opheusden, Marcelo G Mattar, and Tom Griffiths.

Allie considers an easy win for a human may be considered drawn/losing by Stockfish. It would be very interesting to explore this more deeply.

Thank you for your thoughtful comment. Allie's supervision on human game outcomes should teach the model to assign high values to positions that a human can convert (as opposed to positions that are theoretically winning under perfect play), and vice versa. We initially overlooked fundamental factor that explains the correlation of Allie's value prediction with human game outcomes.

To explore this deeper, we compare the accuracy of Allie's value predictions (vs. Stockfish evaluations) in predicting game outcome at various stages of the game. We first filtered for decisive only games (games ending with draws are omitted from the analyses).

Analysis 1. What is the role of metadata (specifically, player Elo) in value prediction?

Table 1: Game outcome prediction accuracy of games with $\le$ 10 Elo gap between players

Table 2: Game outcome prediction accuracy of games with $>$ 100 Elo gap between players

Elo alone contains a lot of information about the likely player to win. In games with a large skill gap (by $\ge$ 100 Elo), Allie outperforms stockfish by 9.4% in middle game in game outcome prediction, which is a lot higher than 2.3% in games with a small skill gap. But like you suggested, Elo is not the full story---in games with almost no skill gap, Allie still outperforms Stockfish in all phases of the game.

Analysis 2. Does Allie "know" which positions humans can/cannot convert better than Stockfish?

Following your suggestions, we qualitatively analyze a few randomly chosen positions from our test set where Allie and Stockfish make different predictions of who will win.

In this position in the opening, black has a knight for 2 pawns, but is unable to castle. The position is sharp, and although Stockfish gives black a decisive advantage (black has 77.0% probability of winning), Allie prefers white (78.8%). Interestingly, black immediately misplays the position and loses. Like you pointed out, Allie does appear to produce better value estimates for such positions that are objectively winning, but difficult for humans to convert.

FEN: r1bq1b1r/ppp2kpp/2n2n2/3P4/2B5/8/PPPP1PPP/RNBQK2R b KQ - 0 6

In this position in the endgame, Allie quite surprisingly considers black winning with about 75% probability when white has an extra queen on the board (stockfish assigns 98% to white winning). Our best guess is that Allie inferred that white is low on time by observing the move history---Allie considers black winning starting at move 34, when white blunders a rook potentially due to being relatively low on time.

FEN: 8/2Q5/p7/6p1/P2P2kp/4P3/4K3/8 w - - 2 51

However, Allie seems to have blind spots with evaluating checkmates/sacrifices, which strong human players are capable of calculating. Arguably, solving checkmates fundamentally relies on search and is difficult to do with a value function alone. One interesting example is this position in the middle game between two strong players. Allie gives black a 67% probability of winning, but white is objectively winning if they find the queen sacrifice Qe7. The human player finds this move, and wins the game as Stockfish predicted.

FEN: 4r1k1/2p2ppp/p1P2P2/1p4qb/1Q6/1BP5/PP3PpP/3R2K1 w - - 4 24

We will incorporate these analyses into our paper in the future.

A user study was done by deploying chess bots online and surveying people afterwards about their experience playing the bots. The deployment of bots on Lichess is a public and controlled process, but it's not clear to me if the deployment and survey collectively require an IRB.

Hi, this is the senior author chiming in. The survey we conducted is IRB exempt as per https://www.ecfr.gov/on/2018-07-19/title-45/part-46#p-46.104(d)(2)(ii) .