Thank you for your review and feedback!

looking a puzzles where Leela gets the next move wrong would be a good first step, i.e. what is the accuracy of your move predictor when the model is wrong, or can you patch to get the correct move.

We focus on puzzles that Leela gets right simply so we can apply our interpretability methods. For example for the move predictor probes: what should be the “ground truth” for the 3rd move in cases where Leela gets the puzzle wrong? We could see if the probe can still predict the 3rd move of the correct continuation, but given that Leela got the puzzle wrong, there’s no reason to expect this. Leela might still use look-ahead along some incorrect continuation, so we could ask how good the probe is at predicting the 3rd move in this incorrect line. But of course, there are many different incorrect lines (vs one correct line), and we don’t have ground truth for which, if any, Leela is considering. For patching, it will massively depend on how invasive an intervention we perform. For example, patching the right squares in the final layer could just override the move output to be whatever we want.

The model internally doing a depth 3 search seems very plausible and I don't think proves the motivating claim of "look-ahead in neural networks"

We don’t follow; if the network is doing a depth 3 search (over possible move sequences), we’d certainly consider that an instance of look-ahead in neural networks. Note that we are not claiming look-ahead arbitrarily many steps into the future, is that the point of confusion?

Could you give a more formal definition of " look-ahead in neural networks" and explain how this work proves it?

By look-ahead in neural networks, we mean internally representing future moves (of the optimal line of play) and using those representations to decide on the current move (see line 45). Our probing results (section 3.3) are our most straightforward evidence for representations of future moves, though the residual stream patching experiments (section 3.1) also suggest this already. The patching experiments in section 3.2 give evidence of such representations being used the way we’d expect, and in particular to decide on the current move. For example, L12H12 seems to move information from future move target squares back to the target square of the next move, and a very targeted ablation in this head has outsized effects on the output.

Does the move probe work on human games or non-puzzle positions?

All puzzles in the Lichess dataset we use come from human games, they are simply selected to be tactically interesting states. In many chessboard states, there are many plausible lines of play, so predicting several moves into the future is fundamentally much less feasible. Using puzzles means that at least there is a clear ground truth correct line, and we can then check whether Leela represents it.

Figure 8, what is the accuracy measuring? There are 64 squares on a chess board, how is random getting above 2% accuracy?

The “probe on random model” line in Fig. 8 is a probe trained on a randomly initialized model, which is different from a randomly initialized probe (or random predictions). Even the activations of a randomly initialized model contain information about the input, they just don’t contain more sophisticated features. So this baseline checks how well future moves can be predicted just by the probe itself, without interesting features from Leela.

What would a falsification look like in this method?

If the experiments we ran didn’t yield non-trivial effects, we’d consider that evidence against look-ahead. For example, a probe trained to predict future moves could have achieved accuracy not much better than the baseline probe, or patching on future move squares could have had effects no bigger than patching on other relevant squares.

the lack of consideration of alternative hypothesizes or attempts to disprove the results weakens the results.

We did implicitly consider alternative hypotheses in every one of our experiments. For example, for the residual stream patching experiment (section 3.1), we wondered whether future move target squares only have big patching effects because these squares tend to be “obviously”/”heuristically” important in the puzzle starting state. That is why we used a very strong baseline, by taking the maximum effect over other squares. While future move target squares might often be important for heuristics, so are many other squares, and under a non-look-ahead hypothesis, it seems likely that the maximum over other squares should typically be larger than the importance of one specific future move square. We have similar baselines to rule out simple alternatives in all our experiments.

We realize that this reasoning might not always be apparent in the paper, so we’ll include the motivation for baselines more explicitly. Thanks for drawing attention to this!

Of course, we can never fully prove that there's no alternative explanation for our results, we can only rule out specific alternative hypotheses. We don’t know of any concrete hypotheses not involving look-ahead that predict or explain our results, but we would be very curious in case you have a specific one in mind.

The patching for example seems so show that you can disrupt the chosen square, but this assumes the activations are encoding this linearly. If they were non-linear or only partially in the patch how would that changes these results?

There seems to be a misunderstanding here. We do not assume any form of linearity for our patching results. We do operate under the hypothesis that information about future moves is localized to their squares, but this is something we argue for with our results rather than an assumption (see line 128).

Thank you for your careful review and great suggestions!

Throughout the paper I was wondering why the policy network and not the value network was used.

Great question, there was no particular reason for this choice except that we decided to focus on only one head for the purposes of exposition. We have re-run all our experiments with the value head (i.e. using the log odds of the win probability instead of the log odds of the top move as the effect metric). As you suspected, the results are qualitatively the same as for the policy head, see the PDF attachment to the general response. We’ll include these in the paper and also clarify that the network has a shared body and two small heads.

I think it would be nice to mention a few more details about Leela in the main paper.

Thank you for the suggestion, we agree this would help readers and will move some information from the appendix to the paper!

It would be interesting to see how the findings regarding look-ahead evolve during training.

We agree this would be very interesting to study, but also that it’s out of scope for this paper. There are public training checkpoints for at least some versions of Leela, so in principle, this would be feasible to study in future work. Applying our exact methods would require finetuning away the history requirement for each checkpoint, which should be possible (albeit somewhat computationally intensive if we want to use many checkpoints in order to even detect potential sudden transitions).

Are there situations where the weaker Leela gets the puzzle right, but the stronger Leela does not? Or is the stronger Leela strictly better?

The weaker Leela very occasionally does better than the strong version “by luck.” For example, there could be a move that looks good to the strong Leela but is, in fact, very bad for a subtle reason that even the strong model misses. If the reasons in favor of this move are themselves somewhat subtle, then the weak model might not even consider it, and just play a mediocre move, rather than the strong model’s bad move. But apart from such edge cases, the difference in playing strength is quite large, and we would be surprised if there are any classes of states where the weak model is systematically stronger.

how has the weaker Leela been trained? Is it plausible that, e.g. stronger MCTS during training of the weaker Leela means look-ahead is much less important, hence why it does not develop look-ahead (or does look-ahead only develop at sufficiently large and strong networks)?

Both models were trained using supervised learning on MCTS rollouts (so the loss functions are the same as in MCTS training, but the data comes from an existing strong network, rather than the network under training). Given that MCTS trains the network to predict the results of tree rollouts, we don’t think strong MCTS would make look-ahead less important (this would only be true if the network was optimized end-to-end to maximize the playing strength of the overall MCTS process). We’d indeed guess that look-ahead becomes more prevalent for stronger networks, but of course, this would require future work to actually test.

The probe accuracy in Fig 8 is relatively high in early layers - how does this fit with the activation patching results that show the largest effect only in medium-to-late layers 6-11?

Good question, we are not entirely sure what the answer is. One possibility is that there is a collection of different mechanisms involved in look-ahead, and not all of them activate in the same cases or are placed in the same layers. Probes might learn to exploit look-ahead mechanisms that are present in early layers, whereas the more targeted activation patching experiments only pick up on a subset of look-ahead mechanisms. L12H12 is already an example of a mechanism that is sometimes involved in look-ahead but is important less often than look-ahead mechanisms in general (such as the piece movement heads, contrast figs. 5 and 7). So it’s plausible that there are also some mechanisms that only probing picks up on.

As far as I am aware the precise details about the training process of this variant of Leela have not been published (in an academic venue). While this is beyond the scope of this paper, having these details (or as many as possible), e.g, in the appendix, would greatly help the scientific community to use this version of Leela as something that can be reliably reproduced.

Indeed, most information about Leela is only available on the Leela Discord (which is public but naturally not designed to convey all that information compactly). We’d be happy to extend appendix A with some additional information (but a full description of Leela’s training details would fill its own paper).

Thank you for answering my clarifying questions and commenting on my (mostly optional) suggestions for improvement. I am positively surprised to see a full analysis of the value-network too. Overall I am happy with the authors' responses and additional results. I stand by my original score - I think this is a great interpretability case study with interesting results, that is ready for publication, well executed, and interesting to the NeurIPS audience.

I have also read the other reviewers' comments and authors' responses and consider all raised issues sufficiently addressed (though I will happily take into account reviewers' future comments in case they disagree). abVe and p7Jw seem to be mainly concerned by the dataset filtering and only showing results on this filtered dataset - while this is understandable criticism at first, I think constructing such a dataset in a reasonable manner is a contribution in itself. The difficulty is that a large neural action/value predictor behaves quite differently from a search-based chess algorithm; in many board states it may be that the neural system does not use look-ahead-search but instead relies on memorization and exploitation of (statistically) similar patterns. It is thus crucial to first identify states where look-ahead-search may be needed. I am also not too worried about the use of the puzzle dataset - previous works (e.g. the 'Grandmaster-level chess without search' that abVe mentioned) have found strong correlation between puzzle performance and actual game-playing strength across a variety of models. The paper claims (and is very clear about this) that convincing evidence of look-ahead-search can be found in these situations. It does not claim that this is the main mechanism that the network uses at all times. Without proper filtering, the evidence for this mechanism might quickly "drown" within the (potentially large) number of situations where no look-ahead search is performed. While it would be interesting to know "how often" the network uses its learned search, this question is beyond the scope of the paper, whose goal is to clearly establish that the paper has learned to use look-ahead search at least sometimes (which I consider a highly non-trivial result).

Thank you for your thoughtful feedback and questions! We answer them below and clarify potential misunderstandings.

the claim that "look-ahead or other sophisticated algorithms should pick up on the importance of this difference, but shallow heuristics should mostly ignore it." required more justification. Some of the tactical patterns picked up by the method are so common that it's conceivable that a "shallow heuristic" is designed for them, rather than look-ahead.

We’d like to clarify that this claim is not a load-bearing assumption for our results. The sentence you quote only motivates why we use corrupted inputs that are similar to clean inputs (in the sense that a weak model gives the same output): if we used corrupted inputs that differ too much from the clean ones, then too many network activations would differ and drown out the evidence of look-ahead.

If our argument for look-ahead was behavioral, i.e., the fact that Leela can solve these supposedly difficult puzzles, then it would indeed be an issue if some of them are also solvable by shallow heuristics. But our argument instead rests on observations about internal representations, and these observations suggest look-ahead irrespective of what exactly the corrupted inputs are. For example, if we could show that future move target squares are unusually important under random corruptions, this would be just as convincing in our mind.

In summary, we agree that shallow heuristics could likely pick up on some of the tactics, but we don’t think this weakens our arguments.

What is the evidence that look-ahead is occurring as opposed to a shallow heuristic designed to pick up on common tactical patterns?

We think all three lines of evidence we present favor look-ahead over shallow heuristics. For example, a shallow heuristic (which decides on the immediate next move based on matching to common patterns) would not need to explicitly represent which future moves will actually be played. But we find in section 3.3 that such representations of future moves exist, as a look-ahead hypothesis would predict. We similarly think that hypotheses that don’t involve look-ahead would not predict or explain any of our other results, but please let us know if you have specific concerns about this.

How often does look-ahead occur?

Good question, but unfortunately difficult to answer precisely, for two reasons:

• It depends on where we’d draw the boundary for “look-ahead” (for example, which effect size would be sufficient in our various experiments). While we think our results are strong enough to conclude that look-ahead is involved in at least some inputs (those with high effect sizes), it’s much less clear where exactly to set the threshold.
• Our methods can only establish a lower bound on how often look-ahead occurs. For example, throughout the paper, we consider representations on future move target squares. In principle, additional look-ahead mechanisms that use entirely different pathways could exist, which we might not pick up on.

With those caveats in mind: Our dataset is 2.5% of the initial Lichess dataset (see lines 83 and 91). On this dataset, we think look-ahead is involved in most states (but as discussed, it depends on where we draw the boundary). However, these 2.5% of states were, of course, not directly selected for high effect sizes; rather, we only used a behavior-based filtering procedure. So the true number of states where look-ahead is important is likely significantly larger than this among Lichess puzzles.

Why aren't the squares of the 2nd move similarly important?

As discussed in line 163: “We are unsure why the squares of the 2nd move aren’t similarly important. This may simply be because the opponent’s move is typically “obvious” in our dataset or because suppressing the opponent’s best response doesn’t reduce the quality of the 1st move.”

If the two hypotheses we give there are right, the apparent unimportance of the 2nd move would largely be an artifact of our experimental setup (studying tactics puzzles where the current player is winning). But we can’t rule that there is some interesting deeper reason that would require a more detailed understanding of Leela’s internal mechanisms.

Why aren't there pawn or king heads, similar to the other piece types?

Great question! There are, in fact, pawn and king heads, they just involve a few complications, and so we decided to ignore them for simplicity. But we see that not mentioning them was a mistake, thank you for drawing attention to that! We’ll rectify this by adding a few comments to the paper:

• A few king heads exist (based on attention patterns). But our dataset doesn’t contain many puzzles where the starting player moves their king on the first move, so we wouldn’t have much data for our piece movement head ablation. This is simply because king moves are much rarer in tactics puzzles.
• Pawns move in different ways depending on context—they usually take one step forward, but they capture diagonally, and they can take two steps at once if they’re in their starting position. Based on attention patterns, it seems that there are distinct heads for these different types of movement, so we decided to ignore pawns to simplify the presentation in the paper. Again, we recognize this omission is also liable to lead to confusion, so we will bring this up in the paper.

Nitpick: the motivation in the first paragraph about chess being different from other domains because in other domains "models can solve their tasks with a single simple algorithm" also applies to chess.

Thank you for pointing this out, our phrasing here is unfortunate and we’ll improve it. What we meant to gesture at is simply the difference in complexity between playing chess well and e.g., finding the path from the root of a tree to a specific leaf (which is the task studied in the most similar prior work).

Thank you for your review and insightful questions!

We agree that “existence proof” was poor wording on our part and will rephrase that to “evidence” in both places, thank you for the suggestion. Thanks also for the minor comments, which we’ll incorporate.

is it definitively possible to draw a hard line between "heuristics" and look-ahead?

This is a great question, and as you say, a full answer is probably a difficult philosophical problem. We give a few initial thoughts below.

Drawing a very precise line around “heuristics” might not be possible (in the sense that the space between ad-hoc heuristics and general reasoning algorithms might be relatively continuous and it’s not exactly clear where the boundary is). Even so, we can probably say that certain algorithms do involve look-ahead and certain others don’t (while leaving the question open for some borderline cases).

Implementing look-ahead by simply counting attacks/defenses in the current state, as you describe, is an interesting example of blurring this boundary. But capturing all relevant considerations this way likely only works well for 1- or perhaps 2-ply look-ahead. In the 3-ply case (which the paper focuses on), a piece might be moved twice (and already in the 2-ply case, heuristics would need to be very careful about counting defenders correctly to account for lines closing or opening up as pieces are moved).

So while in some sense, there is no limit to what heuristics can implement—in the extreme case, a look-up table could implement arbitrary policies—we expect algorithms that explicitly use look-ahead to be far more efficient.

can we ever really distinguish between look-ahead search and heuristics, if the look-ahead search is depth-limited

The thoughts outlined above suggest that a depth limit does not preclude a distinction between simple heuristics and look-ahead: while given enough capacity, both might be able to implement the same policies, they can differ in how much capacity that takes (and for sufficient depths, avoiding explicit look-ahead might be infeasible).

Of course, our paper does not rest on these speculative conceptual claims; we aim to give evidence of explicit look-ahead rather than argue for its inevitability. (As one simple example, the method of counting attacks/defenses would not need to explicitly represent which future moves will actually be taken, so it would not predict our probing results.)

Thank you for your thorough review and suggestions!

Policy vs value network (2.3): Great suggestion, thank you! We have now run all our experiments on Leela’s value head and will include these in the paper. We used the logs odds of the win probability as the effect size for patching experiments. (Probing experiments operate only on the shared network body so don’t differ between heads.) All our findings for the policy head still hold for the value head, see the PDF attached to our general response. The only new finding is that L14H3 seems very important to the value head (Fig. 2 in the attached PDF), but L12H12 is still just as important as it was for the policy head (log odds reduction of 0.49 when we ablate it).

On dataset filtering (2.1): We appreciate that filtering our dataset can seem strange. However, we think this filtering process is essential to test our hypothesis and doesn’t negatively affect the validity of our results. We clear up potential misunderstandings below.

This is quite weird since you also somehow optimize the evaluation dataset to help you find the claimed phenomenon.

Investigating our claimed phenomenon fundamentally requires creating a suitable evaluation dataset somehow. As discussed (e.g. line 80), our claim is only that Leela often uses look-ahead in certain types of states, namely tactically complex ones. Such hypotheses specific to a certain input distribution are omnipresent in mechanistic interpretability (e.g. [1, 2, 3, 4]). To test this hypothesis, we naturally evaluate it on those states for which we claim Leela uses look-ahead. Our filtering process formalizes the vague notion of “tactically complex states.”

may only work on some human created settings -- an extreme case is that I can manually design some test cases that can definitely present this ability but it lacks generality

Indeed, manually creating inputs would be very suspect, so we want to emphasize that we start with an existing dataset of inputs and then apply a simple automated filtering step. Importantly, we also do not use internal representations in any way during filtering.

Also the authors filter out all puzzles that Leela Chess may fail. This is really weird and can largely weaken the conclusion

We focus on puzzles that Leela solves simply so we can apply our interpretability methods. All our methods check whether Leela represents a specific future line. In correctly solved puzzles, we can look for representations of the correct continuation. In puzzles that Leela fails to solve, this wouldn’t tell us as much: Leela may be representing an incorrect continuation (and hence fail the puzzle), but there are many incorrect lines (vs only one correct one). So we can’t test for look-ahead like we can for correctly solved puzzles. We briefly explain this in line 89 but realize that this deserves more space, so we’ll incorporate this explanation in the updated paper.

As to whether this weakens the conclusion: we want to explain why Leela often solves puzzles correctly and don’t claim to explain why it sometimes fails. We’ve tried to be transparent about this (e.g. line 11 in the abstract) but will do another editing pass to make this clear.

if you have already tested it to solve the puzzle, it is quite normal that the representation of policy network can have something closely related with future optimal action

Crucially, we don’t think that Leela solving a puzzle necessarily implies the claims we make about internally represented look-ahead. Drawing such mechanistic conclusions from behavioral evidence is dubious, which is why interpretability is needed in the first place. We agree that the model’s ability to solve difficult puzzles might lead us to expect that its representations would have something to do with future actions. Our paper's contribution lies in actually testing a more precise version of this hypothesis.

Testing other transformer-based chess-playing networks (2.2): This would be an excellent future direction! Unfortunately, the model weights you mention were released only a bit over a month before the NeurIPS deadline. Transferring our experiments to new models would be a non-trivial effort, e.g., since we need to add instrumentation for our patching experiments.

Regarding RL vs. supervised learning, note that the version of Leela we use was, in fact, trained using supervised learning (see line 422 in the appendix for details). The difference is just that it uses data from rollouts of an MCTS-trained Leela, rather than Stockfish evaluations.

Answers to questions:

1. Thank you for the references, we will cite these as additional examples of transformer-based chess models.
2. That’s right, except we patch entire activation vectors on a square, rather than individual neurons. Our procedure for finding corrupted inputs is described in line 120 and appendix D. In brief, we randomly generate corruptions and then pick one that doesn’t change a weak model’s output, but does change Leela’s output a lot. In Fig. 3, we omit the 2nd move target since it is very often the same as the 1st move target (see line 93). Figure 9a demonstrates that the effect is mainly about the 1st and 3rd rather than 2nd move, which is why show the 1st instead of 2nd move.
3. Are there specific aspects that you think are difficult to follow and would benefit from additional figures? That would help us a lot!

[1] Wang et al, 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[2] Hanna et al, 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

[3] Lieberum et al, 2023. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

[4] Nanda et al, 2023. Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level